#wordpress

Social Importer Upgrade

https://beau.blog/2017/01/social-importer-upgrade/

Today I pushed some updates to:

  1. People & Places
  2. Keyring Social Importers

These updates make it so that the Twitter, Foursquare and Instagram importers are now dynamically identifying and indexing People and Places, and marking them with a taxonomy within WordPress. I’ve also added a new system for “reprocessing” old posts which Keyring imported, so that you can go back and perform some function on those posts without having to import them again. You’ll find reprocessing tools under Tools > Import > Reprocess Keyring Data.

Social Importer Upgrade

Reprocessing works by using the locally-stored copy of import data that is saved during the initial import of everything. The system is fully hookable, so you can add other reprocessing routines in via plugin. The core file comes bundled with one that attempts to address an old JSON-data-escaping issue, and I’ve added extensions to the importers listed above which allow you to go back and reprocess your posts for People/Places.

If you’re going to use them, I suggest you run the first one first, then you can run the others in any order you like. Doing the first one first will just make sure that as much of your data as possible is processable.

Social Importer Upgrade

It’s worth mentioning that if you use these reprocessors, they can take a while (especially if you have a lot of data already), and that they will likely create a lot of new data (in the form of People and Place terms being created in their respective taxonomies). After running all of them over all of my data, I have almost 1,800 People and just over 3,000 Places in my database.

The other tool added in this upgrade is the ability to merge terms, which becomes important with all of this data.

Social Importer Upgrade

When the importers are dynamically adding People and Places, they only match based on known identifiers. This means that you’re likely to end up with duplicate entries, especially if you’re processing multiple services (e.g. Foursquare and Instagram). Using the merge tool, you can browse through your entries and select 2 or more, then use the Bulk Actions drop-down to select “Merge” and hit “Apply”. Terms will be merged together as intelligently as possible, which basically means that the shortest slug of the group will be kept, and the longest strings for any conflicting fields will be kept. You can of course edit the resulting composite term afterwards and tweak things as you see fit. If you’re looking for a shortcut to identify duplicate entries, try searching for “-2”, which will give you a list of duplicates, then you’ll need to search for something that will bring up each of the dupes, select, merge, repeat. It’s a little bit tedious, but you’ll only need to do it once for each duplicate, and all future imports should match against the composite entry.

Oh, and one last thing — I threw in a quick map on the details page for Places, which provides a nice quick, visual confirmation that it’s the correct location. For now it’s using a very basic OpenStreetMap example, but I might switch it out to Leaflet at some point, which is pretty nice.

Social Importer Upgrade

People & Places

https://beau.blog/2017/01/people-places/

Over the years, I’ve been working on a system to aggregate data that I publish to other social networks/sites back into my control, on my own WordPress install. Thus far, that has resulted in the creation of Keyring (plugin) to provide an abstracted interface to all of the web services I’m interested in, Keyring Social Importers (plugin) to do the basics of importing the data from different places, and Homeroom (theme) to display it all. Today, I’ve been working on a system that will detect people who are mentioned in an interaction, and link them across posts using a custom taxonomy. It does the same for physical locations, so I’ve called it People & Places.

Essentially, this plugin is just a pair of custom taxonomies, with some specific ways of referring to things. Pretty basic. It gets more interesting though when you update Keyring Social Importers to the trunk version, which will now work in tandem with People & Places to link everything up. I wouldn’t recommend it on a production site just yet — there’s a lot of rough edges still.

When KSI is pulling in content from each service (currently looking at Twitter, Instagram and Foursquare), there’s a new block of code that makes sure People & Places is available, and then looks for certain pieces of data. If it finds them, it bundles up the details, and passes that along in the import process. When posts are actually inserted, it will attempt to link up that post to the People/Places it found. If the People already exist, then they’ll just be linked, in the same way tags work. If they don’t exist yet, then a new Person entry will be created, and that will be used.

I plan to add in a basic term-merging function, so that you can manually (maybe automatically?) identify “duplicate people” across different networks, and intelligently merge their entires (re-linking any posts involved), so that you build up a single, combined view of your interactions with a particular person. I envisage some interesting possibilities with the archive pages for these taxonomies, and that over time it will build a really interesting dataset of your interactions, the places you physically go, etc.

I’ll probably still move the code around a bit, and there are definitely some bugs around duplicates and handling things across different networks, but it seems to be working so far. This is also probably the time to figure out a decent way to allow re-processing of imported data from the raw copy that the importers save in postmeta. Installing this new code will start gathering data on new imported entries, but won’t go back and do the same on all the posts you’ve already got. Rather than deleting all that data and re-importing/processing everything, I’d like to have a simple way to re-process the raw data that’s already stored locally.

All New WordPress.com

Almost 2 years ago, I wrote about how the future of WordPress needed to be a REST-based API, with a JavaScript client on top of that. There were even public rumors that, gasp, we were thinking about porting WordPress to Node.js. Well, while that’s not exactly true, it’s closer to the truth than a lot of people probably realized.

We are in fact using Node.js to power part of WordPress.com now. If you go to https://wordpress.com while logged in, your request is handled by a Node.js server, and the entire UI is written in JavaScript, although the majority of it is actually React.js. Unless you end up back in wp-admin, your admin/editing/posting/dashboard experience on WordPress.com is now handled entirely (well, almost; we’re still working on some pieces still) with JavaScript.

The UI is fully responsive (optimized for multiple screensizes, and flexible in between). Data updates are live (combination of polling and websockets, moving more to sockets over time). No full-page refreshes (it’s a single-page app). All API-driven (which means we can, and are, using the same APIs to power portions of the native mobile apps). Speaking of apps, we’re able to bundle our single-page, JavaScript application as a native app, so we did (OSX for now, Linux and Windows coming very soon). Leveraging our infrastructure, and the power of Jetpack, we can provide self-hosted users with the same experience as those we host directly (with more Jetpack-specific functionality coming soon as well).

This is a lot of change. But it goes deeper than just the entire technology stack we’re working on now. This was a complete culture-shift for Automattic, a now-400-person company traditionally made up of approximately 100% PHP developers. To get here, we at Automattic took a step back and asked ourselves;

What would WordPress.com look like if we were to start building it today?

As part of answering that question, we made a lot of changes internally:

  • Cross-trained all of our PHP developers (and some of our mobile developers!) into modern, performant JavaScript developers.
  • Switched to a completely GitHub-based workflow.
  • Every commit is now peer-reviewed.
  • Shifted to a very component-minded architecture.
  • Moved our WordPress codebase to be entirely API-driven. New features are now only launched as a new/modified API endpoint + data layer + UI layer.
  • Change in thinking from being very “plugin-oriented” (similar to WP-core) to a much more integrated and cohesive way of thinking of things across the web and mobile apps.

So today, in keeping with the DNA of Automattic, which shares the DNA of WordPress, we’re releasing what we’ve been working on as open source. It’s code-named “Calypso” (long story), and we’re extremely proud of what we’ve built over the last ~18 months. I truly hope that this can help guide or influence WordPress.org‘s future.

I wrote this post in the Calypso/WordPress.com Desktop app, and published it via Jetpack. That feels pretty darned good.

One Less Plugin, Thanks Jetpack

As a developer, I’m a huge fan of “red changesets”; when you get to delete more code than you add. Less code means less maintenance, less potential for bugs, security problems, etc. Today I got to “red changeset” the plugins powering this website because I realized I could just go ahead and delete the plugin I was using for update notifications.

I’ve been running a plugin here for a while now that emails me when I have updates available for plugins, themes or core on my WordPress installation. With the latest version of Jetpack though, I get that notification via a simple little indicator on WordPress.com (and I’m there every day already). In addition to the notification, I’ve enabled the auto-update feature for all plugins on all sites connected via Jetpack, so now I don’t even have to think about keeping my plugins up to date.

There’s a lot more cool stuff coming in this Jetpack/WordPress.com integration, and I’m really excited to see (and work on) what we can do to help make life easier for self-hosted users while leveraging the power of WordPress.com.

Simple Search Operators

I’ve long wanted to be able to do some simple, operated-powered searches within WordPress (especially relevant in a project I’m working on at Automattic). After a conversation with Allen, where he wanted to do the same thing, I figured I’d just whip up a quick plugin to see how hard it’d be. Turns out the answer is “not very”.

Simple Search Operators is a quick plugin that expands the functionality of the default search system in WordPress so that you can use a few useful operators (might look at adding some more at some point) [all links in the following list go to live searches on this site, using that operator]:

  • author:beau will limit results to posts written by the author with the username/slug ‘beau’
  • tag:burrito will only return posts which are tagged with ‘burrito’
  • category:posts (or cat:posts) to search for posts categorized as ‘posts’
  • tag:burritofriday cancun author:beau to search for posts containing ‘cancun’, written by ‘beau’, tagged as ‘burritofriday’

Some caveats:

  • Not heavily tested! May well be capable of generating server-melting queries
  • Only supports one of each operator for now
  • Operators and freeform searches may be combined (e.g.: “tag:burrito cancun”)
  • Does not support spaced strings for operators, so you can only do things where a no-space string attached to an operator will get you what you want
  • Because of the way it manipulates query variables, might mess with other plugins or themes in adverse ways. Like I said, not heavily tested.
  • Not available via the WP Plugin Repo yet; will get it up there once it’s baked a little bit

If you’d like to give it a shot, please do. If you’d like to add more operators; shoot me a pull request and we can expand this out a bit.

WordCamp Saratoga

Through some lucky scheduling, I was able to attend both LevelUp Con and WordCamp Saratoga in a single trip. I spoke at WordCamp about how to build a quick Backbone.js application which used WordPress as the backend (interfacing via the REST API). I thought my talk went OK, although I didn’t love it to be honest, and in hindsight I kind of wish I’d dived a bit harder into some better examples of how Backbone works with Views and whatnot. Here are the slides I used:

And if you’re interested in the code, it’s all available via Github. I got a few nice bits of feedback as well, so that was good:

The Future of WordPress: REST API + Javascript single-page app?

DISCLAIMER: These are my personal thoughts only, based on what I’m seeing around the web over the last few years.

Working on o2 for the last few months, and spending more and more time in amongst the Javascript community at events like jQueryConf, BackboneConf and dotJS, I’ve started thinking about the future of WordPress differently.

WordPress currently consists of a large, complex, PHP codebase, sitting on top of a MySQL backend. plugins are primarily PHP, with a light sprinkling on Javascript to mostly provide UI “candy”. Part of the reason for that is probably because:

  1. It’s pretty hard to interact with WordPress on any real depth using Javascript, and;
  2. If you’re trying to provide any custom functionality for WordPress, you pretty quickly end up doing it in PHP because that’s where all of the internal APIs and functionality resides.

That’s been a fine approach for the last 10 years, but I believe that things are changing, and we need to change with them to remain relevant. Full page reloads with heavy server-based everything is no longer really acceptable for a solid UX. With that in mind, here’s a possible future for WordPress, extrapolated out from where some things seem to be going (on the wider web, not necessarily currently within WordPress development):

WordPress core becomes 2 “separate” portions; one is a PHP-based REST-style API. Effectively, a layer that provides a structured API onto our database. That would cover all posts, users, settings, queries, meta data etc that lives within the WP DB. The other piece would be (ideally) a pure Javascript, single-page web application which fully implements the API to deliver what we currently know as wp-admin.

Plugins would create additional endpoints within the backend API, or would supplement existing ones (e.g. post/meta) with additional data and/or actions. They could also be implemented via pure JS, depending on what they were built to do. Themes could potentially be built using no PHP at all — making queries directly to the API via JS, and then using something like Mustache to template the output back to the user. This would have SEO ramifications, but we can always figure something out there, and search engines are constantly improving.

To get there would take a long time, and at many points it would no doubt break all sorts of plugins. This would be a huge shift and I think would actually be really hard to accomplish within the community, which makes me scared because I think it will be rejected as a direction (or at least slowed down a LOT) because of backwards compatibility at the very least. Perhaps we can maintain backwards compatibility for some plugins by at least making the new wp-admin a hybrid app, which leverages a JS-based approach for all truly core functionality/views, and then falls back to full page reloads for plugin-based admin functionality. Without a lot of work, that approach would still break for “integrated” plugins (where they inject settings  into or manipulate existing admin pages), if we assume that all of those pages would become powered by/created via Javascript + API interactions.

Unfortunately a crucial point where backwards compatibility would be important would be the Post Editor, which is also where some big performance/UX improvements could potentially be seen by switching to a largely JS-powered UI.

I don’t know if this is really where WordPress will go, or if it is, exactly how it will get there. There’s a project currently to build a core REST API which I’m eagerly observing and will be trying to get more involved in. It has the potential to become the kernel of the future of WordPress if it’s done right. This is going to be a long road either way, so I’m excited for where we can all go from here.

Do you think WordPress can (or should) move in this direction?