Yesterday I released new versions of both Keyring and the Keyring Social Importers packages, containing a bunch of updates and new additions. If you’re already using them, you should have update notices in wp-admin. If you’re not yet, then download them at the links above, or search for “keyring” in wp-admin under Plugins > Add New.
What’s changed? It’s been a while since the last official release of Keyring, so there’s a bunch to catch up on:
All Google services have been modified to use a shared base service (cuts down on code duplication significantly).
Lots of bugfixes, including token refreshing should now work properly.
The Social Importers haven’t seen an official release since 2017, so there’s a ton going on there as well:
Added a Strava importer (props @mdrovdhal) and introduced a bunch of improvements via iteration (props @marekhrabe). Having another service with map-based data makes me want to add some core to make it easier to map things visually.
Introduced a global option (for all importers) that allows you to set posts to published, draft, private, or pending when importing them. A lot of people were asking for/hacking this in, so I figured I’d just add it to the core package. Being able to import as draft and then selectively publish, or import an entire service to “private” posts is a nice addition.
Lots of improvements and bugfixes to both Twitter (some props @chrishardie) and Swarm/Foursquare.
Added a Pocket importer, again props @roccotripaldi. It works similarly to the Instapaper one, so if you’re using Pocket instead, check it out.
If you’d like to keep an eye on things more closely, or even contribute, check out Keyring, and the Keyring Social Importers on GitHub. It’s been really awesome to see some more contributions to both packages coming in, so I’d love to see more of that.
Then MyFitnessPal automatically syncs the data from Health Mate, and keeps a copy,
before Garmin Connect also syncs a copy to integrate back with other data (like heart rate and activities) .
If I’ve been doing specific activities, then some of the data from Garmin Connect (although not that weight info) will also sync out to Strava and create activity records.
In 7072886 I added a first swing at a YouTube Service definition to Keyring. It’s based heavily on the Google Analytics one that was recently submitted via PR. It’s not part of an official release yet. I’m curious about enabling people to import the videos they publish on YouTube into a WordPress install though, as that feels like something that folks who use it heavily would want as part of their web presence.
Next step will be adding an importer for YouTube, which I'll probably aim to do over the coming weeks.
I recently spoke at a small conference we put on in Detroit. I talked about The Future of the Web, in the context of data ownership, and proprietary vs open platforms. It was the first time I’ve done a talk like this without slides of any kind, so I actually wrote out the whole thing (below). The live version was a little different, as you can hear in the audio (sorry it’s a bit echo-y):
On our current path, the web as you know it today will cease to exist. Ironically, the web of tomorrow will look more like the web of yesterday, when AOL and Prodigy were the way to get online. Everything will exist within a precious few walled gardens, controlled by even fewer massive corporations.
This is not how it was supposed to go down. The web evolved basically from the ground up on principles of decentralization, openness and freedom. “Information wants to be free” was the war cry of the early web.
Somewhere along the way however, a few big companies became very good at capturing and controlling significant portions of what happens online, and now we find ourselves on the cusp of a very different future than what many saw as the full potential of the web.
I’d like to talk to you about one possible alternative, based on my experiences thus far on the internet.
I’m 36 years old, and I started using the web heavily in about 1996, so about 21 years ago. That means 60% of my life I’ve been online.
During that time, a lot of my experiences, interactions, and created memories have happened purely digitally. Along the way, I got to wondering how many of those memories I could guarantee ongoing access to. How many of my own digital memories did I even control?
Back when you took a photo and had it developed; remember hardcopy photos? you had that photo effectively forever. You put it in an album, or in a box, or on the wall, and you pulled it out whenever you wanted to show someone else, or to look at it and relive that particular memory.
In the mid to late 2000s, the equivalent was a service called Flickr. Now, I’m no photographer, but I like to capture my own memories. I uploaded over 4,500 photos to Flickr between 2002 and 2014. Flickr went through some tough times in amongst a Yahoo acquisition and re-org, and I realized two things. One; I didn’t want to pay $25 a year for a premium membership any more, and Two; if I stopped paying, I would lose access to my own photos. My own memories.
That was the first light-bulb moment for me. The second came a few years later, when Twitter was exploding in popularity, and was having trouble scaling their systems. They decided they would impose a limit, which meant that you would no longer be able to access more than 3,200 of your own tweets.
I realized that by recording so many of my thoughts on someone else’s service, I had given up a piece of myself to them. If I had put my thoughts on a system I controlled, then I could choose if they were online or not. I could decide who had access to them. Since I had published them on Twitter though, those decisions were no longer mine.
The final lightbulb was Delicious. Delicious is, still, amazingly, an online bookmarking or link saving service. I used it for years to store and annotate hundreds of links so that I could find and reference them later. Then they got bought. And shut down abruptly. And bought again. Their new owners brought the service back, and started significantly changing how it worked.
I wanted my bookmarks and annotations, but they were tied up in this unstable, changing-for-the-worse web service, with a very hazy future.
Between Flickr, Twitter and Delicious, I realized that if I wanted to retain access to my own memories, to the things I was creating online, then I had to act pretty quickly, since I was coming up on my 3,200 tweet limit, I’d soon have to renew that premium Flickr account, and who knows how long until Delicious disappeared for good. I had to get a copy of everything and put it somewhere that I controlled.
I had worked a lot with WordPress at this point, and I knew that tweets, links, and photos were perfect candidates to be published on a WordPress site. It even had specific concepts for them all, called Post Formats. I set out to build the tools that would allow me to reclaim all of my own content from other web services, and archive it to my own WordPress. A service that, thanks to its open-source DNA, I had complete control over, and knew was not driven by any specific, nefarious, commercial interests.
So I got to work, and wrote a plugin called Keyring that gave me the basic ability to connect WordPress to other online services. Then I wrote the specific systems I needed to import my content from around the web, and since then have expanded that to reclaim…
14,000 tweets from Twitter
Those 4,500 photos on Flickr
6,000 check-ins on Foursquare/Swarm
1,700 bookmarks from Delicious
The full text of 1,300 articles read via Instapaper
700 Instagram pictures
The details of 200 trips from a service called TripIt
all of that going back to around 2002
Today I have 29,000 entries in my personal archive, compiled from all those sources, plus around 300 of my own full length blog posts.
Now this is perhaps an interesting personal story, but you’re probably wondering how it’s relevant to businesses, or for that matter, anyone who’s doing more with their life than posting photos and tweets on the internet.
It’s relevant because today, we see advertising campaigns that end with a facebook.com address. It’s relevant because small businesses are relying on their Yelp ratings to attract customers. It’s relevant because without Google Local listings appearing in a mobile search, you don’t exist to your own neighbors.
I believe that some of the same concepts of data ownership and control we’ve talked about, are critical to the healthy future of the web, and the world it interconnects, whether we fully appreciate it yet or not.
The problem with the way we’re headed today is that while Facebook, Google, and Yelp are all global, multi-billion dollar companies that are probably not going anywhere soon, they’re providing their services 100% on their terms. Not yours. You are welcome in their playground only so long as you agree to, and abide by, their terms of service. You often don’t even own or retain full rights to your own data when playing in their playground. You’re renting a storefront in their marketplace. You neither own, nor control, your own online existence.
So if we use some of the technologies and approaches I mentioned earlier, what does that look like for a business today? How can we shift some of that control back to you? Give you the freedom to make your own choices?
Well, you probably already have a Facebook Page set up for your business, so maybe you can connect that to a fresh WordPress. It pulls down your visual branding, opening hours, and contact details. It uses that as the seed data to automatically set up a simple website.
Now you’ve got your website, powered by WordPress, which you fully control. You bought your own domain, so the site is at a web address that you actually own. No one can decide you’ve violated some terms of service and kick you off. No one can take your address away from you. You can customize the design of your site, use a different theme, add functionality with plugins, the sky’s the limit. No one gets to dictate what you can or cannot do; it’s like instead of renting, you bought the whole building where you’re going to set up your store, and you don’t need a construction permit to fit it out how you’d like.
But the magic part is that when you imported your data, you created a 2-way connection between your WordPress and Facebook. Now you update your opening hours in WordPress, and your Facebook Page is also immediately up to date.
You add new connections to Google and Yelp, so we automatically create listings on those services, and keep them up to date with your site as well. We download copies of your reviews automatically, send you push notifications via the WordPress app when there are things that need your attention, and allow you to interact with the community you’re building via your Instagram account, right there in WordPress. Your content is pulled into your own site where you can exercise complete creative control over it, and use it in ways that Instagram would never dream of, nor necessarily allow.
WordPress becomes the hub of all your online activity. A central place that you control, where you both aggregate and interact with your digital presence. You choose whether it’s presented as a slimmed down, utilitarian tool, mainly for providing information to other services, or a complete and beautiful destination, where potential customers can find whatever they need to know about your business, transact with you directly, or interact with the community you’re building.
Today, you choose the direction you’ll go in. You can choose to rent space on one platform or another, or you can choose to actually own your own piece of the web. You can choose to participate in these business-critical, closed platforms, but to also build your own separate online presence, which you control and define. You choose whether to tie your future to a walled garden which dictates the rules of engagement to you, or to invest in an open platform, which allows you to grow and change but still be a part of the fabric of the web. You have the freedom to choose.
I’ve just released version 1.8 of both Keyring, and the Keyring Social Importers. This version includes a new service file, and an accompanying importer, which allows you to import content from a Jetpack-powered WordPress site, using the WordPress.com REST API. That means any site hosted on WordPress.com, or any self-hosted site with the Jetpack plugin installed. There are also a few key fixes for the Twitter and LinkedIn services/importers, so it’s a nice update.
The new importer will pull across the entire content of posts, including tags. Similar to the Instapaper importer, it attempts to avoid duplicate content issues by marking pages as noindex if they come from imported content.
This is another piece of the puzzle required for me to create a complete archive of my digital footprints over on Dented Reality, now that I’m blogging here. This post should be imported over there automatically within an hour.
Note that currently the importer doesn’t sideload any media items (will add that soon) or support geo data (again, I’ll add that when I get a chance).
I’ve been trying to make small improvements to the Keyring Social Importers package (and People & Places) that I maintain, and have made a number of them over the last few weeks. Here are some details of recent updates which you may have missed:
Added a Nest Camera service and importer. Including recent updates, it will download a snapshot from the specified camera(s) during the hour indicated, auto-tag it using the location of the camera, and also associate it with a Place if People & Places is co-installed.
You can see most of them in action on my site, Dented Reality, which uses them to aggregate most of my online social activity. The People & Places data is not directly exposed yet, but you can see it in the REST API output.
Places support added to TripIt importer.Added a Nest Camera importer.Current list of data reprocessors.
These updates make it so that the Twitter, Foursquare and Instagram importers are now dynamically identifying and indexing People and Places, and marking them with a taxonomy within WordPress. I’ve also added a new system for “reprocessing” old posts which Keyring imported, so that you can go back and perform some function on those posts without having to import them again. You’ll find reprocessing tools under Tools > Import > Reprocess Keyring Data.
Reprocessing works by using the locally-stored copy of import data that is saved during the initial import of everything. The system is fully hookable, so you can add other reprocessing routines in via plugin. The core file comes bundled with one that attempts to address an old JSON-data-escaping issue, and I’ve added extensions to the importers listed above which allow you to go back and reprocess your posts for People/Places.
If you’re going to use them, I suggest you run the first one first, then you can run the others in any order you like. Doing the first one first will just make sure that as much of your data as possible is processable.
It’s worth mentioning that if you use these reprocessors, they can take a while (especially if you have a lot of data already), and that they will likely create a lot of new data (in the form of People and Place terms being created in their respective taxonomies). After running all of them over all of my data, I have almost 1,800 People and just over 3,000 Places in my database.
The other tool added in this upgrade is the ability to merge terms, which becomes important with all of this data.
When the importers are dynamically adding People and Places, they only match based on known identifiers. This means that you’re likely to end up with duplicate entries, especially if you’re processing multiple services (e.g. Foursquare and Instagram). Using the merge tool, you can browse through your entries and select 2 or more, then use the Bulk Actions drop-down to select “Merge” and hit “Apply”. Terms will be merged together as intelligently as possible, which basically means that the shortest slug of the group will be kept, and the longest strings for any conflicting fields will be kept. You can of course edit the resulting composite term afterwards and tweak things as you see fit. If you’re looking for a shortcut to identify duplicate entries, try searching for “-2”, which will give you a list of duplicates, then you’ll need to search for something that will bring up each of the dupes, select, merge, repeat. It’s a little bit tedious, but you’ll only need to do it once for each duplicate, and all future imports should match against the composite entry.
Oh, and one last thing — I threw in a quick map on the details page for Places, which provides a nice quick, visual confirmation that it’s the correct location. For now it’s using a very basic OpenStreetMap example, but I might switch it out to Leaflet at some point, which is pretty nice.
Over the years, I’ve been working on a system to aggregate data that I publish to other social networks/sites back into my control, on my own WordPress install. Thus far, that has resulted in the creation of Keyring (plugin) to provide an abstracted interface to all of the web services I’m interested in, Keyring Social Importers (plugin) to do the basics of importing the data from different places, and Homeroom (theme) to display it all. Today, I’ve been working on a system that will detect people who are mentioned in an interaction, and link them across posts using a custom taxonomy. It does the same for physical locations, so I’ve called it People & Places.
Essentially, this plugin is just a pair of custom taxonomies, with some specific ways of referring to things. Pretty basic. It gets more interesting though when you update Keyring Social Importers to the trunk version, which will now work in tandem with People & Places to link everything up. I wouldn’t recommend it on a production site just yet — there’s a lot of rough edges still.
When KSI is pulling in content from each service (currently looking at Twitter, Instagram and Foursquare), there’s a new block of code that makes sure People & Places is available, and then looks for certain pieces of data. If it finds them, it bundles up the details, and passes that along in the import process. When posts are actually inserted, it will attempt to link up that post to the People/Places it found. If the People already exist, then they’ll just be linked, in the same way tags work. If they don’t exist yet, then a new Person entry will be created, and that will be used.
I plan to add in a basic term-merging function, so that you can manually (maybe automatically?) identify “duplicate people” across different networks, and intelligently merge their entires (re-linking any posts involved), so that you build up a single, combined view of your interactions with a particular person. I envisage some interesting possibilities with the archive pages for these taxonomies, and that over time it will build a really interesting dataset of your interactions, the places you physically go, etc.
I’ll probably still move the code around a bit, and there are definitely some bugs around duplicates and handling things across different networks, but it seems to be working so far. This is also probably the time to figure out a decent way to allow re-processing of imported data from the raw copy that the importers save in postmeta. Installing this new code will start gathering data on new imported entries, but won’t go back and do the same on all the posts you’ve already got. Rather than deleting all that data and re-importing/processing everything, I’d like to have a simple way to re-process the raw data that’s already stored locally.
Yesterday, I released version 1.5 of Keyring, and version 1.4 of the Keyring Social Importers bundle for WordPress. This update moves the Social Importers away from using a postmeta value (keyring_service) and introduces a new taxonomy that keeps track of where posts were imported from. It’s optimized towards management within wp-admin, but you can also use it for front-end queries of your posts. The update for Keyring introduces a new service file for Moves, and fixes a bug in the OAuth2 base service.
The new taxonomy for the Importers is called keyring_services on the backend, and is labeled “Imported From” in the admin UI. It will auto-create itself based on all of the importers installed. You’ll see it within wp-admin under the Posts menu, and will be listed on the “All Posts” listing as well:
Clicking the name of a service under the “Imported From” heading will filter the posts list by that service (e.g. Twitter). The main reason that the taxonomy is exposed through the admin UI is so that you can tweak the slugs if you’d like to. I noticed that on my install, I’d already used things like ‘twitter’ and ‘foursquare’ as tags, and so they had claimed the namespace for that slug. WordPress’ shared terms are annoying like that :). So, if you’d like to use the slugs of source services in URLs, you might want to rename them:
Go to Posts → Tags
Search for and rename the slug for each of the services (e.g ‘twitter’, ‘foursquare’, ‘flickr’). Name the slugs something like ‘twitter-3’
Go to Posts → Imported From and rename the slugs for each service to the “clean” version (without a ‘-2’).
Optionally go back to Posts → Tags and rename those tags again back to the -2 versions.
As part of this change, you’ll want to update any previous posts that you imported to using the new taxonomy. I’ve included a quick and dirty script to do this. It’s called migrate-keyring-postmeta-to-taxonomy.php and can be found in the root of the plugin. To use it, you need to move it to the root of your WordPress install, and then you can just access it through your browser. It’s likely that it’ll run out of memory or time out, but it’s written in a way that you can just run it over and over again until it finishes cleanly. On my server, once it was finished and produced no output, Chrome decided to display a “friendly” error message instead of anything useful. Once that’s done, your existing posts should all be converted over to using the new taxonomy, and there should be no more postmeta entries for keyring_service.
If you’re doing a clean import, I recommend doing it without auto-import enabled, and then once you’ve fully imported everything, enable auto-import and let it run from there.
