Back in 2007, there weren’t enough deployed APIs to make general statements about them. If you own RESTful Web Services, take a look at Appendix A, where I list all the read-write APIs I can find that pass a minimum quality threshold. It’s a joke! Six years later, we have thousands of API designs to look at. What’s more, we have mature designs, designs that have been in production for the better part of a decade. It’s now possible to look at a whole lot of APIs and talk about the field as a whole. Unfortunately, when I look at the field it’s hard to avoid the conclusion that the design frontier kind of stopped moving forward around 2008. Getting things moving again is the goal of my new book, RESTful Web APIs. This is a a near-complete rewrite of RESTful Web Services to deal with today’s problems, co-written with Mike Amundsen, author of Building Hypermedia APIs with HTML5 and Node. Today’s problems are pretty big, but I’m cautiously optimistic, because this has actually happened before. API design was stuck in the year 2000 for about five years, thanks to the legacy of XML-RPC, which grew into SOAP and the heavyweight WS-* protocols. Between 2005 and 2008 there was a rapid shift away from SOAP and towards “REST” (however that was understood), and then we settled in to a new status quo which persists today. The new status quo is definitely preferable to what we had in 2005, but we could be doing a lot better. Here are the three main problems I’ve noticed.
1. APIs Can’t Change
Web sites undergo redesigns all the time. The users complain and eventually get used to the new site. This is an underappreciated feat of engineering that’s only possible because every single piece of a web site—HTML, binary files, and Javascript code—is deployed from the server at runtime. When a web site changes, it’s automatically redeployed to its users’ web browsers. The browsers (if not the users) immediately forget about the old site and ‘execute’ the new one. But redeploying an API is a slow and thankless feat of developer relations. That’s because the server is only in charge of one piece of the API—the data. The rest is distributed across dozens or hundreds of pieces of client-side software, each custom-written by a developer based on their interpretation of human-readable documentation. It’s no surprise that API changes are painful: all that client-side software has to be manually “redeployed”. APIs put out by big companies have sunset periods measured in years and changeovers marked by enormous struggle—witness Twitter’s 2010 “OAuthpocalypse.” APIs put out by smaller companies tend not to change at all after their initial release. No wonder Pinterest is taking so long to release their API. Anything they release, they’ll probably have to support forever. They need a design that will last. Since experiments are so expensive, it’s no surprise that the field as a whole is extremely conservative. It doesn’t have to be this way. Redeploying an API will never be as easy as redeploying a web site, but this isn’t an all-or-nothing proposition. There are techniques—notably, the use of hypermedia—that give API clients a limited ability to adapt to server-side changes. These techniques all involve moving bits of the API out of static human-readable documentation and into machine-readable documents deployed from the server at runtime. Unfortunately, we’ve been avoiding those techniques for the past six years, partly because of the next problem.
2. JSON Can’t Represent Links
This problem became visible shortly after RESTful Web Services was published, as JSON overtook XML as the preferred medium for API data. The problem is that the application/json media type has no built-in support for hyperlinks. Imagine if HTML didn’t support links. Web pages would need human-readable instructions all over the place saying things like “To get to the detail view for this product, append /detail to the current URL.” That sounds insane, but it’s how today’s APIs work. They’re held together with human-readable documentation explaining how to construct URLs for various scenarios. For an API that serves JSON documents, these human-readable rules are the only way to connect one document to related documents—a detail view, a product image, or the next page of a list. JSON provides no reliable way to distinguish between a URL and a string that happens to look like a URL. You can get around this with hacks. When developing the Launchpad API for Canonical, I set a rule that a value in a JSON object whose name ended in “_link” should be treated as a link. In other words, I replaced a bunch of human-readable rules with a single, higher-level rule. But ad hoc rules like this have no real force, and every API will come up with different rules. It’s been a long time coming, but this problem is finally being solved. Within the past couple of years, data formats like Siren, HAL, and JSON-LD have come on the scene, combining JSON data structures with real hyperlink support.
3. Everyone Reinvents Basic Concepts
The thousands of APIs created over the past seven years are superficially similar but mutually incompatible. Everyone has a slightly different view of basic real-world concepts like “person” and “event”. This lack of agreement makes it impossible to create client-side software that can be reused between APIs. It’s not hard to find two APIs that do exactly the same thing but have nothing in common except the application/json media type. To some extent this is a political problem. Big companies that compete with each other don’t want API interoperability. But that only explains why the big companies don’t lead in this space. Without that leadership, it comes down to intertia. It’s easier to publish your own internal model of “person” or “event” than to try and use a standardized vocabulary. Without a shared vocabulary, the metaphors we like to use for our industry—”API economy” or “API marketplace”—are delusions of grandeur. A marketplace is for commodities. We’re a bunch of artisans sitting around fishing for customers. These shared vocabularies do exist. Some have been around for years, like microformats, the IANA registry of link relations, and FOAF. Some are relatively new, like Activity Streams and schema.org’s microdata schemas. But it’s a rare API that uses any of these standards. Why bother? If everyone used them, adopting them would give your users access to useful client-side libraries. But the big companies don’t use them, so the small companies don’t either, so there’s no client-side support. It’s a collective action problem. There’s a technical issue here as well: these standards tend to put irrelevant constraints on the data format you can use. Microformats and microdata schemas were designed for the World Wide Web, and they can only be applied to HTML documents. An Activity Streams document is a application/json document in a specific format. If you’re using Siren or HAL, you can’t use Activity Streams’s concepts of “purchase” or “flag-as-inappropriate”. You must come up with your own versions of the same concepts, which makes the problem a little bit worse.
Getting Unstuck
Back in 2005, getting API design unstuck from the WS-* rut was a matter of convincing developers to stop hitting themselves in the head with a hammer. The benefits were obvious. Now it’s more like convincing developers to adopt a healthy diet to avoid a heart attack down the road. The past few years have seen some encouraging signs of progress. Formats like Siren, HAL, and JSON-LD make it possible to create a modern-looking API that can change over time. We give all these formats space in RESTful Web APIs, along with dozens of other formats you might never have heard of. We can’t solve the political side of the vocabulary problem with a book. But Mike and I are trying to address the technical side with a new data format called ALPS (Application-Level Protocol Semantics). ALPS decouples the semantics of standardized vocabularies—that is, the human-readable meanings of magical strings like “family-name” and “flag-as-inappropriate”—from the data formats those vocabularies require. Here’s a simple example. We’ve automatically ported the schema.org microdata item Event to ALPS: here’s the ALPS document. The schema.org original and the ALPS port both define the real-world concept of “event” in terms of other real-world concepts: “attendee”, “location”, “startDate”, and so on. The original microdata item can only be applied to an HTML 5 document, because no other data format supports microdata. But the ALPS port can be applied to an HTML 5 document, an HTML 4 document (using the rules established by microformats), a Siren document, a HAL document, an Atom feed, and even the no-frills JSON or XML documents used in today’s APIs. The ALPS document acts as a profile of the document containing the data. It sheds light on the meaning of the data without changing the rules for processing the document. Mike and I are working on porting microformats and other standardized models to ALPS. We’re working with the creators of data formats like Siren to come up with good rules for applying ALPS profiles to their documents. We can’t solve the collective-action problem on our own, but we can make it easy to use standardized names for everyday concepts instead of coming up with your own names.
Conclusion
We have the tools to make progress on the biggest problems in API design. Data formats like HAL and Siren make it possible to create a JSON-based API capable of (limited) change over time. The ALPS format, or some future competitor, will let us stop coming up with ten or twenty different names for the same concepts. It’s frustrating that progress has been so slow, but nobody promised an instant fix. Roy Fielding likes to say that REST is “software design on a scale of decades.” It’s been almost fifteen years since the last major revision of HTTP. And if you look at our progress on that time scale, it doesn’t look so bad. Fifteen years ago, we had XML-RPC: a proof-of-concept protocol for software integration over HTTP. A few years later, we had SOAP, and a small number of brittle, over-complicated, hard-to-use APIs that helped a few people get work done. Today, we have “REST”: thousands of inflexible, incompatible, repetitive APIs that help a lot of people get work done. This is what progress looks like. It’s slow. Bad designs have stuck around longer than they should, because one of their bad features is an inability to change. New designs will have some ability to automatically redeploy as they change. You won’t need to wait for an API provider to bump the version number, or for a competitor to come along with a more sophisticated offering. That’s good news, not only for progress today, but for the rate of progress over the next fifteen years.