The API, the whole API and nothing but the API

When programming against a remote service, do you like to be provided with a library (or service stub) or do you prefer “the API, the whole API, nothing but the API”?

A dedicated library (assuming it is compatible with your programming language of choice) is the simplest way to get invocations flowing. On the other hand, if you expect your client to last longer than one night of tinkering then you’re usually well-advised to resist making use of such a library in your code. Save yourself license issues, support issues, packaging issues and lifecycle issues. Also, decide for yourself what the right interaction model with the remote API is for your app.

One of the key motivations of SOAP was to prevent having to get stubs from the service provider. That remains an implicit design goals of the recent HTTP APIs (often called “RESTful”). You should be able to call the API directly from your application. If you use a library, e.g. an authentication library, it’s a third party library, not one provided by the service provider you are trying to connect to.

So are provider-provided (!) libraries always bad? Not necessarily, they can be a good learning/testing tool. Just don’t try to actually embed them in your app. Use them to generate queries on the wire that you can learn from. In that context, a nice feature of these libraries is the ability to write out the exact message that they put on the wire so you don’t have to intercept it yourself (especially if messages are encrypted on the wire). Even better if you can see the library code, but even as a black box they are a pretty useful way to clarify the more obscure parts of the API.

A few closing comments:

– In a way, this usage pattern is similar to a tool like the WLST Recorder in the WebLogic Administration Console. You perform the actions using the familiar environment of the Console, and you get back a set of WLST commands as a starting point for writing your script. When you execute your script, there is no functional dependency on the recorder, it’s a WLST script like any other.

– While we’re talking about downloadable libraries that are primarily used as a learning/testing tool, a test endpoint for the API would be nice too (either as part of the library or as a hosted service at a well-known URL). In the case of most social networks, you can create a dummy account for testing; but some other services can’t be tested in a way that is as harmless and inexpensive.

– This question of provider-supplied libraries is one of the reasons why I lament the use of the term “API” as it is currently prevalent. Call me old-fashioned, but to me the “API” is the programmatic interface (e.g. the Java interface) presented by the library. The on-the-wire contract is, in my world, called a service contract or a protocol. As in, the Twitter protocol, or the Amazon EC2 protocol, etc… But then again, I was also the last one to accept to use the stupid term of “Cloud Computing” instead of “Utility Computing”. Twitter conversations don’t offer the luxury of articulating such reticence so I’ve given up and now use “Cloud Computing” and “API” in the prevalent way.

[UPDATE: How timely! Seconds after publishing this entry I noticed a new trackback on a previous entry on this blog (Cloud APIs are like military parades). The trackback is an article from ProgrammableWeb, asking the exact same question I am addressing here: Should Cloud APIs Focus on Client Libraries More Than Endpoints?]

9 Comments

Filed under API, Automation, Everything, Implementation, Protocols, REST, SOAP

9 Responses to The API, the whole API and nothing but the API

  1. Anyone with some level of distributed computing experience, in particular CORBA (Visigenic => Borland => Inprise => Borland => EOL), would have called out some benefits of client libraries including caching, surface adaption, end-to-end intelligent qos coordination/control, as well as standardized (and hopefully efficient) language/runtime mappings of request/response data. If your client libraries don’t provide this don’t assume you will not eventually (re)build this yourself.

    Every new lightweight RPC eventually ends up looking like CORBA. There are some valid reasons for this.

    I think we are going to see much more control data (in particular metering, chargeback, qos) related being piggybacked on client and server end point exchanges very similar to how networking works but higher in the application stack and contextual (user, service interaction, transaction,….). This is going to be used much more in all those Javascript/Ajax front-ends to throttle/police traffic further upstream. Twitter is already using this though in a somewhat hackery (as they would) form in their new front end.

    That said I personally like the raw HTTP interface (protocol/surface/contract) but that is because I think I can do much better than those building such libraries, ;-).

  2. Pingback: The API, the whole API and nothing but the API

  3. One of my pet peeves about client libraries especially the Amazon AWS SDK is the pulling in of other library dependencies in particular logging libs which are an abomination in modern software/systems development.

    http://williamlouth.wordpress.com/2010/11/24/everything-you-ever-needed-to-know-about-logging/

  4. While it is important to have an understanding of raw HTTP interface, from what I have seen many developers (sometimes includes myself) like to use thin client wrappers (I say thin client wrappers because there are cases where the libraries extend beyond wrapping the API and aim to provide lot more – a separate data model, portability across similar but different vendor APIs, caching etc).

    Assuming that licensing issues are taken care and the library is fairly stable, there are lot of advantages to thin client wrappers – Error handling, blocking/non-blocking IO, Esoteric http client library issues, OAuth, serializing/deserializing requests/responses into language specific types

    As for understanding raw HTTP interface, one really donot need to depend on (in most scenarios) using provider-provided libraries to see what is going on the wire. Tools like apigee API console (http://www.apigee.com) help one to understand the REST interface fairly quickly

  5. A big assumption you are making is that the interface exposed by the library is just a stub for the interface exposed by the service.

    What if it’s not?

    What if the interface exposed by the library is tuned to the capabilities and needs of the client (as opposed to the service) so that it was very stable and only needed to change if the client changed (and didn’t require an update if the service changed)?
    (http://linkednotbound.net/2010/06/09/hypermedia-is-the-clients-lens/)

    What if the library API used an inversion of control so that the client emitted events to and received commands from the library, thereby removing behavioral coupling to the library and any service sitting behind it?
    (http://bill-poole.blogspot.com/2008/04/avoid-command-messages.html)

    What if the library could be fed a URI for the service that it should interact with? What if that URI could be used to download an event filter that told the library what to do with the events it received from the client and how to map those to the requests service’s HTTP interface or to local state changes or to invocation of commands back to the client? (http://linkednotbound.net/2011/01/31/hypermedia-is-an-event-filter/)

    You’d get a library that acted like a “browser” rather than a “stub”. You’d get a library that could be dynamically adapted to many services and was tolerant of service evolution. Yes, you’d get a library that was RESTful client and that’s just not the same thing as a “dedicated library” or “stub”.

    Now, a browser is a whole lot harder to write than a stub, so if a browser library was already available (and it’s client-facing interface met your needs — which is a much bigger issue here than with stubs) you’d be better off just using it.

  6. This is not a sustainable position because what happens in the real world is that someone has to make a layer that talks to the API in whatever language they are using, otherwise low level API calls get sprinkled throughout the code and lots of code gets repeated with cut-and-paste bug propagation. They then share this layer with someone else who also wants to talk to the API, and then they post it to github or whatever, and other people use it, and now you have your service being used by everyone via an API layer written by someone else, who doesn’t fully understand your API, so you have a distorted model, poor error handling etc.

    A more sustainable approach is for the API developer to provide a service access library that exposes a clean abstraction for whatever language it is developed for. It doesn’t have to do caching (which may be best done using a different service anyway) and should have minimal sideways dependencies. At Netflix we call these SALs. We also build another layer on top called an Extended Service Library, that does caching, integrates multiple SALs and exposes an optimized object model to the end application. The ESL is also shared code, but is usually built by the consumers of the service.

  7. Adrian has a point – if the API producer provides the client libraries, supports them and keeps them up to date, there should be no drawbacks in using them in your client. This is not typical however…

  8. Just in case it was not obvious but my reference to “caching” related not just to the actual content but to connection management, request/response buffers,…. pretty standard in high performance middleware systems though granted rarely consider in high latency cloud apps and service interactions.

  9. In my experience, most engineers would agree with your opinion that remote program interfaces, exposed directly, without client libraries, should be more accurately called application protocols, not APIs. Marketing folks, on the other hand, always insist on calling them APIs. Why? The term API is very familiar to developers and it is more closely associated with ease of use. People tend to be less familar with the term protocol and often associate them with with complex, low-level trasport protocols. In our industry, which spends twice as much on marketing than development, it is no surprise that the term API sticks. Engineers however, would be much better served thinking in terms of application protocols instead of APIs.