REST in practice for IT and Cloud management (part 2: configuration management)

What benefits does REST provide for configuration management (in traditional data centers and in Clouds)?

Part 1 of the “REST in practice for IT and Cloud management” investigation looked at Cloud APIs from leading IaaS providers. It examined how RESTful they are and what concrete benefits derive from their RESTfulness. In part 2 we will now look at the configuration management domain. Even though it’s less trendy, it is just as useful, if not more, in understanding the practical value of REST for IT management. Plus, as long as Cloud deployments are mainly of the IaaS kind, you are still left with the problem of managing the configuration of everything that runs of top the virtual machines (OS, middleware, DB, applications…). Or, if you are a glass-half-full person, here is another way to look at it: the great thing about IaaS (and host virtualization in general) is that you can choose to keep your existing infrastructure, applications and management tools (including configuration management) largely unchanged.

At first blush, REST is ideally suited to configuration management.

The RESTful Cloud APIs have no problem retrieving resource descriptions, but they seem somewhat hesitant in the way they deal with resource-specific actions. Tim Bray described one of the challenges in his well-considered Slow REST post. And indeed, applying REST to these “do something that may take some time and not result exactly in what was requested” scenarios is a lot less straightforward than when you’re just doing document/data retrieval. In contrast you’d think that applying REST to the task of retrieving configuration data from a CMDB or other configuration store would be a no-brainer. Especially in the IT management world, where we already have explicit resource models and a rich set of relationships defined. Let’s give each resource a URI that responds to HTTP GET requests, let’s turn the associations into hyperlinks in the resource presentation, let’s mint a MIME type to represent this format and we are out of the office in time for a 4:00PM tennis game when all the courts are available (hopefully our tennis partners are as bright as us and can get out early too). This “work smarter not harder” approach would allow us to present this list of benefits in our weekly progress report:

-1- A URI-based scheme makes the protocol independent of the resource topology, unlike today’s data stores that usually struggle to represent relationships between stores.

-2- It is simpler to code against than CIM-over-HTTP or WS-Management. It is cross-platform, unlike WMI or JMX.

-3- It makes it trivial to browse the configuration data from a Web browser (the resources themselves could provide an HTML representation based on content-type negotiation, or a simple transformation could generate it for the Web browser).

-4- You get REST-induced caching and scalability.

In the shower after the tennis game, it becomes apparent that benefit #4 is largely irrelevant for IT management use cases. That the browser in #3 would not be all that useful beyond simple use cases. That #2 is good for karma but developers will demand a library that hides this benefit anyway. And that the boss is going to say that he doesn’t care about #1either because his product is “the single source of truth” so it needs to import from the other configuration store, not reference them.

Even if we ignore the boss (once again) it only  leaves #1 as a practical benefit. Surprise, that’s also the aspect that came out on top of the analysis in part 1 (see “the API doesn’t constrain the design of the URI space” highlight, reinforced by Mark’s excellent comment on the role of hypertext). Clearly, there is something useful for IT management in this “hypermedia” thing. This will largely be the topic of part 3.

There are also quite a few things that this RESTification of the configuration management store doesn’t solve:

-1- The ability to query: “show me all the WebLogic instances that run on a Windows host and don’t have patch xyz applied”. You don’t have much of a CMDB if you can’t answer this. For an analogy, remember (or imagine) a pre-1995 Web with no search engine, where you can only navigate by starting from your browser home page and clicking through static links step by step, or through bookmarks.

-2- The ability to retrieve the configuration change history and to compare configurations across resources (or to a reference configuration).

This is not to say that these two features cannot be built on top of a RESTful IT resource model. Just that they are the real meat of configuration management (rather than a simple resource-by-resource configuration browser) and that your brilliant re-architecture hasn’t really helped in addressing them. Does a RESTful foundation make these features harder to build? Not necessarily, but there are some tricky aspects to take care of:

-1- In hypermedia systems, the links are usually part of the resource representation, not resources of their own. In IT management, relationships/associations can have their own lifecycle and configuration properties.

-2- Be careful that you can really maintain the address of a resource. It’s one thing to make sure that a UUID gets maintained as a resource configuration changes, it’s another to ensure that a dereferenceable URI remains unchanged. For example, the admin server of a cluster may move over time from one node to another.

More fundamentally, the ability to deal with multiple resources at the same time and/or to use the model at different levels of granularity is often a challenge. Either you make your protocol more complex to account for this or your pollute your resource model (with a bunch of arbitrary “groups”, implicit or explicit).

We saw this in the Cloud APIs too. It typically goes something like this: you can address an individual server (called “foo”) by sending requests to http://Cloudprovider.com/server/foo. Drop the “foo” part of the URL and now you can address all the servers, for example to retrieve their configuration or possibly to reboot them. This gives me a way of dealing with multiple resources at time, but only along the lines pre-defined by the API. What if I want to deal only with the servers that host nodes of a given cluster. Sorry, not possible. What if the servers have different hosts in their URIs (remember, “the API doesn’t constrain the design of the URI space”)? Oops.

WS-Management, in the SOAP world, takes this one step further with Selectors, through which you can embed some kind of query, the result of which is what you are addressing in your message. Or, if all you want to do is GET, you can model you entire datacenter as one giant virtual XML doc (a document which is never assembled in practice) and use WSRF/WSDM’s “QueryExpression” or WS-Management’s “FragmentTransfer” to the same effect. BTW, I have issues with the details of how these mechanisms work (and I have described an alternative under the motto “if you are going to suffer with WS-Addressing, at least get some value out of it”).

These are all non-RESTful atrocities to a RESTafarian, but in my mind the Cloud REST API reviewed in part 1 have open Pandora’s box by allowing less-qualified URIs to address all instances of a class. I expect you’ll soon see more precise query parameters in these URIs and they’ll look a lot like WS-Management Selectors (e.g. http://Cloudprovider.com/server?OS=Linux&CPUType=X86). Want to take bets about when a Cloud API URI format with an embedded regex first arrives?

When you need this, my gut feeling is that you are better off not worrying too much about trying to look RESTful. There is no shame to using an RPC pattern in the right circumstances. Don’t be the stupid skier who ends up crashing in a tree because he is just too cool for the using snowplow position.

One of the most common reasons to deal with multiple resources together is to run queries such as the “show me all the WebLogic instances that run on a Windows host and don’t have patch xyz applied” example above. Such a query mechanism recently became a DMTF standard, it’s called CMDBf. It is SOAP-based and doesn’t attempt to have anything to do with REST. Not that it didn’t cross the mind of a bunch of people, lead by Michael Coté when CMDBf first emerged (read the comments too). But as James Governor rightly predicted in the first comment, Coté heard “dick” from us on this (I represented HP in CMDBf and ended up being an editor of the specification, focusing on the “query” part). I don’t remember reading the entry back then but I must have since I have been a long time Coté fan. I must have dismissed the idea so quickly that it didn’t even register with my memory. Well, it’s 2009 now, CMDBf v1 is a DMTF standard and guess what? I, and many other SOAP-the-world-till-it-shines alumni, are looking a lot more seriously into what’s in this REST thing (thus this series of posts for me). BTW in this piece Coté also correctly predicted that CMDBf would be “more about CMDB interoperation than federation” but that didn’t take as much foresight (it was pretty obvious to me from the start).

Frankly I am still not sure that there is much benefit from REST in what CMDBf does, which is mostly a query interface. Yes the CMDBf query and its response go over SOAP. Yes in this case SOAP is mostly a useless wrapper since none of the implementations will likely support any WS-* SOAP header (other than paying the WS-Addressing tax). Sure we could remove it and send plain XML over HTTP. Or replace the SOAP wrapper with an Atom wrapper. Would it be anymore RESTful? Not one bit.

And I don’t see how to make it more RESTful. There are plenty of things in the periphery the query operation that can be made RESTful, along the lines of what I described above. REST could make the discovery/reconciliation tasks of the CMDB more efficient. The CMDBf query result format could be improved so that from the returned elements I can navigate my way among resources by following hyperlinks. But the query operation itself looks fundamentally RPCish to me, just like my interaction with the Google search page is really an RPC call that happens to return a Web page full of hyperlinks. In a way, this query (whether Google or CMDBf) can at best be the transition point from RPC to REST. It can return results that open a world of RESTful requests to you, but the query invocation itself is not RESTful. And that’s OK.

In part 3 (now available), I will try to synthesize the lessons from the Cloud APIs (part 1) and configuration management (this post) and extract specific guidance to get the best of what REST has to offer in future IT management protocols. Just so you can plan ahead, in part 4 I will reform the US health care system and in part 5 I will provide a practical roadmap for global nuclear disarmament. Suggestions for part 6 are accepted.

11 Comments

Filed under API, Application Mgmt, Automation, Cloud Computing, CMDB, CMDB Federation, CMDBf, DMTF, Everything, IT Systems Mgmt, Mashup, Mgmt integration, Modeling, REST, SOAP, SOAP header, Specs, Standards, Utility computing

11 Responses to REST in practice for IT and Cloud management (part 2: configuration management)

  1. Holy crap! A RESTful (I hate that phrase, we need a better one) interface to CMDBs would top-notch. You should hunt down this fella Glenn Twiggs (who worked on the BMC CMDB at one point) and ask him about it, he’s one of the people I stole that thinking from ;)

  2. Pingback: William Vambenepe’s blog » Blog Archive » REST in practice for IT and Cloud management (part 1: Cloud APIs)

  3. Stu

    Hi William,

    Long response. I’ll comment on this post bottom-up, if you don’t mind.

    “But the query operation itself looks fundamentally RPCish to me, just like my interaction with the Google search page is really an RPC call that happens to return a Web page full of hyperlinks.”

    This is a fundamental disconnect. The Google search page is not an RPC call, at least not in the usual sense of statically binding to a pre-agreed contract that is oriented around a programming platform’s object model and call stack.

    One has to GET the hypertext document that describes how to generate a search URI or how to generate a search media type that’s POSTed to some kind of processing research. The closest analogy to RPC-land would be a CORBA DII, COM Automation, or Reflective RMI call. In SOAP-land, it would be the equivalent of loading the WSDL at runtime, parsing it, figuring out how to call the appropriate operation, and then invoking that operation. Yes, it sounds onerous, but I find that’s usually because you’re trying to fit an RPC mindset onto something that is quite different. Things like DII and Automation were annoying to work with because they were bags on the side of static RPC systems, and not fundamental to their respective architectures.

    In your exploration of hypermedia, you’ve been neglecting the importance of its ability to describe interfaces, like forms, dynamically. When you ask, “Want to take bets about when a Cloud API URI format with an embedded regex first arrives?”, this basically underscores the problem with “Cloud API’s”. What web application has ever succeeded by standardizing URI format?

    One of the benefits of a Google-like search interface is that it’s not a query interface, it’s a uniform interface that’s exposing a search gateway. Architecturally, these place major responsibility differences on the client. One requires deep a priori knowledge of the schema; the other arguably doesn’t.

    “[A query] can return results that open a world of RESTful requests to you, but the query invocation itself is not RESTful. And that’s OK.”

    BTW, I agree with this in principle, that Query services (which I believe aren’t overly RPC-like, but that’s another matter ;), could generate hypermedia docs, even if the query invocation itself isn’t RESTful. That’s OK, and a great example of this in the SemWeb world would be a SPARQL endpoint, something that really does tunnel a query interface through HTTP.

    Earlier in your post, you say that REST doesn’t help solve:
    “-1- The ability to query: “show me all the WebLogic instances that run on a Windows host and don’t have patch xyz applied”. You don’t have much of a CMDB if you can’t answer this. For an analogy, remember (or imagine) a pre-1995 Web with no search engine, where you can only navigate by starting from your browser home page and clicking through static links step by step, or through bookmarks.”

    That’s the same web we have today, the difference is that Google and Microsoft have built multi-billion dollar businesses automating this spidering for us – caching, analyzing, and indexing the results along the way.

    There’s something very powerful about being able to access and analyze information without requiring portions of the “middleware” to understand much of its underlying schema or semantics. That is what enables longevity in software, enabling the segregation of interface definitions, so that usage and formats and semantics to evolve on one plane, while another plane remains stable. That’s arguably the power of a lot of the container-oriented hypermedia formats out there like Atom, Atompub, etc. – they give us the ability to grow incremental infrastructure that will be useful for *decades*, such as the ability to manipulate and present collections of content, without having to reinvent the wheel for every sub-sub-IT-industry segment.

    “In hypermedia systems, the links are usually part of the resource representation, not resources of their own. In IT management, relationships/associations can have their own lifecycle and configuration properties.”

    I’m don’t see how this is any more tricky in a hypermedia system than in any system that manages data. We’ve all seen E-R data designs that don’t enable lifecycle and metadata associated with their relationships when they probably needed it. What resources are made available & how their are linked is up to the underlying servers that support those resources. There are plenty of web content management systems where a link has rich life-cycle and configuration information associated with it.

    “Be careful that you can really maintain the address of a resource. It’s one thing to make sure that a UUID gets maintained as a resource configuration changes, it’s another to ensure that a dereferenceable URI remains unchanged. For example, the admin server of a cluster may move over time from one node to another.”

    Managing long-lived identifiers has long been a problem in data management systems, and certainly RESTful solutions don’t solve this, because arguably it’s inherently a social / governance problem, i.e. maintaining persistent/quality identifiers is fundamentally something that requires economic investment, not technological wizardry.

    To this specific case though, a variety of approaches have been suggested, from the “Cool URI’s” approach for the SemWeb, to PURLs to non-dereferencable URIs that are related via owl:sameAs. One could also opt for the DNS approach that GSLB takes – ensure at least one host remains stable in the face of an IP change, and pray that caching works out.

    One final note:

    Your focus has been very much on “querying”, i.e. classic CMDB chores. But what about the “management” part of configuration management, i.e. actually enacting change on these resources? Hypermedia advocates exposing a variety of links for such state-transitions, along with potentially unique media types to describe interfaces to those transitions. In other words, it would potentially be a way to enable our configuration management systems to actually DO things with their configuration data, in a uniform way, not just analyze it.

    This basically what I’ve found lacking in the WS-RM or WS-Man approaches, something I noted in a longish blog post almost a year back that I distilled into four general guidelines.

    Basically it comes down to, let’s really look at hypermedia as a new general model for building distributed systems, something that tends to have attributes over distributed objects or XML document exchange.

  4. Hi Stu,

    Thanks for the detailed comment. Here are a few responses to some of your many points:

    – WRT to RPC I don’t use that term in a way that implies OO. I just mean that you call a procedure by passing its input parameters and you get results back. Basically the way good old Sun RPC was defined. With no assumption of long-lived objects or any other persistence on the remote machine. In the Google case, I call a procedure called “search” by passing as input a set of keywords. I get back a list of URLs with a small description for each. It seems to match pretty well by that definition of RPC.

    – the “forms” part is interesting, but how does this translate to machine-to-machine interactions? How is my CMDB, for example, going to realize that this page represent an event notification subscription form and that this other one represents a query on the historical config of the resource? Unless you turn these forms (exposed via a common interface) into machine-readable interfaces of their own? Whether you do it though mime types or WSDL doesn’t matter as much as the fact that you have to have an agreement if there is no human sitting in front of the machine. What am I missing?

    – WRT to the pre-1995 web, I agree that technically we have the same web. But in practice we don’t because of the work Google and Microsoft have put into providing these search services. Sure they are just gigantic web apps, but in practice they are part of the Web and they shape (in many ways) how web apps are created and how they are consumed. And what the web is useful for.

    – And I very much agree that “there’s something very powerful about being able to access and analyze information without requiring portions of the “middleware” to understand much of its underlying schema or semantics”. For one thing, that’s what made Google possible. But not just that.

    – Agree that relationships can be made resources. It’s just that this is not predominant in REST systems. But nothing stops REST-for-IT from doing it.

    – Yep, there are hacks to try to deal with stale references, like the semweb approaches you list. BTW, WSRF had a child spec called WS-ReferenceRenewal that did just that but was never published. GGF wrote WS-Naming for that too.

    – Finally, wrt to enacting changes, I have to some extent covered this in part 1, when looking at the IaaS cloud APIs. One of the issues is the difference between the config you can get and what you can set. I don’t think it’s impossible but it takes a level of abstraction and templatization that goes beyond the current “make it look like a physical machine” approach. But some smart people may well figure out the right abstraction model, on which such a hypermedia-driven control system can be effective. Do you happen to know any such person?

  5. Stu

    Hi William,

    – WRT RPC, I still think there’s a major disconnect. I really wasn’t attempting to focus on an OO here. I was saying, fundamentally, the Web is not like RPC in *at all* except that they share Request-Response semantics, and that you could layer an RPC interface on top of the Web if you’d like, but the Web itself isn’t one. I’ll take a step back and provide broader context, some of which I’m sure you’re aware of, forgive me if it comes across as pedantic…

    Firstly, with Google, there’s no interface defined as “Search”. The interface is uniform – GET, POST, etc. To know what method to use and what data to send, the client has to dig into a different interface layer (i.e. into the data representation) and look at what hyperlinks exist there, and in what context. All interactions basically start with either a GET to a bookmarked URI or with a prior cached representation, and flow from there.

    That’s very different from a model where you have a Search interface that was pre-compiled into stubs & skeletons. Which is why I was suggesting the Web is much more like DII or COM Automation or Reflection, wherein you don’t pre-compile against a concrete interface, you query the interface, match strings, match argument lists, etc. It’s much more dynamic (and, in practice, feels cumbersome, because it was usually an afterthought to the original RPC model).

    Secondly, by “object-model”, I really meant “data-structure & memory-model”. RPC largely was about marshaling programmatic data-structures with portable serialization formats like XDR. The primacy of the interface, however, was oriented around the shape the server programmer wanted, with the primary intent being to “hide” the network interaction and make it look like a regular programmatic interaction. In SOA terms, it was a “producer-oriented contract”. Clients were forced to conform to what the server wanted; the burden was on them.

    Whereas, network transfer protocols, like FTP, HTTP, NNTP, etc., weren’t designed by programmers so much as network engineers. They are “Network APIs”, in that they weren’t based around any particular programmatic data structures or serialization formats. They used text and/or MIME entities, and were designed around the intent of the client (a “consumer-oriented contract”). The intent was to provide the network interaction as an explicit abstraction. This arguably leads to greater scale and reliability for client developers, at the cost of increased programmatic inconvenience for server developers who couldn’t dictate things to the client. HTTP+URI arguably is the culmination to date of these many decades of work on transfer protocols (with URI being the primary innovation).

    – Regarding Forms. These can be (and are) machine readable, though most of the ones we’ve seen in practice have been intended for hybrid human/machine consumption (XForms is a good example, but so are XHTML forms). Forms are the equivalent of describing an interface the way CORBA DII, COM IDispatch or java.lang.reflect.* do. Even an AtomPub interface describes a this kind of interface at a coarse grained level (“put image/* things here that belong in such-and-such category”).

    What’s missing is a widely standardized way to denote, in a machine readable way, what the intent of that form is. There are practical approaches out there: RDFa, Microformats like XMDP, or eschewing XHTML and just using well-specified JSON or XML hypertext documents.

    As to whether you describe an interface in MIME types or WSDL, there is a big difference. Describing a document with a schema-language, like “XSD”, is one thing. Even using SOAP-sans-WSDL would more be reasonable (using SOAPAction to denote intent). Both of these arguably can preserve the interaction model of hypermedia. But, WSDL is about describing a different interaction model that just happens to tunnel on top of HTTP. One describes operations and ports to note a variety of message exchange patterns. You’re jumping out of hypertext land and into Operation and Ports land all of a sudden. Is it bad? Perhaps not, if that’s what is needed. I often joke that WSDL needs REST anyway, since RESTful use of HTTP is the most common way to discover WSDL documents (i.e. surfing to your SOAP gateway with a ?WSDL query string). But it’s quite a jump in both infrastructure and processing model.

    – You suggested:
    “Agree that relationships can be made resources. It’s just that this is not predominant in REST systems.”

    The nascent systems you’ve analyzed in the Cloud API space, I agree. But, I suggest taking a look at how social syndication & aggregation sites like Facebook, Friendfeed, etc., provide lifecycle and metadata services to information relationships. Those things are absolutely using machine-readable data to do it too (even if the bulk of the content is being presented to an end-user).

    – WRT changes .. I figure we’ll come up with a good abstraction model as an industry some day. It might not be hypermedia-driven, but I hope it’s as extensible. I have no idea when this will happen. I do hope the work we’re doing will contribute to progressing this in some way, but after learning many hard lessons …. there’s a long, bumpy road ahead.

  6. Hi Stu and thanks for your perseverance.

    On the RPC stuff all you say is correct as far as the underlying technology. But I tend to look at this along the lines of the wave/particle duality of light (sorry for sounding pretentious). Use the viewpoint that works well for what you care about (e.g. light as waves for studying interference). I think there are many practical purposes where Google usage is more akin to RPC.

    Look at the opensearch box in your browser. It doesn’t re-discover the contract every time. It has loaded an interface description (in the opensearch format) which is specifically defined as a “search” interface. Same thing for all the sites that have a “search this site via Google” feature. They don’t do a GET on google.com, they have a hard-coded mechanism to build a Google search URL that restricts the search to the site at hand. It is not very different from “a Search interface that was pre-compiled into stubs & skeletons”.

    So I think the RPC and the hypermedia models apply best to different ways to use Google.

    In the context of CMDB and config stores (what I really care about here, the Google parenthesis is more of a distraction), it’s hard to imagine that most clients will take a hypertext approach (let’s go to assetmgmt.mycompany.com and see what page I get back) versus a contract (implicit or explicit) drive approach to integration. At least for the next few years, they’ll be in effect an implicit “search/query” function for system to system integration. Maybe not for the ultimate human interface. But I am mostly focused on automation so I care more about system to system than UI in this context.

    On the rise of “network protocols” (or “on the wire protocols” as I usually refer to them) you’re spot on and it’s a good thing. But POP and IMAP (and FTP) are also on-the-wire and they are closer in feel to RPC than to hypermedia. Though I agree that hypermedia systems in general make for better on-the-wire protocols because of fewer shared assumptions. And that indeed HTTP+URI is a culmination, at least for now. But it’s still wondering what exactly of this mechanism provides what value and how applicable it is to different applications. Which is what this blog series is all about.

    I don’t disagree with you on WSDL, though it is especially WSDL 1.1. Technically WSDL 2.0 doesn’t force this. But in practice WSDL is WSDL 1.1 (even if people use 2.0, they use it in the 1.1 mental model). The decision in WSDL 1.1 to assign a single URI to a port (and therefore prevent fine-grained URI-based addressing) is what let to the need for EPRs and potentially the single most damaging decision in the SOAP/WS stack. Sad, really.

    On the change/abstraction question, I share your hope. But I must say that I am a bit dismayed on the current focus on hypervisor-style, VM-based virtualization (“fake machines” as opposed to true “virtualized machines”). I understand all the practical benefits, but I am also pretty sure that the VMware-style VM is not the right building block for this abstraction. It will take a while to get to the right level of maturity. Middleware might be about to become cool again…

    Thanks again. Your comments are helping shape up the wrap-up of this series (part 3).

  7. Stu

    FWIW, I still think we have a disconnect on RPC, but I’ll drop it :)

    Regarding OpenSearch, I actually think that’s an example of hypermedia interaction, not RPC. You’re GETting the Opensearch document or keeping a cached representation around. I think is is *very* different from compiling stubs & skeletons. But elaborating beyond this is probably for another time.

  8. Stu,

    Yes, this clearly calls for coffee. Or a beer.

  9. Pingback: William Vambenepe — REST in practice for IT and Cloud management (part 3: wrap-up)

  10. Pingback: William Vambenepe — Amazon proves that REST doesn’t matter for Cloud APIs

  11. Pingback: What Language Does the Cloud Speak, Now and In the Future?