William Vambenepe's blog

The CMDBF (CMDB Federation) group just released a first public draft of the specification. Here is a direct link to it (PDF, 1MB). That’s a very nice achievement for a group that was a bit slow to find its pace but has been very productive since the beginning of 2007. Having the interop test drive our efforts had a lot to do with it, this is an approach to repeat the next time around.

This spec will look a lot more familiar to people used to reading WS-* specs than to people used to reading ITIL manuals. The concepts and use cases are (hopefully) consistent with ITIL, but the meat of the spec is about defining interoperable SOAP-based message exchanges that realize these use cases. The spec is about defining SOAP payloads, not IT management best practices. Please adjust your expectations accordingly and you’ll like the spec a lot more.

The most useful and important part (in my mind at least), is the definition of the Query service (section 4). This is used in many interactions. It is used by clients to query a CMDB that federates many MDRs (Management Data Repositories). It is used by the clients to go interact directly with the MDRs is they choose to. And it is used by the federating CMDB to retrieve data from the MDRs. Even in the more static scenarios, in which data is replicated from the MDRs into the CMDB (instead of true federation with no replication), the Query service is still the way for clients to access federated data from the CMDB.

So, yet another query language? Indeed. But one that natively supports concepts that are key to the kind of queries most useful for CMDB scenarios, namely relationships traversal and types. This is a topological query language. I hope the simple example in section 4.2 gives a good example of what this means.

Neither SQL nor XPath/XQuery has this topology-friendly approach (which is not to say that you can’t create useful queries on CMDB data with these query languages). On the other hand, there is one query language that is inherently relationship-oriented, that has native support for the notion of class and that has received a lot more attention, interop testing and implementation experience than our effort. One that I would have loved for us to leverage in CMDBF. It’s SPARQL. But semantic web technologies seem once again to be doomed by the perception that they are too much “out there” despite all the efforts of its proponents to make them connect with XML and other non-RDF views of the world.

Final caveat, this is a first draft. It is known to be incomplete (you’ll find text boxes that describe known gaps) and not all features in the spec were tested at the interop. We need more interop, review and development before it is robust. And most importantly, a lot of the difficult aspects of federation aren’t sufficiently addressed (reconciliation, model differences, source tracing, administrative metadata…) But it is at a point where we think it gives a good idea of how we are approaching the problem.

Equipped with this foreword, I wish you a pleasant read.

One thing I have constantly witnessed in the SOAP design work I have been involved in is the fear of headers. There are several aspects to this, and it’s very hard to sort out the causes and the consequences as they are now linked in a vicious cycle.

For one thing, headers in SOAP messages don’t map well to RPC and we all know SOAP’s history there. Even now, as people have learned that they’re supposed to say they are not doing RPC (”look, my WSDL says doc/literal therefore I am not doing RPC”), the code is still RPC-ish with the grand-children of the body being serialized into Java (or another language) objects and passed as arguments to an operation inside a machine-generated stub. No room for pesky headers.

In addition (and maybe because of this view), tools often bend over backward to prevent you from processing headers along with the rest of the message, requiring instead some handlers to be separately written and registered/configured in whatever way (usually poorly documented as people aren’t expected to do this) the SOAP stack requires.

And the circle goes on, with poor tooling support for headers being used to justify staying away from them in SOAP-based interfaces.

I wrote above that “headers in SOAP messages don’t map well to RPC” but the more accurate sentence is “headers that are not handled by the infrastructure don’t map well to RPC”. If the headers are generic enough to be processed by the SOAP stack before getting to the programmer, then the RPC paradigm can still apply. Halleluiah. Which is why there is no shortage of specs that define headers. But those are the specs written by the “big boys” of Web services, the same who build SOAP stacks. They trust themselves to create code in their stacks to handle WS-Addressing, WS-Security and WS-ReliableMessaging headers for the developers. Those headers are ok. But the little guys shouldn’t be trusted with headers. They’d poke their eyes out. Or, as Elliotte Rusty Harold recently wrote (on a related topic, namely WS-* versus REST), “developers have to be told what to do and kept from getting their grubby little hands all over the network protocols because they can’t be trusted to make the right choices”. The difference is that Elliotte is blasting any use of WS-* while I am advocating a use of SOAP that gives developers full access to the network protocol. Which goes in the same direction but without closing the door on using WS-* specs when they actually help.

I don’t mean to keep banging on my IBM friends, but I’ve spent a lot of time developing specs with well respected IBM engineers who will say openly that headers should not contain “application information”, by which they mean that headers should be processed by the SOAP stack and not require the programmer to deal with them.

That sounds very kind, and certainly making life easy for programmers is good, but what it does is rob them of the single most useful feature in SOAP. Let’s take a step back and look at what SOAP is. If we charitably leave aside the parts that are lipstick on the SOAP-RPC pig (e.g. SOAP data model, SOAP encoding and other RPC features), we are left with:

• the SOAP processing model
• the SOAP extensibility model
• SOAP faults

Of these three, the first two are all about headers. Loose the ability to define headers in your interface, you loose 2/3 of the benefits of SOAP (and maybe even more, I would argue that SOAP faults aren’t nearly as useful as the other two items in the list).

The SOAP processing model describes how headers can be processed by intermediaries and, most important of all, it clearly spells out how the mustUnderstand attribute works. So that everyone who processes a SOAP message knows (or should know) that a header with no such attribute (or where the attribute has a value of “false”) can be ignored if not recognized, while the message must not be processed if a non-understood header has this attribute set to “true”. It doesn’t sound like much, but it is huge in terms of reliably supporting features and extending an existing set of SOAP messages.

When they are deprived of this mechanism (or rather, scared away from using it), people who create SOAP interfaces end up re-specifying this very common behavior over and over by attaching it to parts of the body. But then it has to be understood and implemented correctly every time rather than leveraging a built-in feature of SOAP that any SOAP engine must (if compliance with the SOAP spec has any meaning) implement.

Case in point, the wsman:OptimizeEnumeration element in WS-Management. The normal pattern for an enumeration (see WS-Enumeration) is to first send an Enumerate request to retrieve an enumeration context and then a series of Pull requests to get the data. In this approach, the minimum number of interactions to retrieve even a small enumeration set is 2 (one Enumerate and one Pull). The OptimizeEnumeration element was designed to allow one to request that the Enumerate call be also the first Pull so that we can save one interaction. Not a big deal if you are going to send 500 Pull messages to retrieve the data, but if the enumeration set is small enough to retrieve all the data in one or two Pull messages then you can optimize traffic by up to 50%.

WS-Management decided to do that with an extension element in the body rather than a SOAP header. The problem then is what happens when a WS-Enumeration implementation that doesn’t use the WS-Management additions gets that element. There is no general rule in SOAP about how to handle this. In this case, the WS-Enumeration specs says to ignore the extensions, but it only says so in the “notation conventions” section, and as a SHOULD, not a MUST.

Not surprisingly, some implementers made the wrong assumption on what the presence of wsman:OptimizeEnumeration means and as a result their code faults on receiving a message with this option, rather than ignoring it. Something that would likely not have happened had the SOAP processing model been used rather than re-specifying this behavior.

Using a header not only makes it clear when an extension can be ignored, it also provides, at no additional cost, the ability to specify when it should not be ignored. Rather than baking this into the spec as WS-Management tries to do (not very successfully), using headers allows a sender to decide based on his/her specific needs and capabilities whether support for this extension is a must or not in this specific instance. Usually, flexibility and interoperability tug in opposing directions, but using headers rather than body elements for protocol extensions improves both!

Two more comments about the use of headers in WS-Management:

• It shows that it’s not all IBM’s fault, because IBM had no involvement in the development of the WS-Management standard,
• WS-Management is far from being the worst offender at under-using SOAP headers, since it makes good use of them in other places (which probably wouldn’t have been the case had the first bullet not been true). But not consistently enough.

Now let’s look at WS-ResourceTransfer for another example. This too illustrates that those who ignore the SOAP processing model just end up re-inventing it in the body, but it also gives a beautiful illustration of the kind of compromises that come from design by committee (and I was part of this compromise so I am as guilty as the other authors).

WS-ResourceTransfer creates a set of extensions on top of the WS-Transfer specification. The main one is the ability to retrieve only a portion of the document as opposed to the whole document. Defining this extension as a SOAP header (as WS-Management does it) leverages the SOAP processing model. Doing so means you don’t have to write any text in your spec about whether this header can be ignored or not, and the behavior can be specified clearly and interoperably by the sender, through the use of the mustUnderstand attribute. In some case (say if the underlying XML document is in fact an entire database with gigabytes of data) I definitely want the receiver to fault if it doesn’t understand that I only want a portion of the document, so I will set mustUnderstand to true. In other cases (if the subset is not much smaller than the entire document and I am willing to wad through the entire document to extract the relevant data if I have to) then I’ll set mustUnderstand to “false” so that if the receiver can’t honor the extension then I get the whole document instead of a fault.

But with WS-ResourceTransfer, we had to deal with the “headers should not contain application information” view of the world in which it is heresy to use headers for these extensions (why the portion of the document to retrieve is considered “application information” while addressing information is not is beyond my reach and frankly not a debate worth having because this whole dichotomy only makes sense in an RPC view of the world).

So, between the “this information must be in the body” camp and the “let’s leverage the SOAP processing model” camp (my view, as should be obvious by now), the only possible compromise was to do both, with a mustUnderstand header that just says “look in the body to find the extensions that you must follow”. So much for elegance and simplicity. And since that header applies to all the extensions, you have to make all of them mustUnderstand or none of them. For example, you can’t have a wsrt:Create message in which the wsmex:Metadata extension must be understood but the wsrt:Fragment extension can be ignored.

WS-Management and WS-ResourceTransfer share the characteristic of being designed as extensions to existing specifications. But even when creating a SOAP-based interaction from scratch, one benefits from using the SOAP processing model rather than re-inventing ways to express whether support for an option is required or not. I have seen several other examples (in interfaces that aren’t public like the two examples above) that also re-invent this very common behavior for no good reason. Simply because developers are scared of headers.

These examples illustrate a simple point. Most of the value in SOAP resides in how headers are used in the SOAP processing model and the SOAP extensibility model. Steering developers away from defining headers is robbing them of that value. Rather than thinking in terms of header and body, I prefer to think of them as header and tail.

My CMDBF colleague from BMC Van Wiles has a short update on the state of the CMDBF work. This is an occasion for me to point to his blog. I know that several of my readers are very interested in the CMDBF work, and they should probably monitor Van’s posts in addition to mine.

Like Van, I see the pace accelerating, which is good. More importantly, the quality of the discussion has really improved, not just the quantity. We’ve spent way too much time in UML-land (come on people, this is a protocol, get the key concepts and move on, no need to go in UML to the same level of detail that the XML contains), but the breakthrough was the move to pseudo-schema, and from there to example XML instance docs and finally to java code for the interop. I am having a lot of fun writing this code. I used this opportunity to take XOM on a road test and it’s been a very friendly companion.

Now that we’re getting close to the CMDBF interop, I feel a lot better about the effort than I did at the beginning of the year. It’s still going to be challenging to make the interop. Even if we succeed, the interop only covers a portion of what the spec needs to deliver. But it will be a very good milestone.

This HTML document titled “XMLFrag header” describes a proposal I wrote for a header that would allow one to target a SOAP message to a subset of the resource to which the message is sent. This is useful in cases where messages are used to interact with systems that have a well-known model, as is often the case in the IT systems management world. If you’ve ever used or been intrigued by WS-Management’s “FragmentTransfer” header, or WS-ResourceProperties’s “QueryResourceProperties” message or WS-RessourceTransfer’s “Expression” element then you may be interested by a different approach that is operation-independent. Please read the example at the beginning of the proposal for an explanation of what XMLFrag does.

I think it’s nifty, but I am not sure how much need for this there is. Are the composition/mash-up scenarios that this allows important enough to replace well established alternatives, such as WS-Management’s “FragmentTransfer” header? I am not sure. Which is why I am putting this out, to see if there’s any interest.

One possible approach would be to generalize WS-Management’s current mechanism to support the features presented here. This could be done very easily and in a backward-compatible way by declaring that the response wrapper (wsman:XmlFragment) is not applied for all dialect but defined in a per-dialect fashion. The currently defined dialects would keep using the wsman:XmlFragment wrapper, but a new dialect could be defined that didn’t use a wrapper and would behave like the XMLFrag element defined in my proposal.

A few additional notes and comments:

This proposal has been called “WS-SubversiveAddressing” by some of my IBM friends who have very definite ideas about what belongs in SOAP headers, what belongs in the body and what headers are appropriate to use as reference properties in an EPR. And this proposal seems to break all these rules. But since the rules seem more inspired by lack of flexibility in the WebSphere message routing/processing capabilities than by true architectural constraints I am not too worried. I would even say that this proposal represents the only kind of header that really make sense to use as reference properties. Headers that sometimes are set by the sender explicitly and sometimes are hard-coded in the EPR used. Using reference properties for headers that only come from reference properties (and aren’t expected to ever be set explicitly by the sender) is a sign of lack of ability of the stack to route messages based on URL. A too-frequent limitation of Java SOAP stacks. But I am digressing…

I didn’t think there was anything worthy of a patent in this, but in these sad times you can’t be sure that someone is not going to try to get one for it, so just to be sure HP published this a year ago in Research Disclosure so that no-one can patent it. If you want to check, it starts on page 627 of the May 2006 Research Disclosure (not available on-line unless you have an account w/ them, but you can order it). So in reality, this document has already been public for a year, but in a pretty hidden form. This post just gives it a little bit more visibility.

This is not a spec. It is just a description of what a spec could do. It doesn’t have normative language, it doesn’t provide formal syntax (pseudo-schema, XSD and/or other) and it doesn’t address some of the details that would be needed for interoperability (e.g. what namespaces declarations are in context for the XPath evaluation).

Why use a car fleet as an example for illustration? For the same reason that the SML spec uses a university (class/student) example. To pick a domain that is different from the expected domain of application of the technology to not invite modeling discussions that are irrelevant to the proposal and to not biaise people’s view of what this could be used for.

Is there a relationship between this and federation efforts? Sort of. This could be very useful when exposed by a a federator, if the model is very hierachical. But it doesn’t work so well for graph-type models. Which is what CMDBF does and which is why CMDBF is coming up with a more graph-oriented approach. But that too is a different topic.

Omri Gazitt presumably couldn’t find sleep last Friday night, so he wrote a well thought-out blog post instead. Well worth the read. His view on this is both broad and practical. There is enough in this post to, once again, take me within inches of trying WCF. But the fact that for all practical purposes the resulting code can only be deployed on Windows stops me from making this investment.

And since he still couldn’t sleep he penned another entry shortly after. That one is good but a bit less convincing. Frankly, I don’t think the technical differences between Java/C# and “dynamic languages” have much to do with the fact that stubs hurt you more often than not when developing code to process XML messages. With a sentence like “in a typed language, the conventional wisdom is that generating a proxy for me based on some kind of description of the service will make it easier for me to call that service using my familiar language semantics” Omri takes pain to avoid saying whether he agrees with this view. But if he doesn’t (and I don’t think he does), you’d think that he’d be in a pretty good position (at least on the .NET side) to change the fact that, as he says “the way WSDL and XSD are used in platforms like J2EE and .NET tends to push you towards RPC”…

I haven’t used .NET since writing C# code back when HP was selling the Bluestone J2EE server and I was in charge of Web services interoperability, so I have limited expertise there. But Java has the exact same problem with its traditional focus on RPC (just ask Steve). I am currently writing a prototype in Java for the CMDB Federation specification that is still at an early stage. All based on directly processing the XML (mostly through a bunch of XPath queries) and it makes it a breeze to evolve as the draft spec changes. Thank you XOM (and soon Nux).

I very much agree with the point Omri is making (that relying on metadata to add complexity in order to remove it) is an issue, but it’s not just for dynamic languages.

As part of the work around the convergence of WS-Management and WSDM, HP, IBM, Intel and Microsoft just published a first version of a specification called WS-ResourceCatalog. This specification provides a way to list management endpoints for resources. For example, the BMC on a server motherboard could host a catalog that lists the management endpoints for its different components (network card, CPU, disk, etc). This is an attempt to bring more consistency to discovery scenarios.

The spec has been submitted to the DMTF for its consideration as part of its Web services-based management protocol efforts. The submission includes a list of issues related to the spec, so it’s pretty clear that it’s nowhere near done. Rather than hammering things out even longer (trust me, it’s been too long already), we decided to hand it over as is to the DMTF and let its members decide how to handle the issues. And any other change they wish to make.

Since I am on a roll with off-topic posts…

I accidentally ran into some Web pages and scripts I wrote between 1994 and 1996. Mostly experiments with Web technologies that were emerging at the time. Some have pretty much disappeared (VRML), some are still pretty useful but slowly on their way out (CGI) but many of them are very prominent now. I found a bunch of Python scripts I wrote back then, some Java apps and applets and even a Minesweeper game written in JavaScript. And the impressive thing is that even though those were all pretty early technologies at the time, these programs seem to run just fine today with the latest virtual machines and interpreters for their respective languages. Kuddos to the people who have been growing these technologies while maintaining backward compatibility. Speaking of technologies that were emerging at the time and have made it big since then, all these were served from a Linux server and the Python stuff was developed on a Linux desktop (Slackware was the distribution of choice).

The previously released SML 1.0 and SML-IF 1.0 specifications have been submitted to W3C for standardization (actually the submission happened on 2/28 but W3C has acknowledged it today). My guess is that the fact that this announcement comes on the same day that SCA 1.0 is released is not going to decrease the confusion between these efforts.

A look at the “specifications” page of the “Open SOA” web site (the site used by the companies that created the SCA and SDO specifications) reveals a long list of specs with a release date of tomorrow. It’s like stumbling on the quarterly announcement of a publicly traded company the day before the announcement… except without the profit potential.

There is no link at this point, so no luck accessing the specifications themselves (unless one feels lucky and wants to try guessing the URLs based on those used for previously posted documents…) but we now know what they are and that they are coming out tomorrow:

• SCA Assembly Model V1.00
• SCA Policy Framework V1.00
• SCA Java Common Annotations and APIs V1.00
• SCA Java Component Implementation V1.00
• SCA Spring Component Implementation V1.00
• SCA BPEL Client and Implementation V1.00
• SCA C++ Client and Implementation V1.00
• SCA Web Services Binding V1.00
• SCA JMS Binding V1.00
• SCA EJB Session Bean Binding V1.00

The second one is the one I’ll read first.

After taking the form of two early drafts (versions 0.5 and 0.65), the SML specification has now reached v1.0, along with its sidekick the SML-IF specification. You can find both of them at serviceml.org. This is where the happy bunch that assembled to create these specs would normally part ways. Not quite true in this case since there is related work about to be tackled by a very similar set of people (more on this later), but at least we are not going to touch SML and SML-IF anymore. They are ready for submission to a standards body where further modifications will take place (more on this later too).