Dave Chappell and David Berry have recently published an article in SOA Magazine titled “The Next-Generation, Grid-Enabled Service-Oriented Architecture”. I had unexpectedly gotten a quick overview of this work a few weeks ago, when I ran into Dave Chappell at the Oracle gym (since I was coming out of an early morning swim it took Dave a couple of seconds to recognize me, as I walked through the weight room leaving a trail of water behind me). Even if you are more interested in systems management than middleware, this article is worth your reading time because it describes a class of problems (or rather, opportunities) that cut across middleware and IT management. Namely, providing the best environment for scalable, reliable and flexible SOA applications. In other words, making the theoretical promises of SOA practically achievable in real scenarios. The article mentions “self-healing management and SLA enforcement” and it implies lots of capabilities to automatically provision, configure and manage the underlying elements of the Grid as well the SOA applications that make use of it. Those are the capabilities that I am now working on as part of the Oracle Enterprise Manager team. And the beauty of doing this at Oracle, is that we can work on this hand in hand with people like Dave to make sure that we don’t create artificial barriers between middleware and systems management.
Grid-enabled SOA article
Filed under Articles, Everything, Grid, Grid-enabled SOA, IT Systems Mgmt, Tech
A review of OVF from a systems management perspective
I finally took a look at OVF, the virtual machine distribution specification that was recently submitted to DMTF. The document is authored by VMware and XenSource, but they are joined in the submission to DMTF by some other biggies, namely Microsoft, HP, IBM and Dell.
Overall, the specification does a good job of going after the low-hanging fruits of VM distribution/portability. And the white paper is very good. I wish I could say that all the specifications I have been associated with came accompanied by such a clear description of what they are about.
I am not a virtualization, operating system or hardware expert. I am mostly looking at this specification from the systems management perspective. More specifically I see virtualization and standardization as two of the many threads that create a great opportunity for increased automation of IT management and more focus on the application rather than the infrastructure (which is part of why I am now at Oracle). Since OVF falls in both the “virtualization” and “standardization” buckets, it got my attention. And the stated goal of the specification (“facilitate the automated, secure management not only of virtual machines but the appliance as a functional unit”, see section 3.1) seems to fit very well with this perspective.
On the other hand, the authors explicitly state that in the first version of the specification they are addressing the package/distribution stage and the deployment stage, not the earlier stage (development) or the later ones (management and retirement). This sidesteps many of the harder issues, which is part of why I write that the specification goes after the low-hanging fruits (nothing wrong with starting that way BTW).
The other reason for the “low hanging fruit” statement is that OVF is just a wrapper around proprietary virtual disk formats. It is not a common virtual disk format. I’ve read in several news reports that this specification provides portability across VM platforms. It’s sad but almost expected that the IT press would get this important nuance wrong, it’s more disappointing when analysts (who should know better) do, as for example the Burton Group which writes in its analysis “so when OVF is supported on Xen and VMware virtualization platforms for example, a VM packaged on a VMware hypervisor can run on a Xen hypervisor, and vice-versa”. That’s only if someone at some point in the chain translates from the Xen virtual disk format to the VMware one. OVF will provide deployment metadata and will allow you to package both virtual disks in a TAR if you so desire, but it will not do the translation for you. And the OVF authors are pretty up front about this (for example, the white paper states that “the act of packaging a virtual machine into an OVF package does not guarantee universal portability or install-ability across all hypervisors”). On a side note, this reminds me a bit of how the Sun/Microsoft Web SSO MEX and Web SSO Interop Profile specifications were supposed to bridge Passport with WS-Federation which was a huge overstatement. Except that in that case, the vendors were encouraging the misconception (which the IT press happily picked up) while in the OVF case it seems like the vendors are upfront about the limitations.
There is nothing rocket-science about OVF and even as a non-virtualization expert it makes sense to me. I was very intrigued by the promise that the specification “directly supports the configuration of multi-tier applications and the composition of virtual machines to deliver composed services” but this turns out to be a bit of an overstatement. Basically, you can distribute the VMs across networks by specifying a network name for each VM. I can easily understand the simple case, where all the VMs are on the same network and talking to one another. But there is no way (that I can see) to specify the network topology that joins different networks together, e.g. saying that there is a firewall between networks “blue” and “red” that only allows traffic on port 80). So why would I create an OVF file that composes several virtual machines if they are going to be deployed on networks that have no relationships to one another? I guess the one use case I can think of would be if one of the virtual machines was assigned to two networks and acted as a gateway/firewall between them. But that’s not a very common and scalable way to run your networks. There is a reason why Cisco sells $30 billions of networking gear every year. So what’s the point of this lightweight distributed deployment? Is it just for that use case where the network gear is also virtualized, in the expectation of future progress in that domain? Is this just a common anchor point to be later extended with more advanced network topology descriptions? This looks to me like an attempt to pick a low-hanging fruit that wasn’t there.
Departing from usual practice, this submission doesn’t seem to come with any license grant, which must have greatly facilitated its release and the recruitment of supporters for the submission. But it should be a red flag for adopters. It’s worth keeping track of its IP status as the work progresses. Unless things have changed recently, DMTF’s IP policy is pretty weak so the fact that works happens there doesn’t guarantee much protection per se to the adopters. Interestingly, there are two sections (6.2 about the virtual disk format and 11.3 about the communication between the guest software and the deployment platform) where the choice of words suggests the intervention of patent lawyers: phrases like “unencumbered specification” (presumably unencumbered with licensing requirements) and “someone skilled in the art”. Which is not surprising since this is the part where the VMWare-specific, Xen-specific or Microsoft-specific specifications would plug in.
Speaking of lawyers, the section that allows the EULA to be shipped with the virtual appliance is very simplistic. It’s just a human-readable piece of text in the OVF file. The specification somewhat naively mentions that “if unattended installs are allowed, all embedded license sections are implicitly accepted”. Great, thanks, enterprises love to implicitly accept licensing terms. I would hope that the next version will provide, at least, a way to have a URI to identify the EULA so that I can maintain a list of pre-approved EULAs for which unattended deployment is possible. Automation of IT management is supposed to makes things faster and cheaper. Having a busy and expensive lawyer read a EULA as part of my deployment process goes against both objectives.
It’s nice of the authors to do the work of formatting the specification using the DMTF-approved DSPxxxx format before submitting to the organization. But using a targetnamespace in the dmtf.org domain when the specification is just a submission seems pretty tacky to me, unless they got a green light from the DMTF ahead of time. Also, it looks a little crass on the part of VMware to wrap the specification inside their corporate white paper template (cover page and back page) if this is a joint publication. See the links at http://www.vmware.com/appliances/learn/ovf.html. Even though for all I know VMware might have done most of the actual work. That’s why the links that I used to the white paper and the specification are those at XenSource, which offers the plain version. But then again, this specification is pretty much a wrapper around a virtual disk file, so graphically wrapping it may have seemed appropriate…
OK, now for some XML nitpicking.
I am not a fan of leaving elementformdefault set to “unqualified” but it’s their right to do so. But then they qualify all the attributes in the specification examples. That looks a little awkward to me (I tend to do the opposite and qualify the elements but not the attributes) and, more importantly, it violates the schema in appendix since the schema leaves attributeFormDefault to its default value (unqualified). I would rather run a validation before makings this accusation, but where are the stand-alone XSD files? The white paper states that “it is the intention of the authors to ensure that the first version of the specification is implemented in their products, and so the vendors of virtual appliances and other ISV enablement, can develop to this version of the specification” but do you really expect us to copy/paste from PDF and then manually remove the line numbers and header/footer content that comes along? Sorry, I have better things to do (like whine about it on this blog) so I haven’t run the validation to verify that the examples are indeed in violation. But that’s at least how they look to me.
I also have a problem with the Section and Content elements that are just shells defined by the value of their xsi:type attribute. The authors claim it’s for extensibility (“the use of xsi:type is a core part of making the OVF extensible, since additional type definitions for sections can be added”) but there are better ways to do extensibility in XML (remember, that’s what the X stands for). It would be better to define an element per type (disk, network…). They could possibly be based on the same generic type in XSD. And this way you get more syntactic flexibility and you get the option to have sub-types of sub-types rather than a flat list. Interestingly, there is a comment inside the XSD that defines the Section type that reads “the base class for a section. Subclassing this is the most common form of extensibility”. That’s the right approach, but somehow it got dropped at some point.
Finally, the specification seems to have been formated based on WS-Management (which is the first specification that mixed the traditional WS-spec conventions with the DMTF DSPxxxx format), which may explain why WS-Management is listed as a reference at the end even though it is not used anywhere in the specification. That’s fine but it shows in a few places where more editing is needed. For example requirement R1.5-1 states that “conformant services of this specification MUST use this XML namespace Universal Resource Identifier (URI): http://schemas.dmtf.org/ovf”. I know what a conformant service is for WS-Management but I don’t know what it is for this specification. Also, the namespace that this requirement uses is actually not defined or used by this specification, so this requirement is pretty meaningless. The table of namespaces that follows just after is missing some namespaces. For example, the prefix “xsi” is used on line 457 (xsi:any and xsi:AnyAttribute) and I want to say it’s the wrong one as xsi is usually assigned to “http://www.w3.org/2001/XMLSchema-instance” and not “http://www.w3.org/2001/XMLSchema” but since the prefix is not in the table I guess it’s anyone’s guess (and BTW, it’s “anyAttribute”, not “AnyAttribute”).
By this point I may sound like I don’t like the specification. Not at all. I still stand with what I wrote in the second paragraph. It’s a good specification and the subset of problems that it addresses is a useful subset. There are a few things to fix in the current content and several more specifications to write to complement it, but it’s a very good first step and I am glad to see VMware and XenSource collaborating on this. Microsoft is nominally in support at this point, but it remains to be seen to what extent. I haven’t seen them in the past very interested in standards effort that they are not driving and so far this doesn’t appear to be something they are driving.
Filed under DMTF, Everything, IT Systems Mgmt, OVF, Specs, Standards, Tech, Virtualization, VMware, XenSource
Moving on
I am now an Oracle employee. My last day at HP was last Friday. I have a lot of excellent memories of my almost nine years there. And the company is (finally) very serious about software and investing a lot in it. HP Software is a very good place to be. But so is Oracle and the very interesting position I was offered convinced me that now was the time to go. So I am now in the Enterprise Manager group with the title of Architect. More specifically, I am in the part of EM that manages Middleware and Applications. Which also means that I’ll get to interact with the ex-Bluestone people who were my colleagues at HP Middleware and later joined Oracle’s application server team (like Greg). And I just learned today that David Chappell (with whom I collaborated on several specs) recently joined that group too. This is a happening place.
Filed under Everything, HP, Off-topic, Oracle
Tutorial and pseudo-algorithm for CMDBF Query operation
[UPDATE: an updated version of the algorithm that complies to version 1.0 of the specification (and not 0.95 as in this post) is now available]
The CMDBF Query operation (section 4 of the CMDBF specification) quickly becomes very intuitive to use, but it can first look a little strange to people used to SQL-style queries, because of its graph-based nature. I hope the normative text and the examples in the spec mitigate this. But just in case, here is some additional information that doesn’t belong in the spec but can be useful. Think of it as a very first draft of a primer/tutorial about that Query interface.
The easiest way to think about this interface is to think graphically. Imagine that you’re not writing your query in XML, but instead creating it in a GUI. You want to find all Windows XP machines that are owned by someone in the marketing department. First you create a circle in the GUI that represents the machine (through a Visio-like drag-and-drop into the query composer window). You right-click on that circle and in the right-click menu you select the option called “type” and you set it to “computerSystem”. Then you right-click to select “add property constraint”. You enter “OS_Version” as the property name (if your GUI tools is any good it will present you with a list to choose from, based on the previously selected type) and “WindowsXP” as the value. Already you’ve created a query that selects all Windows XP machines. Not a bad start.
But you only want the machines that are owned by marketing people. So go ahead and create another circle to represent a person. Use similar right-click actions to set the type to “person” and to set the “department” property to “marketing”.
If you submit the query at this point, you’ll get a list of all Windows XP machines and all people in the marketing department. So, rather than reducing the result (by removing machines not owned by marketing people), you have expanded it (by adding records for the marketing people). Oops.
The next step is to create a relationship that constrains the machine to belong to someone in marketing. This is done very simply in our handy GUI tool by drawing an arrow that goes from the “person” circle to the “machine” circle. This requires that there be a relationship between the two. If we want to further ensure that this relationship is of type “owns” (and not “uses” for example), we do this by right-clicking on the arrow (like we do on circles), selecting “type” and setting its value to “owns”.
If we run the query now, we get the list of all Windows XP machines owned by marketing people. We also get the list of the marketing people who own these machines and we get the relationships between people and machines. So we now have want we wanted. But maybe a little more. Maybe we only care about the list of machines, we don’t want to retrieve all the data about the marketing people. As long as we know the machines are owned by marketing people, that’s all that we care about. We don’t need to know what specific person owns what specific machine. We could simply ignore the people and relationships in the response. Or we could enrich the query to specify that the people and relationship need not be returned (but they are still part of the query in the sense that they limit what machines get returned).We do this by right-clicking on the “person” circle, and selecting the “suppress” option. Similarly, select the “suppress” option on the relationship (the arrow) too.
This query will return a list of all Windows XP machines that are owned by someone in marketing, and nothing else.
Here is what the query could look like graphically (here I assume that the GUI tools represents the fact that the arrow and the “person” circle have the “suppress” option selected on them by turning their solid lines into dotted lines and their text into italics):
The most intuitive way to think about what happens when the query gets processed is that the program looks for all instances of the patterns described by the query. In other words, it tries to superimpose the requested graph everywhere on the graph of available data and selects all the instances where the requested graph “fits”.
What does the GUI tool do behind the scene to turn this query into the proper XML, as described by the spec?
For each circle, it creates an <itemTemplate> element. For each arrow, it creates a <relationshipTemplate> element and sets its <source> and <target> elements to the right item templates. For each constraint on a circle or arrow (i.e. when we set the type or when we set the value or a give property) it creates the appropriate selector and embeds it in the <itemTemplate> or <relationshipTemplate> that corresponds to this circle or arrow. Finally, it sets the @dropDirective attribute to “true” on all the <itemTemplate> and <relationshipTemplate> elements that corresponds to circles and arrows on which the “suppress” option was selected.
Here is what the resulting query looks like in XML:
<query xmlns="http://schemas.cmdbf.org/0-9-5/datamodel">
<itemTemplate id="machine">
<propertyValueSelector namespace="http://example.com/computerModel" localName="OS_Version">
<equal>Windows XP</equal>
</propertyValueSelector>
<recordTypeSelector namespace="http://example.com/computerModel" localName="computerSystem"/>
</itemTemplate>
<itemTemplate id="person" dropDirective="true">
<propertyValueSelector namespace="http://example.com/peopleModel" localName="department">
<equal>marketing</equal>
</propertyValueSelector>
<recordTypeSelector namespace="http://example.com/peopleModel" localName="person"/>
</itemTemplate>
<relationshipTemplate id="administers" dropDirective="true">
<recordTypeSelector namespace="http://example.com/computerModel" localName="owns"/>
<source ref="person"/>
<target ref="machine"/>
</relationshipTemplate>
</query>
Note: like all query language, the actual query depends of course on the underlying model. In this example, I assumed that the OS version is represented as a property of the machine. More commonly, the OS will be a node of its own that has a relationship with the machine. So you’d have another circle for the OS. With a property constraint on that circle (version=”WindowsXP”) and a line representing a “runs” relationship between the machine circle and the OS circle. Similarly, “marketing” could be a node of its own that people have a relationship with, rather than just a property of each person. None of this changes the logic behind the Query operation.
Now, this is nice for the user of the query, but what about the poor developer who gets the 50-pages spec thrown on his/her desk and has 2 weeks to make sense of it and implement the server side of the query? I’ve said above that the program “tries to superimpose the requested graph everywhere on the graph of available data and selects all the instances where the requested graph fits” but that’s a lot easier to write as a sentence than to implement. So here is a pseudo-algorithm to help.
1) for each itemTemplate calculate the set of all items that obey all the selectors in the template. Call this the working set for the itemTemplate.
2) for each relationshipTemplate calculate the set of all relationships that obey all the selectors in the template. Call this the working set for the relationshipTemplate.
3) set need_to_loop = true
4) while (need_to_loop == true)
4.1) set need_to_loop = false
4.2) for each relationshipTemplate RT4.2.1) let ITsource be the itemTemplate that is referenced as source by RT
4.2.2) let ITtarget be the itemTemplate that is referenced as target by RT
4.2.3) for each relationship R in the working set for RT4.2.3.1) if the source of R is not in the working set for ITsource, then remove R from the RT working set
4.2.3.2) if the target of R is not in the working set for ITtarget, then remove R from the RT working set
4.2.3.3) if RT has a source/@minimum or a source/@maximum attribute4.2.3.3.1) find the list L1 of all relationships in the working set for RT that have the same source as R
4.2.3.3.2) if RT has source/@minimum and the cardinality of L1 is less than this minimum then remove all relationships in L1 from the RT working set
4.2.3.3.3) if RT has source/@maximum and the cardinality of L1 is more than this maximum then remove all relationships in L1 from the RT working set4.2.3.4) if RT has a target/@minimum or a target/@maximum attribute
4.2.3.4.1) find the list L2 of all relationships in the working set for RT that have the same target as R
4.2.3.4.2) if RT has target/@minimum and the cardinality of L2 is less than this minimum then remove all relationships in L2 from the RT working set
4.2.3.4.3) if RT has target/@maximum and the cardinality of L2 is more than this maximum then remove all relationships in L2 from the RT working set4.3) for each itemTemplate IT
4.3.1) let sourceRTset be the set of all relationshipTemplates that references IT as its source
4.3.2) let targetRTset be the set of all relationshipTemplates that references IT as its target
4.3.3) for each item I in the IT working set4.3.3.1) for each relationshipTemplate sourceRT in sourceRTset, if there is no relationship in the working set for sourceRT that uses I as its source, remove I from the IT working set and set need_to_loop to true
4.3.3.2) for each relationshipTemplate targetRT in targetRTset, if there is no relationship in the working set for targetRT that uses I as its source, remove I from the IT working set and set need_to_loop to true
5) process all directives (dropDirective or propertySubsetDirective) to remove or pair down items and relationships as requested
6) package the resulting items and relationships in a way that conforms to the CMDBF response message format (including putting each item in the <nodes> element with the appropriate @templateId attribute and putting each relationship in the <edges> element with the appropriate @templateId).
There are all kinds of optimizations possible here (like breaking out of loops once an instance has been removed, or not recalculating L1 and L2 over and over again for relationships in the same working set that share a source/target), but this is the most basic form or the algorithm. The goal is to illuminate the spec, not to provide an optimal implementation strategy.
In this post, I have focused on describing and illustrating the topological aspects of the query language. The other concepts that come into play are the Selector and the Directive mechanisms. But these are a lot more familiar to people used to SQL and I think they are sufficiently explained in the spec. So I have assumed here (in steps 1, 2 and 5 of the pseudo-algorithm) that they are well understood.
Filed under CMDB Federation, CMDBf, Everything, Graph query, Pseudo-algorithm, Query, Specs, Standards, Tutorial
First release of the CMDBF specification
The CMDBF (CMDB Federation) group just released a first public draft of the specification. Here is a direct link to it (PDF, 1MB). That’s a very nice achievement for a group that was a bit slow to find its pace but has been very productive since the beginning of 2007. Having the interop test drive our efforts had a lot to do with it, this is an approach to repeat the next time around.
This spec will look a lot more familiar to people used to reading WS-* specs than to people used to reading ITIL manuals. The concepts and use cases are (hopefully) consistent with ITIL, but the meat of the spec is about defining interoperable SOAP-based message exchanges that realize these use cases. The spec is about defining SOAP payloads, not IT management best practices. Please adjust your expectations accordingly and you’ll like the spec a lot more.
The most useful and important part (in my mind at least), is the definition of the Query service (section 4). This is used in many interactions. It is used by clients to query a CMDB that federates many MDRs (Management Data Repositories). It is used by the clients to go interact directly with the MDRs is they choose to. And it is used by the federating CMDB to retrieve data from the MDRs. Even in the more static scenarios, in which data is replicated from the MDRs into the CMDB (instead of true federation with no replication), the Query service is still the way for clients to access federated data from the CMDB.
So, yet another query language? Indeed. But one that natively supports concepts that are key to the kind of queries most useful for CMDB scenarios, namely relationships traversal and types. This is a topological query language. I hope the simple example in section 4.2 gives a good example of what this means.
Neither SQL nor XPath/XQuery has this topology-friendly approach (which is not to say that you can’t create useful queries on CMDB data with these query languages). On the other hand, there is one query language that is inherently relationship-oriented, that has native support for the notion of class and that has received a lot more attention, interop testing and implementation experience than our effort. One that I would have loved for us to leverage in CMDBF. It’s SPARQL. But semantic web technologies seem once again to be doomed by the perception that they are too much “out there” despite all the efforts of its proponents to make them connect with XML and other non-RDF views of the world.
Final caveat, this is a first draft. It is known to be incomplete (you’ll find text boxes that describe known gaps) and not all features in the spec were tested at the interop. We need more interop, review and development before it is robust. And most importantly, a lot of the difficult aspects of federation aren’t sufficiently addressed (reconciliation, model differences, source tracing, administrative metadata…) But it is at a point where we think it gives a good idea of how we are approaching the problem.
Equipped with this foreword, I wish you a pleasant read.
Filed under CMDB Federation, CMDBf, Everything, Specs, Tech
Gutting the SOAP processing model
One thing I have constantly witnessed in the SOAP design work I have been involved in is the fear of headers. There are several aspects to this, and it’s very hard to sort out the causes and the consequences as they are now linked in a vicious cycle.
For one thing, headers in SOAP messages don’t map well to RPC and we all know SOAP’s history there. Even now, as people have learned that they’re supposed to say they are not doing RPC (“look, my WSDL says doc/literal therefore I am not doing RPC”), the code is still RPC-ish with the grand-children of the body being serialized into Java (or another language) objects and passed as arguments to an operation inside a machine-generated stub. No room for pesky headers.
In addition (and maybe because of this view), tools often bend over backward to prevent you from processing headers along with the rest of the message, requiring instead some handlers to be separately written and registered/configured in whatever way (usually poorly documented as people aren’t expected to do this) the SOAP stack requires.
And the circle goes on, with poor tooling support for headers being used to justify staying away from them in SOAP-based interfaces.
I wrote above that “headers in SOAP messages don’t map well to RPC” but the more accurate sentence is “headers that are not handled by the infrastructure don’t map well to RPC”. If the headers are generic enough to be processed by the SOAP stack before getting to the programmer, then the RPC paradigm can still apply. Halleluiah. Which is why there is no shortage of specs that define headers. But those are the specs written by the “big boys” of Web services, the same who build SOAP stacks. They trust themselves to create code in their stacks to handle WS-Addressing, WS-Security and WS-ReliableMessaging headers for the developers. Those headers are ok. But the little guys shouldn’t be trusted with headers. They’d poke their eyes out. Or, as Elliotte Rusty Harold recently wrote (on a related topic, namely WS-* versus REST), “developers have to be told what to do and kept from getting their grubby little hands all over the network protocols because they can’t be trusted to make the right choices”. The difference is that Elliotte is blasting any use of WS-* while I am advocating a use of SOAP that gives developers full access to the network protocol. Which goes in the same direction but without closing the door on using WS-* specs when they actually help.
I don’t mean to keep banging on my IBM friends, but I’ve spent a lot of time developing specs with well respected IBM engineers who will say openly that headers should not contain “application information”, by which they mean that headers should be processed by the SOAP stack and not require the programmer to deal with them.
That sounds very kind, and certainly making life easy for programmers is good, but what it does is rob them of the single most useful feature in SOAP. Let’s take a step back and look at what SOAP is. If we charitably leave aside the parts that are lipstick on the SOAP-RPC pig (e.g. SOAP data model, SOAP encoding and other RPC features), we are left with:
- the SOAP processing model
- the SOAP extensibility model
- SOAP faults
Of these three, the first two are all about headers. Loose the ability to define headers in your interface, you loose 2/3 of the benefits of SOAP (and maybe even more, I would argue that SOAP faults aren’t nearly as useful as the other two items in the list).
The SOAP processing model describes how headers can be processed by intermediaries and, most important of all, it clearly spells out how the mustUnderstand attribute works. So that everyone who processes a SOAP message knows (or should know) that a header with no such attribute (or where the attribute has a value of “false”) can be ignored if not recognized, while the message must not be processed if a non-understood header has this attribute set to “true”. It doesn’t sound like much, but it is huge in terms of reliably supporting features and extending an existing set of SOAP messages.
When they are deprived of this mechanism (or rather, scared away from using it), people who create SOAP interfaces end up re-specifying this very common behavior over and over by attaching it to parts of the body. But then it has to be understood and implemented correctly every time rather than leveraging a built-in feature of SOAP that any SOAP engine must (if compliance with the SOAP spec has any meaning) implement.
Case in point, the wsman:OptimizeEnumeration element in WS-Management. The normal pattern for an enumeration (see WS-Enumeration) is to first send an Enumerate request to retrieve an enumeration context and then a series of Pull requests to get the data. In this approach, the minimum number of interactions to retrieve even a small enumeration set is 2 (one Enumerate and one Pull). The OptimizeEnumeration element was designed to allow one to request that the Enumerate call be also the first Pull so that we can save one interaction. Not a big deal if you are going to send 500 Pull messages to retrieve the data, but if the enumeration set is small enough to retrieve all the data in one or two Pull messages then you can optimize traffic by up to 50%.
WS-Management decided to do that with an extension element in the body rather than a SOAP header. The problem then is what happens when a WS-Enumeration implementation that doesn’t use the WS-Management additions gets that element. There is no general rule in SOAP about how to handle this. In this case, the WS-Enumeration specs says to ignore the extensions, but it only says so in the “notation conventions” section, and as a SHOULD, not a MUST.
Not surprisingly, some implementers made the wrong assumption on what the presence of wsman:OptimizeEnumeration means and as a result their code faults on receiving a message with this option, rather than ignoring it. Something that would likely not have happened had the SOAP processing model been used rather than re-specifying this behavior.
Using a header not only makes it clear when an extension can be ignored, it also provides, at no additional cost, the ability to specify when it should not be ignored. Rather than baking this into the spec as WS-Management tries to do (not very successfully), using headers allows a sender to decide based on his/her specific needs and capabilities whether support for this extension is a must or not in this specific instance. Usually, flexibility and interoperability tug in opposing directions, but using headers rather than body elements for protocol extensions improves both!
Two more comments about the use of headers in WS-Management:
- It shows that it’s not all IBM’s fault, because IBM had no involvement in the development of the WS-Management standard,
- WS-Management is far from being the worst offender at under-using SOAP headers, since it makes good use of them in other places (which probably wouldn’t have been the case had the first bullet not been true). But not consistently enough.
Now let’s look at WS-ResourceTransfer for another example. This too illustrates that those who ignore the SOAP processing model just end up re-inventing it in the body, but it also gives a beautiful illustration of the kind of compromises that come from design by committee (and I was part of this compromise so I am as guilty as the other authors).
WS-ResourceTransfer creates a set of extensions on top of the WS-Transfer specification. The main one is the ability to retrieve only a portion of the document as opposed to the whole document. Defining this extension as a SOAP header (as WS-Management does it) leverages the SOAP processing model. Doing so means you don’t have to write any text in your spec about whether this header can be ignored or not, and the behavior can be specified clearly and interoperably by the sender, through the use of the mustUnderstand attribute. In some case (say if the underlying XML document is in fact an entire database with gigabytes of data) I definitely want the receiver to fault if it doesn’t understand that I only want a portion of the document, so I will set mustUnderstand to true. In other cases (if the subset is not much smaller than the entire document and I am willing to wad through the entire document to extract the relevant data if I have to) then I’ll set mustUnderstand to “false” so that if the receiver can’t honor the extension then I get the whole document instead of a fault.
But with WS-ResourceTransfer, we had to deal with the “headers should not contain application information” view of the world in which it is heresy to use headers for these extensions (why the portion of the document to retrieve is considered “application information” while addressing information is not is beyond my reach and frankly not a debate worth having because this whole dichotomy only makes sense in an RPC view of the world).
So, between the “this information must be in the body” camp and the “let’s leverage the SOAP processing model” camp (my view, as should be obvious by now), the only possible compromise was to do both, with a mustUnderstand header that just says “look in the body to find the extensions that you must follow”. So much for elegance and simplicity. And since that header applies to all the extensions, you have to make all of them mustUnderstand or none of them. For example, you can’t have a wsrt:Create message in which the wsmex:Metadata extension must be understood but the wsrt:Fragment extension can be ignored.
WS-Management and WS-ResourceTransfer share the characteristic of being designed as extensions to existing specifications. But even when creating a SOAP-based interaction from scratch, one benefits from using the SOAP processing model rather than re-inventing ways to express whether support for an option is required or not. I have seen several other examples (in interfaces that aren’t public like the two examples above) that also re-invent this very common behavior for no good reason. Simply because developers are scared of headers.
These examples illustrate a simple point. Most of the value in SOAP resides in how headers are used in the SOAP processing model and the SOAP extensibility model. Steering developers away from defining headers is robbing them of that value. Rather than thinking in terms of header and body, I prefer to think of them as header and tail.
Filed under Everything, SOAP, SOAP header, Specs, Standards, Tech
CMDBF interoperability testing (alpha edition)
Last week, a bunch of us from the CMDBF author companies got together in a room for our first interoperability testing event. It was based on a subset of the spec. As usual with these kinds of events, the first hurdle was to get the network setup right (we had a mix of test endpoints running on remote servers and on our laptops; getting all laptops to talk to one another on a local network was easy; getting them all to talk to the remote endpoints over the internet was easy too; but getting both at the same time took a bit of work).
Once this was taken care of, the interop tests went pretty smoothly. A few problems were found but they were fixed on the fly and by the end we had happy MDRs (Management Data Repositories) talking to happy CMDBs who themselves were accessed by happy client applications. All this using the CMDBF-defined query and update interfaces.
Next steps are to update the spec with the lessons from the interop and to complete it with a few additional features that we put out of scope for the interop. Stay tuned.
Comments Off on CMDBF interoperability testing (alpha edition)
Filed under CMDB Federation, CMDBf, Everything, Implementation, Specs, Standards
Standards are good for customers… right?
Standards are good for customers. They avoid vendor lock-in. They protect the customer’s investment. Demanding standards compliance is a tool customers have to defend their interests when dealing with vendors. Right?
Well, in general yes. Except when standards become tools for vendors to attempt to confuse customers.
In the recent past, I have indirectly witnessed vendors liberally using the “standard” word and making claims of compliance with (and touting the need to conform to) specifications…
- that have barely been submitted for standardization (SML),
- that haven’t even been published in any form (CMDBF), or
- that don’t even exist as a draft (CML – no link available, and for a reason).
Doesn’t something sound fishy when the logic goes through such self-negating statements as: “standards are good for you because they give you a choice of vendor. And we are the only vendor who supports standard X so you need to buy from us.” Especially when if it was true that the vendor in question implemented standard X, then it would not be their software that I would want to buy from them but their time machine.
All this doesn’t negate the fundamental usefulness of standards. And I don’t mean to attack the three specifications listed above either. They all have a very good potential to turn out to be useful. HP is fully engaged in the creation of all three (I am personally involved in authoring them, which is generally why wind of these exaggerated vendor claims eventually get back to me).
Vendors who are used to creating proprietary environments haven’t all changed their mind. They’ve sometimes just changed their rhetoric and updated their practices to play the standards game (changing the game itself in the process, and often not for the better). Over-eagerness should always arouse suspicion.
Comments Off on Standards are good for customers… right?
Filed under Business, CMDB Federation, CMDBf, CML, Everything, SML, Standards
CMDBF update
My CMDBF colleague from BMC Van Wiles has a short update on the state of the CMDBF work. This is an occasion for me to point to his blog. I know that several of my readers are very interested in the CMDBF work, and they should probably monitor Van’s posts in addition to mine.
Like Van, I see the pace accelerating, which is good. More importantly, the quality of the discussion has really improved, not just the quantity. We’ve spent way too much time in UML-land (come on people, this is a protocol, get the key concepts and move on, no need to go in UML to the same level of detail that the XML contains), but the breakthrough was the move to pseudo-schema, and from there to example XML instance docs and finally to java code for the interop. I am having a lot of fun writing this code. I used this opportunity to take XOM on a road test and it’s been a very friendly companion.
Now that we’re getting close to the CMDBF interop, I feel a lot better about the effort than I did at the beginning of the year. It’s still going to be challenging to make the interop. Even if we succeed, the interop only covers a portion of what the spec needs to deliver. But it will be a very good milestone.
Filed under Everything, Implementation, Specs, Standards, Tech
XMLFrag SOAP header
This HTML document titled “XMLFrag header” describes a proposal I wrote for a header that would allow one to target a SOAP message to a subset of the resource to which the message is sent. This is useful in cases where messages are used to interact with systems that have a well-known model, as is often the case in the IT systems management world. If you’ve ever used or been intrigued by WS-Management’s “FragmentTransfer” header, or WS-ResourceProperties’s “QueryResourceProperties” message or WS-RessourceTransfer’s “Expression” element then you may be interested by a different approach that is operation-independent. Please read the example at the beginning of the proposal for an explanation of what XMLFrag does.
I think it’s nifty, but I am not sure how much need for this there is. Are the composition/mash-up scenarios that this allows important enough to replace well established alternatives, such as WS-Management’s “FragmentTransfer” header? I am not sure. Which is why I am putting this out, to see if there’s any interest.
One possible approach would be to generalize WS-Management’s current mechanism to support the features presented here. This could be done very easily and in a backward-compatible way by declaring that the response wrapper (wsman:XmlFragment) is not applied for all dialect but defined in a per-dialect fashion. The currently defined dialects would keep using the wsman:XmlFragment wrapper, but a new dialect could be defined that didn’t use a wrapper and would behave like the XMLFrag element defined in my proposal.
A few additional notes and comments:
This proposal has been called “WS-SubversiveAddressing” by some of my IBM friends who have very definite ideas about what belongs in SOAP headers, what belongs in the body and what headers are appropriate to use as reference properties in an EPR. And this proposal seems to break all these rules. But since the rules seem more inspired by lack of flexibility in the WebSphere message routing/processing capabilities than by true architectural constraints I am not too worried. I would even say that this proposal represents the only kind of header that really make sense to use as reference properties. Headers that sometimes are set by the sender explicitly and sometimes are hard-coded in the EPR used. Using reference properties for headers that only come from reference properties (and aren’t expected to ever be set explicitly by the sender) is a sign of lack of ability of the stack to route messages based on URL. A too-frequent limitation of Java SOAP stacks. But I am digressing…
I didn’t think there was anything worthy of a patent in this, but in these sad times you can’t be sure that someone is not going to try to get one for it, so just to be sure HP published this a year ago in Research Disclosure so that no-one can patent it. If you want to check, it starts on page 627 of the May 2006 Research Disclosure (not available on-line unless you have an account w/ them, but you can order it). So in reality, this document has already been public for a year, but in a pretty hidden form. This post just gives it a little bit more visibility.
This is not a spec. It is just a description of what a spec could do. It doesn’t have normative language, it doesn’t provide formal syntax (pseudo-schema, XSD and/or other) and it doesn’t address some of the details that would be needed for interoperability (e.g. what namespaces declarations are in context for the XPath evaluation).
Why use a car fleet as an example for illustration? For the same reason that the SML spec uses a university (class/student) example. To pick a domain that is different from the expected domain of application of the technology to not invite modeling discussions that are irrelevant to the proposal and to not biaise people’s view of what this could be used for.
Is there a relationship between this and federation efforts? Sort of. This could be very useful when exposed by a a federator, if the model is very hierachical. But it doesn’t work so well for graph-type models. Which is what CMDBF does and which is why CMDBF is coming up with a more graph-oriented approach. But that too is a different topic.
Filed under Everything, SOAP, SOAP header, Specs, Tech, XMLFrag
Omri on SOAP and WCF
Omri Gazitt presumably couldn’t find sleep last Friday night, so he wrote a well thought-out blog post instead. Well worth the read. His view on this is both broad and practical. There is enough in this post to, once again, take me within inches of trying WCF. But the fact that for all practical purposes the resulting code can only be deployed on Windows stops me from making this investment.
And since he still couldn’t sleep he penned another entry shortly after. That one is good but a bit less convincing. Frankly, I don’t think the technical differences between Java/C# and “dynamic languages” have much to do with the fact that stubs hurt you more often than not when developing code to process XML messages. With a sentence like “in a typed language, the conventional wisdom is that generating a proxy for me based on some kind of description of the service will make it easier for me to call that service using my familiar language semantics” Omri takes pain to avoid saying whether he agrees with this view. But if he doesn’t (and I don’t think he does), you’d think that he’d be in a pretty good position (at least on the .NET side) to change the fact that, as he says “the way WSDL and XSD are used in platforms like J2EE and .NET tends to push you towards RPC”…
I haven’t used .NET since writing C# code back when HP was selling the Bluestone J2EE server and I was in charge of Web services interoperability, so I have limited expertise there. But Java has the exact same problem with its traditional focus on RPC (just ask Steve). I am currently writing a prototype in Java for the CMDB Federation specification that is still at an early stage. All based on directly processing the XML (mostly through a bunch of XPath queries) and it makes it a breeze to evolve as the draft spec changes. Thank you XOM (and soon Nux).
I very much agree with the point Omri is making (that relying on metadata to add complexity in order to remove it) is an issue, but it’s not just for dynamic languages.
Comments Off on Omri on SOAP and WCF
Filed under Everything, Implementation, SOAP, Tech, XOM
All things (not very carefully) considered
Another off-topic entry to add to the CrazyStats category. Today’s NPR’s “All Things Considered” included a report called “States Fret at Easing of Border Security Plan” which talked about “Operation Jump Start”, so described:
“For about a year, National Guard troops have been rotating in and out of outposts along the [US-Mexico] border. Soldiers stayed visible under blue tents right on the border to deter illegal crossers while scanning the landscape, reporting anyone who did cross.”
It then goes on:
“The deterrent worked. The number of crossers apprehended by the Border Patrol since last October is down by about one-third, while drug seizures are up.”
The implication seems to be that would-be illegal immigrants were deterred by the presence of the troops and that drug traffickers were not deterred but were more often caught thanks to the help of the troops (who presumably either directly caught drug carriers or freed up Border Patrol resources to go after them). Success! But what if the result had been the exact opposite? More crossers apprehended and fewer drug seizures. Couldn’t that just as easily be interpreted to mean that the troops helped in catching more crossers while providing reinforcements that deterred drug traffickers? When opposite results can be interpreted to both mean success the test is suspicious.
Filed under CrazyStats, Off-topic
A new management catalog proposal
As part of the work around the convergence of WS-Management and WSDM, HP, IBM, Intel and Microsoft just published a first version of a specification called WS-ResourceCatalog. This specification provides a way to list management endpoints for resources. For example, the BMC on a server motherboard could host a catalog that lists the management endpoints for its different components (network card, CPU, disk, etc). This is an attempt to bring more consistency to discovery scenarios.
The spec has been submitted to the DMTF for its consideration as part of its Web services-based management protocol efforts. The submission includes a list of issues related to the spec, so it’s pretty clear that it’s nowhere near done. Rather than hammering things out even longer (trust me, it’s been too long already), we decided to hand it over as is to the DMTF and let its members decide how to handle the issues. And any other change they wish to make.
Filed under Specs, Standards, Tech, WS-ResourceCatalog
Want to play a minesweeper game?
Since I am on a roll with off-topic posts…
I accidentally ran into some Web pages and scripts I wrote between 1994 and 1996. Mostly experiments with Web technologies that were emerging at the time. Some have pretty much disappeared (VRML), some are still pretty useful but slowly on their way out (CGI) but many of them are very prominent now. I found a bunch of Python scripts I wrote back then, some Java apps and applets and even a Minesweeper game written in JavaScript. And the impressive thing is that even though those were all pretty early technologies at the time, these programs seem to run just fine today with the latest virtual machines and interpreters for their respective languages. Kuddos to the people who have been growing these technologies while maintaining backward compatibility. Speaking of technologies that were emerging at the time and have made it big since then, all these were served from a Linux server and the Python stuff was developed on a Linux desktop (Slackware was the distribution of choice).
Filed under Everything, Game, JavaScript, Minesweeper, Off-topic, Tech
We won’t get rid of SSN-based authentication anytime soon…
… because the issue has been mixed up with the whole terrorism/DHS hysteria. Game over. So now we have “Real ID” which won’t stop any terrorist but somehow is marketed as an anti-terrorist measure. I don’t like this law because it is too focused on physical identification (ID card) and not virtual identification. Trying to impersonate someone in person is difficult, dangerous (you risk being arrested on the spot or at least having your face captured by a security camera) and doesn’t scale. Doing it virtually is easy, safe and scales (you can even do it from anywhere in the world, including places where labor is cheap and the FBI doesn’t reach much). So this is where the focus should be. Also, this law is not respectful of privacy (the “unencrypted bar code” issue, even though if someone really wanted to systematically capture name and address from ID cards today they could take a picture of the ID and OCR it, the Real ID-mandated bar code would only make things a little easier).
On the other hand, I also can’t go along with the detractors of this law when they go beyond pointing out its shortcomings and start ranting about this creating a national ID card. While it’s true that this is what it effectively does, someone needs to explain to me why this is bad and why this would make the US a “police state”. If really such IDs are so damaging to liberties, why is it ok for every state to have them? What makes a national ID more dangerous than a state ID?
I agree that the Real ID effort is a bad cost/benefit trade off in terms of protection against terrorism. But leaving terrorism aside, we do need a robust (not necessarily perfect) way to authenticate people to access bank accounts and other similar transactions. In that respect, something like Real ID is needed. And in that context, the cost/benefit trade-off can be hugely positive if you think of how much impersonation costs and how much friction it creates in the country’s economy.
As long as we live in denial about what a Social Security number represents and as long as we can’t think sanely about terrorism, there can’t be an answer to the authentication problem.
Comments Off on We won’t get rid of SSN-based authentication anytime soon…
Filed under Everything, Identity theft, Off-topic, Security, SSN
Agriculture Department and Census Bureau to the rescue
An article in today’s New York Times reports that “the Social Security numbers of tens of thousands of people who received loans or other financial assistance from two Agriculture Department programs were disclosed for years in a publicly available database”.
Almost there folks! But tens of thousands is not enough, we need to cover everyone. The simplest effective way to dent the “identity-theft” (or more exactly “impersonation”) wave is to go beyond this first step and publish on a publicly accessible web site all social security numbers ever issued and the associated names. And get rid once and for all of the hypocritical assumption that SSN have any authentication value. We need a reliable authentication infrastructure (either publicly-run as a government service or privately-run, that’s a topic for another day) and this SSN-based comedy is preventing its emergence by giving credit issuers (and others) a cheap and easy way to pretend that they have authenticated their customers.
Over the last couple of years, I have received two alerts that my SSN and other data have been “compromised” (one when Fidelity lost a laptop containing data about everyone enrolled in HP’s retirement plan and one from a university) and my wife has received three. Doesn’t this sound like a bad joke going on for too long (and I should know about bad jokes going on for too long, they are my specialty)? And of course this doesn’t count the thousands of employees at dentist, medical offices, and many other businesses that have at some point had access to my data (and anybody else’s).
So, to the IT people at the Census Bureau I say “keep going”! But of course that’s not the reaction they had. The rest of the NY Times articles goes on with the usual hypocritical (or uninformed) lamentations about putting people’s identities at risk. “We took swift action when this was brought to our attention, and took the information down.” says an Agriculture Department spokeswoman. And of course there is the usual “credit report monitoring” offer (allowing the credit report agencies to benefit from both sides of the SSN-for-authentication debacle). Oblivious to the reality even though it manifests itself further down in the article: “The database […] is used by many federal and state agencies, by researchers, by journalists and by other private citizens to track government spending. Thousands of copies of the database exist.”
Another quote from the article: “Federal agencies are under strict obligations to limit the use of Social Security numbers as an identifier”. The SSN is a fine identifier. It’s using it as a mean of authentication that’s the problem.
[UPDATE] This is now a Slashdot thread. The comments are pouring in. Some get it (like here, and here). This one seems to get it too but then goes on to advocate dismantling the social security system which at this point is only connected by name to the issue at hand.
[UPDATED 2008/7/2: Sigh, sigh and more sigh while reading this article. The cat is so far out of the bag that a colony of mice has taken residency in it. The goal shouldn’t be to try to make the SSN hard to get, it should be to make it useless to criminals. That approach isn’t even mentioned in the article.]
Filed under Everything, Identity theft, Off-topic, Security, SSN
SML submitted to W3C
The previously released SML 1.0 and SML-IF 1.0 specifications have been submitted to W3C for standardization (actually the submission happened on 2/28 but W3C has acknowledged it today). My guess is that the fact that this announcement comes on the same day that SCA 1.0 is released is not going to decrease the confusion between these efforts.
Comments Off on SML submitted to W3C
Filed under Everything, SML, Standards, Tech
Coming up: SCA 1.0
A look at the “specifications” page of the “Open SOA” web site (the site used by the companies that created the SCA and SDO specifications) reveals a long list of specs with a release date of tomorrow. It’s like stumbling on the quarterly announcement of a publicly traded company the day before the announcement… except without the profit potential.
There is no link at this point, so no luck accessing the specifications themselves (unless one feels lucky and wants to try guessing the URLs based on those used for previously posted documents…) but we now know what they are and that they are coming out tomorrow:
- SCA Assembly Model V1.00
- SCA Policy Framework V1.00
- SCA Java Common Annotations and APIs V1.00
- SCA Java Component Implementation V1.00
- SCA Spring Component Implementation V1.00
- SCA BPEL Client and Implementation V1.00
- SCA C++ Client and Implementation V1.00
- SCA Web Services Binding V1.00
- SCA JMS Binding V1.00
- SCA EJB Session Bean Binding V1.00
The second one is the one I’ll read first.
Comments Off on Coming up: SCA 1.0
Filed under Everything, SCA, Standards, Tech
SML 1.0 is out
After taking the form of two early drafts (versions 0.5 and 0.65), the SML specification has now reached v1.0, along with its sidekick the SML-IF specification. You can find both of them at serviceml.org. This is where the happy bunch that assembled to create these specs would normally part ways. Not quite true in this case since there is related work about to be tackled by a very similar set of people (more on this later), but at least we are not going to touch SML and SML-IF anymore. They are ready for submission to a standards body where further modifications will take place (more on this later too).
Filed under Everything, SML, Standards, Tech
CMDB Federation white paper released
As reported earlier, some of movers and shakers in IT management got together last year to standardize the way to federate configuration repositories. Since then the only link available has been to a press release, which is embarrassing to say the least. Well, cmdbf.org recently came up and it is now serving a white paper that describes in more details the vision that we (the companies involved) are pursuing by this collaboration.
By linking to them from the same page, we have (by some definition) successfully federated the press release and the white paper. Way to go!
Comments Off on CMDB Federation white paper released
Filed under CMDB Federation, CMDBf, Everything, Standards