William Vambenepe's blog

I have nothing against Animoto. From what I know about them (mostly from John’s podcast with Brad Jefferson) they built their system, using EC2, in a very smart way.

But I do have something against their story being used to set the benchmark for infrastructure flexibility. For those who haven’t heard it five times already, the summary of “their story” is ramping up from 50 to 5000 machines in a week (according to the podcast). Or from 50 to 3500 (according to the this AWS blog entry). Whatever. If I auto-generate my load (which is mostly what they did when they decided to auto-create a custom video for each new user) I too can create the need for a thousands of machines.

This was probably a good business decision for Animoto. They got plenty of visibility at a low cost. Plus the extra publicity from being an EC2 success story (I for one would never have heard of them through their other channels). Good for them. Good for Amazon who made it possible. And who got a poster child out of it. Good for the facebookers who got to waste another 30 seconds of their time straining their eyes. Everyone is happy, no animal got hurt in the process, hurray.

That’s all good but it doesn’t mean that from now on any utility computing solution needs to support ramping up by a factor of 100 in a week. What if Animoto had been STD’ed (slashdoted, technoratied and dugg) at the same time as the Facebook burst, resulting in the need for 50,000 servers? Would 1,000 X be the new benchmark? What if a few of the sites that target the “lonely guy” demographic decided to use Animoto for… ok let’s not got there.

There are three types of user requirements. The Animoto use case is clearly not in the first category but I am not convinced it’s in the third one either.

1. The “pulled out of thin air” requirements that someone makes up on the fly to justify a feature that they’ve already decided needs to be there. Most frequently encountered in standards working groups.
2. The “it happened” requirements that assumes that because something happened sometimes somewhere it needs to be supported all the time everywhere.
3. The “it makes business sense” requirements that include a cost-value analysis. The kind that comes not from asking “would you like this” to a customer but rather “how much more would you pay for this” or “what other feature would you trade for this”.

When cloud computing succeeds (i.e. when you stop hearing about it all the time and, hopefully, we go back to calling it “utility computing”), it will be because the third category of requirements will have been identified and met. Best exemplified by the attitude of Tarus (from OpenNMS) in the latest Redmonk podcast (paraphrased): sure we’ll customize OpenNMS for cloud environments; as soon as someone pays us to do it.

Forrester’s Jean-Pierre Garbani wrote a short report last month about the current offering and future plans of Oracle’s IT management group (where I work).

As the report points out, Oracle’s IT management products don’t always enjoy a level of industry attention commensurate with the value they deliver. This report will hopefully help fix this.

Forrester: “Oracle Focuses On Business Value”.

Two months ago, HP announced the acquisition of EDS.

One month later, HP Software announced a slew of new service management products, including an updated version (7.5) of Universal CMDB (from the Mercury acquisition).

One month later (today), according to BMC (with supporting quote from an EDS exec), “EDS Asia Pacific Standardises on BMC Software Atrium CMDB to Improve Service Delivery”.

As an ex-colleague pointed out to me, the acquisition isn’t closed yet. Still.

As far as I know, the W3C is still reviewing the proposal that was made to them to create a new working group to standardize WS-Transfer, WS-ResourceTransfer, WS-Enumeration and WS-MetadataExchange. The suggested name, “Web Services Resource Access Working Group” or WS-RAWG is likely, if it sticks, to end up being shortened to WS-RAW. Which is a bit more cruel than needed. I’d say it’s simply half-baked.

There are many aspects to the specifications and features covered by the proposal. Some goodness, some badness and some ugliness. This post analyzes the good, points at the bad and hints at the ugly. Like your average family-oriented summer movie.

The good

The specifications proposed for W3C standardization describe a way to provide some generally useful features for SOAP messages. Some SOAP messages can get very long. In some cases, I know ahead of time what portion of the long messages promised by the contract (e.g. WSDL) I want. Wouldn’t it be nice, as an optimization, to let the message sender know about this so they can, if they are able to, filter down the message to just the part I want? Alternatively, maybe I do want the full response but I can’t consume it as one big message so I would like to get it in chunks.

You’ll notice that the paragraph above says nothing about “resources”. We are just talking about messaging features for SOAP messages. There are precedents for this. WS-Security can be used to encrypt a message. Any message. WS-ReliableMessaging can be used to ensure delivery of a message. Any message. These “quality of service” specifications are mostly orthogonal to the message content.

WS-RT and WS-Enumeration provide a solution to the “message filtering” and “message chunking”, respectively. But they only address them in the context of a GET-like operation. They can’t be layered on top of any SOAP message. How useful would WS-Security and WS-ReliableMessaging be if they had such a restriction?

If W3C takes on part of the work listed in the proposal, I hope they’ll do so in a way that expends the utility of these features to all SOAP messages.

And just like WS-Security and WS-ReliableMessaging, these features should be provided in a way that leverages the SOAP processing model. Such that I can judiciously use the soap:mustUnderstand header to not break existing services. If I’d like the message to be paired down but I can handle the complete message if need be, I’ll set this attribute to false. If I can’t handle the full message, I’ll set the attribute to true and I’ll get an error if the other party doesn’t understand this extension. At which point I can pick an alternative way to get the task accomplished. Sounds pretty basic but it’s amazing how often this important feature of SOAP (which heralds from and extends XML’s must-ignore semantics) is neglected and obstructed by designers of SOAP messages.

And then there is WS-MetadataExchange. While I am not a huge fan of this specification, I agree with the need for a simple, reliable way to retrieve different types of metadata for an endpoint.

So that’s the (potential) good. A flexible and generally useful way to pair-down long SOAP messages, to chunk them and to retrieve metadata for SOAP endpoints.

The bad

The bad is the whole “resource access” spin. It is not actually intrinsically bad. There are scenarios where such a pattern actually fits. But the way that pattern is being addressed by WS-RT and friends is overly generalized and overly XML-centric. By the latter I mean that it takes XML from an agreed-upon on-the-wire interchange format to an implicit metamodel (e.g. it assumes not just that you agree to exchange XML-formated data but that your model and your business logic are organized and implemented around an XML representation of the domain, which is a much more constraining requirement). I could go on and on about this, especially the use of XPath in the PUT operation. In fact I did go on and on with it, but I spun that off as a separate entry.

In the context of the W3C proposal at hand, this is bad because it burdens the generally useful features (see the “good” section above) with an unneeded and limiting formalism. Not to mention the fact that W3C kind of already has its resource access mechanism, but I’ll leave that aspect of the question to Mark and various bloggers (see a short list of relevant posts at the end of this entry).

The resource access part might be worth doing (one more time), but probably not in the same group as things like metadata discovery, message filtering and message chunking, which are not specific to “resource access” situations. And if someone is going to do this again, rather than repeating the not too useful approaches of the past, it may be good to consider alternatives.

The ugly

That’s the politics around this whole deal. There is, as you would expect, a lot more to it than meets the eye. The underlying drivers for all this have little to do with REST/WS or other architecture considerations. They have a lot to do with control. But that’s a topic for another post (maybe) when more of it can be publicly discussed.

A lot of what I describe in this post was already explained in the WS-ManagementHammer post from a couple of months ago. But that was before the W3C proposal and before WS-MetadataExchange was dragged into the deal. So I thought it might be useful to put the analysis in the context of that proposal. And BTW, this is a personal opinion, not an Oracle position (which is true in general for everything on this blog but is worth repeating specifically for this post).

WS-Management and WS-ResourceTransfer (WS-RT) both provide a mechanism to modify the XML representation of the state of a resource in a fine-grained way. The mechanisms differ a bit: WS-Management defines a SOAP header and distinguishes PUT from DELETE at the WS-Transfer operation level, while WS-RT uses the SOAP body and tunnels “modes” (remove, modify, insert) on top of the PUT WS-Transfer operation. But in their complete form both use XPath to point to any arbitrary nodeset and update it.

WS-ResourceProperties (WS-RP) takes a simpler approach. While it too supports XPath-driven retrieval of the content, it doesn’t attempt to provide an XPath-like level of flexibility when it comes to updating the content. All it offers is SET, INSERT, UPDATE and DELETE operations at the level of a property (a top-level child of the XML representation) and nothing more granular.

In this respect at least, WS-RP makes a better choice than its competitor and its aspiring successor.

First, XPath-driven updates sound easy but in fact are hard to specify. Not surprisingly, the current specifications do a pretty incomplete job at it. They often seem to assume that the XPath used to target the value to change returns only one node, but nothing guarantees this. If it picks up more than one node, do you replace all these nodes by the new values as a block (the new values get inserted once, presumably at the location of the first selected node) or do you replace each selected node by all the new values (in which case they get duplicated as needed)? Also, the specifications say nothing about what constitutes compatibility between the targeted nodes and the replacement nodes. One might assume that a “don’t be stupid” approach is all that’s needed. But there is no obvious line between “stupid” and “useful”. Does a request to replace a text node by an attribute node make sense? Not in a strongly-typed world, but a more forgiving implementation might just insert the text value of the attribute in the place of the text node to get to a valid result. What about replacing an element by a text node? Some may reject it for incompatible types but, unless the schema prevents mixed content, it may well result in a perfectly valid document. All in all, specifying a reliable way to edit XML is a pretty hairy task. Much harder than reading XML. It requires very careful considerations that have very little do with on-the-wire protocol considerations. Which is why doing this as part of a SOAP specification is a strange choice. The XQuery group is much more qualified for this. There must be a reason why that group decided to punt on this until they had taken care of the easier “read” case.

Second, it’s usually not all that useful anyway. Which is why the lack of precision in WS-Management’s specification of the fragment PUT haven’t really been a problem so far: people haven’t fully implemented that feature. A lot of the implementations are backed by a CIMOM, an MBean or some other OO store. In these stores, the exposed granularity is typically at the attribute level. The interactions used by programmers and consoles are also at that level. The XPath-driven update is then only used as a mechanism to update many properties at once (rather than going deep into individual properties) but that’s using a machine gun to kill a fly. The WS-RP approach supports these use cases without calling on XPath.

Third, XPath-driven PUT is really hard to implement unless your back-end store happens to be an XML database. You may end up having to write your own XPath parser and interpreter, an exercise during which you will face some impedance mismatches. Your back-end store may not have notions of property order for example, or attribute versus element. How do you handle these XPath instructions? And what kind of interoperability results from implementers having to make these decisions on their own? Implementing XPath selection on a GET is a lot simpler. All it assumes is that there is an XML serialization of the result, on which you can run the XPath expression before shipping it out. That XML serialization is a given in the SOAP world already. But doing an XPath-driven PUT injects XML considerations in your store itself, not just in the communication path.

Those are the practical reasons. In short, it makes the specifications at best complex and at worst non-interoperable, for a feature that is rarely needed. That should be enough already, but there are some architectural reasons to stay away too.

WS-Transfer is sometimes sold for REST over SOAP. And fragment-level WS-Transfer (what WS-Management and WS-RT do) is then REST on steroids. Sorry, not true. REST on crack if anything.

I am not a REST expert, but I know enough to understand that “everything has a URI” really means “anything meaningful has a URI”. It’s the difference between a crystal structure and a pile of mud. REST lets you interact directly with any node in the crystal, but there is a limited number of entities that are considered worthwhile of being a node. There is design involved (sorry, you can’t suddenly fire your architects, as attractive as that sounds). You can’t point to the space between two nodes in the crystal. XPath-on-top-of-WS-Transfer, on the other hand, lets you plunge your spoon anywhere in the pile of mud and scoop out whatever happens to be there.

Let’s take a look at WS-Federation (here is the latest draft), the only specification in a standard body that I know of that is currently using WS-RT. Whether it’s a wise choice or not for them, from a governance perspective, is a separate topic that I won’t cover here (answer: no. oops).

From a technical perspective, it is interesting to see how they went about using WS-RT PUT. They use it to update pseudonyms. But even though there is an XML representation for the pseudonyms, they don’t want to allow users to update any arbitrary part of that XML. So they create a specific dialect (the fed:FilterPseudonyms defined in section 6.1) that lets you, based on semantics that are meaningful in the specific domain covered by the specification, point to pseudonyms.

I believe most potential users of WS-RT PUT are in the same case as WS-Federation and are better served by a domain-specific way to identify entities of interest. At least the WS-Federation authors realized it rather than saying “great, WS-RT XPath fragment PUT gives us all this flexibility for free” and settling their implementers with the impossible task of producing interoperable implementations. Of course this begs the question of why WS-Federation uses WS-RT in the first place. A charitable interpretation is to pin this on overzealous re-use of all things WS-*. A more cynical interpretation sees this as a contrived precedent manufactured in an attempt “prove” that WS-RT provides features of general use rather than specific to the management domain.

Having described at length why XPath-driven updates aren’t as useful as they may seem, I can still think of two cases where a such a generic mechanism to modify an XML document could be useful. One is if the resource actually is a document (as opposed to having its state represented by a document). For example, a wiki page. But I haven’t exactly noticed wiki creators and users clamoring for wiki-over-SOAP, have you? The other situation is if you have a true model-driven system that is supported by a comprehensive system description and validation framework. The kind of thing that SML is trying to deliver. By using Schematron (rather than just XSD which is very limited in its expressivity beyond mere syntactical validation) to provide model validation. This would, in theory, allow the requester to validate the updated model before sending the change request. The change would still be validated on the receiver side (either explicitly or implicitly because a non-valid new model would simply fail when applied to the system), but the existence of the validation framework guarantees a high rate of successs (the sender would rarely send non-valid change requests). That’s very nice and exciting, but we don’t have this. SML is, as far as I can see, going nowhere fast in terms of adoption. Standardizing a model exchange protocol for that use case is, at this point in time, premature. Maybe one day.

You’ve just driven into Standardstown. It’s getting late and you need a place to stay. Your GPS navigation system has five listings under “accommodations”, with the following descriptions:

W3C campground

This campground provides well-equipped tents (free wi-fi throughout the camp). It has the most developed community feeling of all nearby accommodations. Every evening residents gather around a bonfire and the camp elders sing cryptic songs. At the end, the elders nod in approval of the moral of the song. Most campers don’t understand the lyrics but they like the melody. There is a recurring argument about how much soap the campground management should provide to guests. Old timers want to do away with this practice, but management is afraid that business travelers won’t patronize the camp if they are not provided with plenty of soap. The camp is located along a river, downstream from a large factory. When stuff floats down from the factory and lands on the shore of the camp, they call it a submission and thank for factory. So far, the attempts to build a clubhouse from the factory rubbish has mainly created eyesores.

OASIS housing development

This housing development is an option for accommodation because its management will give a plot of land to almost anyone who asks. More specifically, there needs to be at least three of you in the car. If you’re on your own, a common trick is to go pick up the village drunk (offer him a drink) and the village idiot (tell him you want his advice). They can usually be found on the main plaza, arguing about the requirements of imaginary users. Once you have your plot of land, the OASIS management maintains electric power, water and sewer but you can do pretty much what you want otherwise. If you just need temporary housing, you can just pitch a tent. As a result of this approach, there are several houses abandoned half-way through construction. This can make it hard to find your way to the house you are looking for. Residents typically don’t know anything about what’s going on in the house next door. You’ll find nice families living next to a crack house.

Motel DMTF

This motel is hard to find because it hides behind high walls. Even once you’re inside, there are segregated areas. Chances are your room card will give you access to the pool deck but not the clubhouse. Make a mental note of the way to the emergency exits, because there is no evacuation map on the wall (the map exists, but it’s considered confidential). We’ve heard that the best suites have a special door for direct access to the management office. After you leave, you can’t tell your friends what happened there. This review itself probably breaks some confidentiality rule.

WS-I Resort

This time-share resort is the newest development in town. By the time it got built, all the good land was taken so they had to build on land fragments leased from other hotels. The facilities are new and nice, but the owners association is dysfunctional. We’ve been told the feud started when a co-owner tried to organize a private mime show on shared land. Whatever the origin of the disagreement it has resulted in veto rules being commonly invoked, stopping most of the activities that the resort was originally planning to offer. But it remains a good option if you just need a place to sleep. The resort marketing has been pretty efficient: before doing business with you, many local companies will demand to see a receipt to show that you slept there.

Hotel ISO

Just getting a reservation there is a month-long process, so this is not an option if you’re already in town. Unfortunately, if you plan to do business with the local government you are expected to patronize this hotel. If that’s your case, the solution is to sleep in one of the other places in town and just go to this hotel for breakfast. Once there, order their breakfast special (called the “fast-track ruber stamp” which, unfortunately, tastes as bad as it sounds) and staple the breakfast receipt to your hotel bill. That should satisfy the city hall staff that they can do business with you.

I can’t shake the feeling that if Steve Vinoski, Steve Jones and Stuart Charlton had a drink together they’d actually agree on pretty much any distributed computing question that is worded in specific and unambiguous terms.

If you are not subscribed to their three blogs (and I don’t understand why you would not be if you have enough free time to read mine), here is a quick summary of the discussion so far:

Steve Vinoski writes an article critical of RPC approaches. Steve Jones doesn’t agree and explains why in a review of the article. Steve Vinoski is not impressed by the content of the review and even less by the tone. Stu sides with Steve Vinoski.

I think they all agree that, all other things equal, it is a good thing to facilitate the task of developers by providing them with intuitive interfaces. They also all agree that you can’t write distributed applications that shield the developer from the existence of a network. The key questions then boil down to:

• what degree of network awareness do you require from developers (or what degree do you award them, for a more positive spin)?
• what are the most appropriate programming constructs to expose that “optimal” degree of network awareness to the developers?

These questions don’t necessarily require words like “REST”, “RPC” and “JAXM” to be thrown around, other than merely as illustrative examples. In fact, the discussion so far seems to indicate that the questions are less likely to be resolved as long as these words are involved.

Once these questions are answered, we can compare the existing toolkits/frameworks (and yes, even architectural styles) to see which ones come closer to the ideal level of network-awareness and which ones present the most useful abstractions for that level. Or how each one can be improved to come closer to the sweetspot. Of course, there isn’t one level of network-awareness that is ideal for all cases, but my guess is that most enterprise applications are not too far apart on this.

The landscape for the post-BEA-acquisition Oracle Fusion Middleware portfolio has been publicly released. You can read a list of all the components. Tracing the history of each back to Oracle-internal developments or acquired companies is left as an exercise for the reader. There are plenty of hints, starting with some of the product names (WebLogic, Tuxedo…). The components with more generic names (SOA Suite, SOA Governance) require a little bit more digging. While the filiation might be of interest to people as a way to map the go-forward plan to current products, in the long term it doesn’t matter as much as the overall quality, consistency and integration. Which is what the stack is optimized for.

The announcement also contains enough podcasts to keep the whole family entertained during the long drive to your campground of choice over the July 4 weekend. If the kids complain, tell them it’s that or Prairie Home Companion and they’ll surrender.

[UPDATED 2008/7/15: The folks at MWD just published their analysis of the go-forward plan (free sign-up required).]

This Forbes article (via John) channels 3Tera’s Bert Armijo’s call for standardization of utility computing. He calls it “Open Cloud” and it would “allow a company’s IT systems to be shared between different cloud computing services and moved freely between them“. Bert talks a bit more about it on his blog and, while he doesn’t reference the Forbes interview (too modest?), he points to Cloudscape as the vision.

A few early thoughts on all this:

• No offense to Forbes but I wouldn’t read too much into the article. Being Forbes, they get quotes from a list of well-known people/companies (Google and Amazon spokespeople, Forrester analyst, Nick Carr). But these quotes all address the generic idea of utility computing standards, not the specifics of Bert’s project.
• Saying that “several small cloud-computing firms including Elastra and Rightscale are already on board with 3Tera’s standards group” is ambiguous. Are they on-board with specific goals and a candidate specification? Or are they on board with the general idea that it might be time to talk about some kind of standard in the general area of utility computing?
• IEEE and W3C are listed as possible hosts for the effort, but they don’t seem like a very good match for this area. I would have thought of DMTF, OASIS or even OGF first. On the face of it, DMTF might be the best place but I fear that companies like 3Tera, Rightscale and Elastra would be eaten alive by the board member companies there. It would be almost impossible for them to drive their vision to completion, unlike what they can do in an OASIS working group.
• A new consortium might be an option, but a risky and expensive one. I have sometimes wondered (after seeing sad episodes of well-meaning and capable start-ups being ripped apart by entrenched large vendors in standards groups) why VCs don’t play a more active role in standards. Standards sound like the kind of thing VCs should be helping their companies with. VC firms are pretty used to working together, jointly investing in companies. Creating a new standard consortium might be too hard for 3Tera, but if the VCs behind 3Tera, Elastra and Rightscale got together and looked at the utility computing companies in their portfolios, it might make sense to join forces on some well-scoped standardization effort that may not otherwise be given a chance in existing groups.
• I hope Bert will look into the history of DCML, a similar effort (it was about data center automation, which utility computing is not that far from once you peel away the glossy pictures) spearheaded by a few best-of-bread companies but ignored by the big boys. It didn’t really take off. If it had, utility computing standards might now be built as an update/extension of that specification. Of course DCML started as a new consortium and ended as an OASIS “member section” (a glorified working group), so this puts a grain of salt on my “create a new consortium and/or OASIS group” suggestion above.
• The effort can’t afford to be disconnected from other standards in the virtualization and IT management domains. How does the effort relate to OVF? To WS-Management? To existing modeling frameworks? That’s the main draw towards DMTF as a host.
• What’s the open source side of this effort? As John mentions during the latest Redmonk/Willis IT management podcast (starting around minute 24), there needs to a open source side to this. Actually, John thinks all you need is the open source side. Coté brings up Eucalyptus. BTW, if you want an existing combination of standards and open source, have a look at CDDLM (standard) and SmartFrog (implementation, now with EC2/S3 deployment)
• There seems to be some solid technical raw material to start from. 3Tera’s ADL, combined with Elastra’s ECML/EDML presumably capture a fair amount of field expertise already. But when you think of them as a starting point to standardization, the mindset needs to switch from “what does my product need to work” to “what will the market adopt that also helps my product to work”.
• One big question (at least from my perspective) is that of the line between infrastructure and applications. Call me biased, but I think this effort should focus on the infrastructure layer. And provide hooks to allow application-level automation to drive it.
• The other question is with regards to the management aspect of the resulting system and the role management plays in whatever standard specification comes out of Bert’s effort.

Bottom line: I applaud Bert’s efforts but I couldn’t sleep well tonight if I didn’t also warn him that “there be dragons”.

And for those who haven’t seen it yet, here is a very good document on the topic (but it is focused on big vendors, not on how smaller companies can play the standards game).

[UPDATED 2008/6/30: A couple hours after posting this, I see that Coté has just published a blog post that elaborates on his view of cloud standards. As an addition to the podcast I mentioned earlier.]

[UPDATED 2008/7/2: If you read this in your feed viewer (rather than directly on vambenepe.com) and you don't see the comments, you should go have a look. There are many clarifications and some additional insight from the best authorities on the topic. Thanks a lot to all the commenters.]

The acquisition of IONA by Progress Software has been pretty widely written about (sometimes ironically). But that’s not the only thing happening in that neck of the wood. Less widely reported (but still covered on ZDNet, here and here) was their acquisition of MindReef. For the inside perspective, head over to Dan Foody’s blog.

This is yet another confirmation of the fact that testing and IT management are getting ever closer together. And for good reasons, if you want to better integrate application management tasks across the application’s lifecycle. Other signs of this were the recent acquisition of the e-Test suite from Empirix by Oracle (driven by Oracle’s application management team, not by the JDev team) and, some time ago, the fact that HP decided to hang on to Mercury’s testing business rather than spinning it off.

From the Progress perspective, the IT management side of course comes from the earlier Actional acquisition (that company itself having previously merged with WestBridge). From my earlier standards work I have worked with people from Actional, WestBridge and MindReef and there is an impressive wealth of SOAP and WS experience in these teams. What remains to be seen is how much management value can be derived from a very “on the wire” (as opposed to deep in the implementation) view of the interaction. Another challenge for SOAP-centric vendors, which might have been a driver for these acquisitions, is realization that SOAP is going to be only one component of the integration landscape, not its foundation. It may be the JMS of B2B integration, but it won’t be its TCP/IP as was once assumed.