William Vambenepe's blog

Many of us have been thinking (a bit) and talking (a lot) about the relationship between Clouds and good old IT management.  John understands both sides and produced a few good posts (like this one).

Maybe it’s just a coincidence that both Hyperic and CA recently made such announcements. In any case, it gives the impression that time has come for some actual product capabilities in the area of managing Cloud-based systems.

I haven’t investigated either, so keep your slideware shields up, but this is what I read:

From Javier Soltero’s “Announcing HQ 4.0″: “It also provides the first cloud-friendly management agent which allows users to manage cloud based virtual machines securely and reliably from either inside the cloud, or from HQ 4.0 installations inside your datacenter”. John approves.

And at CA World, according to InformationWeek, “CA will announce a partnership with Amazon to provide management capabilities around Amazon’s EC2 utility computing platform, potentially including discovery of software running on EC2 instances, performance monitoring, configuration management, software deployment capabilities and provisioning”.

When someone looks into these two products (and others, soon to follow or alrady out and that I have missed), it will be interesting to see how these Cloud-friendly capabilities relate to the good old capabilities of management products: “software discovery”, “perf monitoring”, “config management”, “software deployment”, “provisioning”. That all sounds pretty familiar. Is it just a matter of pointing the old tools to an EC2 IP address? Is it all new capabilities, done in a new way? Or, more realistically, where does it land between these extrems? Where do you want them to land? It’s not so obvious.

Utility computing comes with an expectation of additional flexibility (now that is obvious). When tweaking IT management tools to address the domain, does one leave “in datacenter” capabilities the same and branch off to do cool things in the new land? Or do you raise the level of flexibility accross the board?

In other words, rather than snickering at them, maybe we should praise IT management vendors for whom the “look, I do Clouds” marketing spiel is just a repackaging of normal IT management features. Because it may mean that they’ve raised the bar on “in datacenter” automation capabilities. These Opsware and BladeLogic acquisitions have to come in somewhere, don’t they?

BTW, both of the announcements above also perpetuate the confusion between providing utility services (CA’s extended SaaS offering, Hyperic’s release of a pre-packaged Hyperic AMI) and the ability to manage Cloud-based systems. It’s all crammed in the same announcement/article because, hey, it’s all Cloud stuff.

Speaking of CA World, if I was there I would go to this session. At least for old time sake, and maybe to get some interesting ideas. Hopefully Don will blog about it after he is done presenting later today.

Microsoft’s PDC is taking place this week and more details were shared with the attendees about project Oslo, an effort announced last year to drastically improve the use of models across the application lifecycle. Some code is available (I think the Quadrant code is only for PDC attendees but the Oslo SDK is available to everyone). I am not at PDC, I didn’t see any presentation and I didn’t download any code. But Microsoft has also posted technical details on MSDN and, as far as I am concerned, that’s the most time-effective way to spend a couple of hours learning about Oslo. BTW, the way they share these early design descriptions and accept to make their evolution public is admirable.

For those who only want to spend 10 minutes rather than 2 hours, here are the thoughts that came to my mind as I was reading.

Overall I am somewhat underwhelmed, but not necessarily in a bad way. I know that’s a little schizophrenic so let me explain. After hearing a lot about how Oslo was the next big thing in modeling, it is a little surprising to read a document that can be summarized as “modeling is good, so go create some SQL tables and store them in a RDBMS”. That’s the underwhelming part. But on the other hand, it is more down to earth and practically-minded than I feared. And this is just a summary, in truth there is more than just “use SQL”.

Half of the MSDN documentation basically explains how to use SQL Server to store application models (as of today, the “Developing Models for the Metadata Store” section has only one sub-section, “SQL Server Guidelines for Modeling in the Oslo Repository“). Does this mean that all .NET applications will eventually have to carry with them a deployment of SQL Server 2008 even if they don’t use it to store the their operational data? Sure there are a few extra repository services (e.g. finer-grained change auditing) but most Oslo repository services are generic SQL Server features. That section has quite a lot of T-SQL, but it’s pretty readable. It also has a lot of dependencies on following naming conventions which makes me think that directly creating T-SQL code is not the best approach.

Fortunately there is an alternative, the “M” language. It’s a schema language with a built-in constraint mechanism. I found it more data-oriented (as opposed to resource-oriented) than I expected. Even though “each model is really a set of data structures, relationships, and constraints in serialized form“, there is a lot more support for data structures and constraints than for relationships. It’s just a foreign key. Relationships aren’t items and don’t have any property (or “field” as they’re called in “M”). For example, the relationship between a student’s enrollment record and a given class can’t have, as property, the grade that the student got for that class (as in the example in section 4.1.4 of the second LC of SML). To model this in “M” you need to create another item (e.g. “courseEnrollment”) and have a relationship from the student to that item and another one from that item to the “course” item itself. Or to replace the foreign key in the student table with a complex structure that contains both the foreign key and the properties of the relationship. At the end it has the same expressiveness potential, but in a less streamlined form. I assume Microsoft took this approach for performance reasons.

I am going on a limb here, but it may also be a difference between development-time concerns and operation-time concerns. During development (all the way to testing and packaging), you can still mostly get away with a relatively simple containment structure. You care about the components of your application and how they are packaged inside or next to one another. Sure you care about who calls who outside of the deployment unit but that’s not as core a concern as getting your class dependencies right, your tests in order and your installer configured. In fact, some of the “who calls who” bindings will be only be realized at runtime. Oslo, at least so far, clearly seems more focused on development time than operations so support for a relationship-rich model may not seem critical. At operations time, on the other hand, you don’t really care so much about how things were packaged before installation. You care a lot more about who invokes who (especially for modern distributed applications), what the network layout is, what resources a ticket is attached to, etc. The model looks a lot more like a graph with complex relationships. Something that “M” doesn’t seem ideally suited for.

Except for this caveat, I like “M”. It’s not anti-XML (you can represent values as XML if you’d like) but it avoids the “the answer is XML/XSD what is the question” approach to modeling that is sometimes a little too prevalent. “M” is a much better schema language for IT systems than XSD. I especially like its approach to types. A value is not intrinsically of a given type. A type is a condition that you happen to meet or not at the current time (”take heart little field, you can be anything you want when you grow up”). As such, you can be of several types at the same time. Refined types are potatoes inside potatoes (not sure if “M” supports definition of types as unions and/or intersection of existing types, for intersection I want to write something like”type NewType : OldType1 where this in OldType2” but there is no “this” in “M”). That approach to types (and the way constraints leverage types) is reminiscent of RDF/OWL. It’s a classification more than a typification, but I understand why they didn’t want to call it “class”. The similarities with RDF/OWL don’t go any further. As I wrote earlier, “M”is very data-focused and not resource-focused: as far as I can tell “M” types are defined syntactically, not semantically (the semantics come as a consequence). For example, I don’t think that you can assert that a given item representing a person is of type “friendly” if there is no corresponding data in the item. You’d have to first create a boolean field called “friendly” and define that those that have that field set to “true” are of type “friendly”. Unlike in RDF/OWL where you can just assert that a subject is “friendly”.

Here is another reason why you can’t have “semantics-only” types: “if you do not specify the type of a field or value, M infers a type for it“. Two things don’t sound quite right to me here. First a detail: the sentence (like others in the doc) talks about “the” type of a field of value, while there can be more than one. More importantly, what’s the point of this feature? How does it help me to have my IRC nickname classified as a post code or as a password just because it happens to be made of a compatible combination of letters and numbers? Maybe it makes sense as a storage optimization, but why does it make sense to expose this to the user?

I also like the way “extents” work. The current description of that feature is pretty limited, but based on how it is used in other parts I think one of its usages is to support a non-OO equivalent to inheritance: create two extents, one for the “superclass” and one for the “subclass” where each only contains the properties/fields defined at that level. You should get both of them in order to have the full picture (all the fields). This is, if I understand it correctly, similar to something I have been (unsuccessfully so far because “XML doesn’t do it this way”) trying to sell to the DMTF CMDBf working group: model inheritance through a set of non-overlapping records rather than dealing with a type hierarchy on record types. It’s not just that it makes relational storage easier (even though it does and that’s probably why “M” does it this way), it also makes your query/select operations a lot easier to specify and implement.

All in all (and without having gone through the exercise of defining actual models in “M”), it seems like a fine schema language (except that its dependency on the CLR base types is unpractical for users outside of the Microsoft universe) but I am not sure if it is beefy enough to be a good IT management metamodel. When the document says that “the Oslo repository provides open and flexible access to the data it contains, which enables direct access to SQL Server views of the underlying data. There are no complex data access layers or APIs” it sounds better than saying “it’s just SQL, so map your model to it and if you want relationships or type inheritance just build it on top of it and quit whining”. But it is an admission of limitation at the same time as a claim of simplicity. I also smell an assumption that LINQ will provide enough hand-holding that non-SQL-savvy developers will be ok. We’ll see.

And then there is MGrammar. Things get a little confusing at that point if you try to relate MGrammar to “M”. Actually, the FAQ states that “the M language consists of three parts: MGraph, MSchema and MGrammar“. This came a bit as a surprise to me since at that point I had finished reading (not in details but not too quickly either) the “M” documentation and I hadn’t seen these names mentioned once. Looks like there is some documentation consistency issues here, but that’s hardly surprising considering this is a “hyper-early (pre-alpha)” release as Doug Purdy puts it.

I think that everything that I have referred to as “M” above is MSchema.

MGrammar is something different altogether: it’s the source of the Domain Specific Language (DSL) references we’ve been hearing in relationship with Oslo. Technically, MGrammar is a BNF on steroids plus an automatically generated parser for your syntax. Cute. I assume that “M” (i.e. MSchema) is built as MGrammar-defined DSL but I am not sure why I would care. I am all for reuse and if someone at Microsoft thought that there was something reusable in the way they defined MSchema then it’s a good thing to expose this tool. But where does it come into play in application modeling? The last thing I want is people inventing completely independent languages to describe different domains. I am all for specialization, but a common underlying metamodel is pretty nice when you have to make sense of a whole system. I don’t see any such commonality in MGrammar: as far as I can tell it can be used to define anything from PostScript to sonnets.

From the FAQ, the connection point between MGrammar and MSchema is MGraph (MGrammar languages are parsed into an MGraph, MSchema “builds on MGraph”). That’s nice, but since neither the MSchema nor the MGrammar documentation mention MGraph I don’t really know what to make of this. David Chappell’s white paper also mentions MSchema and MGrammar but not MGraph. The introduction to the MGrammar Language Specification states that “the data that results from Mg [a.k.a. MGrammar] processing is compatible with Mg’s sister language, The Oslo Modeling Language, M, which provides a SQL-compatible schema and query language that can be used to further process the underlying information“. Compatible? I need more information here. In any case, MGrammar sounds like a fun project for a techie. Who am I to deny Microsoft engineers their fun. Jokes aside, I am probably missing something here seeing how prevalent the DSL message is in all discussions of Oslo. Look at the “highlights of this book” section for the upcoming Oslo/M book from the creators of the “M” language: half of it is about the DSL support and there must be a reason beyond pure geekery. As a side note, if you buy this book you need to understand what little shelf life it will have (I can give you a good price on a lightly-used Hailstorm/”.Net my services” specification book).

Aside from the “M” language itself, there are a few models described in the documentation. One corresponds to BPMN (actually, it says that it “closely aligns with” BPPMN 1.1, does this imply that they are not quite the same?). The fact that this model supports imports from Visio is a nice feature.

The Application model (one of the places where you can see “extents” in action) scares me a little bit because I doubt that two different people would use the same “extents” to describe the same software elements. Unless of course that’s being done for them by a pre-defined mapping to their development framework (.NET) enacted by their common development tool (Visual Studio). Which may be the assumption. Yet, the Application model is defined in generic terms, not Microsoft-specific (with a couple of slip-ups, like a WebApplicationModule being defined as a “Web application (module) implemented by IIS or WAS“. Maybe I’ll feel better about the generic applicability of this Application model when I see a full-fledged description (e.g. including relationship semantics as captured in foreign key field names) and an example.

At the bottom of that Application model, there is a lonely “Manageable” type to use if you have a LifecycleState field. This reinforces my impression that despite the claims to link development time with operational time, a lot of the focus to date has been on the former rather than the latter.

The ServiceModel model will look familiar to people familiar with SCA and is presumably complementary to the WorkflowModel and WorkflowServiceModel models, both of which are directly mapped to Windows Workflow Foundation. I guess that’s where Oslo and Dublin touch one another. I am still glad they are now clearly separated.

There is also a “Quadrant” model which concerns me a bit (it seems to be used to store customization of the Quadrant UI which, while convenient to store straight in the repository, doesn’t strike me as necessarily belonging there).

At this point, the question is not whether Microsoft can build Oslo as it is currently defined. SQL Server 2008 already exists, the usage guidelines aren’t unrealistic and even the “M-to-T-SQL” translation doesn’t seem too hard for Microsoft to implement (the SDK presumable already contains an implementation). I have no doubt they can deliver the system they describe. What I don’t know is whether and how it will be actually useful.

Describing “M” in details is good. Describing how the repository is implemented on top of SQL Server 2008 is interesting but not so relevant. What I’d like to see is a description of how all this gets used. How does it change the Visual Studio experience? How does it change the installation process/format? How does it support round-tripping between lifecycle stages (e.g. if the developer changes the workflow model, does that original BPMN model get consequently updated)? How does it relate to SLAs and policies? How does it apply to application monitoring? How does it apply to configuration management, to the change process? Etc. In short, what’s the Oslo ecosystem going to be.

These questions aren’t completely ignored in the MSDN documentation, but they are dispensed with in a couple of pages: “Application Development and Lifecycle Improvements” and “IT Operations Benefits“. The former states, for example, that “having the Oslo repository act as a central location for these models also enables a connection between the design and implementation models. This connection helps prevent these models from becoming disconnected during the development process“. Which all sounds good but is just a set of assertions that we have heard many times before (not just from Microsoft). How do “M” and the Oslo repository really make this true?

On the “IT Operations Benefits” side, things are equally blurry: “the Oslo repository can store all types of machine and application configuration data. When consistently updated, this configuration data is a catalog of the current state of all monitored machines and applications in the environment“. Notice the “when consistently updated” hand wave. That’s kind of the crux if you really want to manage across the lifecycle. How will they achieve this consistency? By centralizing all changes through a model-driven controller a la SDM/SML? Through ongoing discovery and/or change notifications? By relying on good old ITIL/MOF processes?

The FAQ declares that “having a common approach does not necessarily correlate to one physical store, but more of a federated model and we believe that some of the new Repository, along with existing investments in both Configuration Management Database (CMDB) and Team Foundation Server (TFS), will form the foundation for a common Microsoft metadata strategy and should be supported across our set of products“. OK, but who is the source of truth for application configuration data? The Oslo repository or the CMDB? Is one the desired state and the other the observed state? Does the CMDB go back to simply being a Service Desk (and if so, does the Oslo repository take on the responsibility to enforce change processes, something that requires more than the security model in Oslo)? If the CMDB is still going to use SML as its metamodel, how do you efficiently federate across such different metamodels as SML (i.e. XSD + schematron + relationships) and “M”?

Lots of questions remaining. What will Oslo have turned into in a few years? A business process design/implementation/monitoring suite (there is a strong workflow feel to many parts)? A generic drag-and-drop programming environment (”the fact that entire features are already described by models means that for a wide array of application and component categories you can start using visual tools to design and implement your components“)? A control center for end to end application management? All of the above? Nothing?

This was just a quick brain dump after reading the documents. Actually, I just realized it somehow got pretty long (congrats if you’re still reading). I hope this post is not too disorganized. Oslo is an interesting effort, but, as Microsoft is first to admit, it’s at a very early stage. I am just surprised that this first release spends so much time on the “how” rather than the “what”. Maybe it’s just because I only got my information from the MSDN documentation. We’ll see when more content from PDC finds its way online. I just want the slides, watching recorded presentations is rarely time-efficient (and you can expect them to require Silverlight).

Speaking of Silverlight, there is this new site on Oslo if you think watching some videos is worth installing Silverlight. Those screenshots don’t motivate me sufficiently.

[UPDATED 2008/10/30: Rather than going to bed I Googled around a bit and found a  post by Martin Fowler that answers some of my questions about MGrammar, MGraph and MSchema. MGraph is for instances, MSchema is for types. It answers some plumbing question, but I still have questions about expected usage and relevance to applications modeling.]

[UPDATED 2008/10/30: I also found the recordings and slides from past PDC sessions. Nice job Microsoft for this quick turnaround time, even if you require Sliverlight and/or the PPTX viewer. The sessions are:

• TL23 A Lap around "Oslo" (Doug Purdy, Vijaye Raji)
• TL27 "Oslo": The Language (Don Box, David Langworthy)
• TL18 "Oslo": Customizing and Extending the Visual Design Experience (Don Box, Florian Voss)
• TL28 "Oslo": Repository and Models (Chris Sells)

The first two sessions (deliverd Tuesday) have a replay and slides, the others should, I assume, follow soon.]

[UPDATED 2008/11/3: A nice overview of Oslo by Aaron Skonnard. Unlike most other Oslo articles over the last week, this one tries to paint the (yet-to-be-realized) full picture of the Oslo ecoystem. He mentions that "other Microsoft products and technologies are expected to build on Oslo to provide other runtimes. A few that have already been announced include Microsoft System Center (Operations Manager) and Team Foundation Server (TFS) in Visual Studio Team System". It's interesting that he qualifies System Center to be more specifically "operations manager" rather than "configuration manager" but I wouldn't read too much into it at this point.]

My ex-colleagues at HP have just released a new version of the HP Systinet SOA governance product. Congrats guys.

Just a question. What’s up with the “version 3.00″ thing? We used to talk about “v1″ and “v2″. Then came the whole “Web 2.0″ silliness and we all replaced the “v” prefix with a “dot oh” suffix. Fine. But am I now supposed to say “dot oh oh”? And, more important, where will it stop? Is Santa Claus going to be bellowing “dot oh oh oh” later this year?

Or is it the price? Three dollars?

Since versioning is a big part of SOA management, I guess HP wanted to show that they had thought extra hard about the question and reflect this in their product name. In any case, no-one beats Oracle for granular version number (for example, JDeveloper 10.1.1.0.0 was released today).

More seriously, I noted with interest mentions of BPEL and SCA support in Systinet 3.00, but I couldn’t find any specific about what this means on the HP site. Anyone has more info? Also, no mention of GIF in the release announcement?

The Oracle Enterprise Manager team is inviting customers to an application management roundtable next week in San Francisco. You’ll learn about recent application management acquisitions (Moniforce, ClearApp and e-TEST), product direction and integration strategy. What we’d like to learn in return is your thoughts, needs and requirements for application management. To that end, we’ll need you to RSVP and to prepare a 5-10 minutes presentation about your application management challenges.

Here is the agenda:

• Introduction
• Customer Presentations on Application Management
• Oracle’s Approach to Application Management
  • Real User Monitoring (Moniforce)
  • End2end Performance Monitoring (ClearApp)
  • Application Quality Management (e-TEST)
• Breakout Sessions
• Composite & SOA Application Management
  • E-Business Suite Application Management
  • Siebel Application Management
  • BRM Application Management
  • PeopleSoft Application Management

It will take place at the Four Seasons Hotel (757 Market St) from 9:00AM to 1:00PM (but don’t forget to RSVP before showing up).

You don’t have to be registered for Oracle Open World (OOW) to attend, but of course it’s been timed to be convenient for people who come to OOW.

Speaking of OOW, here is a list of all the sessions about Enterprise Manager from the conference agenda search engine. Also packaged as a nicely-formatted and chronologically-ordered PDF. For those interested in the recent application management acquisitions, check out these sessions:

About Moniforce

• S298518 (Improve Performance of Your Oracle E-Business Suite and Siebel Applications with Oracle’s Real User Experience Insight)
• S298536 (Go Beyond Web Analytics: Build Business Intelligence with Oracle Real User Experience Insight)
• S298516 (How Real User Monitoring Can Improve Application Performance: Go Beyond Web Analytics and Systems Monitoring)

About ClearApp

• S298534 (Application Transaction Management with Oracle Enterprise Manager: The Key to End-to-End Monitoring)

About e-TEST

• S298707 (Application Testing Best Practices: Real-World Customer Testimonials)
• S298706 (Optimizing Application Performance: Application Testing Suite to the Rescue)

About Auptyma

• S298534 (Application Transaction Management with Oracle Enterprise Manager: The Key to End-to-End Monitoring)
• S298524 (Application Diagnostics for DBAs: Visibility into Your Application That the Middle-Tier Administrator Cannot Provide You)
• S298525 (Diagnosing Java Application Issues in Production: Gaining Performance Insight That Even Developers Do Not Have )
• S300236 (Oracle Enterprise Manager Hands-on Lab: SOA Management and Java Application Diagnostics)

Just for fun, check out Chris Muir’s 10 things we probably wont see at OOW08. The scary part is that of these ten unlikely things the least unlikely is item #1…

BTW, I’ll be at OOW next week (probably Wednesday and Thursday) so if you plan to be there and would like to meet let me know.

There is more and more information coming out about Oslo in anticipation of the Microsoft PDC in October.

David Chappell recorded a video about it last month. More recently Doug Purdy and Don Box each posted a short description of Oslo. Don describes the goal of Oslo as “simplify the process of developing, deploying, and managing software”. But when he lists ancestor technologies to illustrate that “Microsoft has been moving in this direction for over a decade now”, they are all about development, not management: COM type libraries, .NET metadata attributes, XAML. Interesting that neither SDM nor SML gets a mention. Neither did SCA by the way, but I wasn’t really expecting that one… :-)

Maybe the I am the only one looking for a SDM/SML echo here, just because I came to hear of Oslo through the DSI angle. Am I wrong to see Oslo as an enabler for DSI? This eWeek article doesn’t have anything to do with IT management. Reading it, Oslo is all about allowing people to write code through drag and drop. Yawn. And Don Box endorses the article.

Maybe it’s just me (an IT management guy more than a software development guy) but I don’t care so much about how the application model is created. I care a lot more about what it allows you to do in terms of IT management. Please don’t make me pull out the often-quoted figure about the percentage of IT budget spent on operations versus development/licensing. The eWeek piece fails to excite me, but fortunately David Chappell’s video interview is a lot more aligned with my thinking, so I still hold hopes for Oslo as an IT management enabler. Here is my approximate transcript of an example that David provides (at around 4:20) in the video:

“If someone comes to you and says i’ve got this business process and the SLA is not being met, what do you do? You’ve got to trace this through the right business process and the right application that supports that part of the process and find the machine it runs on and maybe look at the workflow that implements it and maybe look at the services that it provides. This involves talking to business analysts, or the IT pros or the architect or the developer, all of whom have their own view of the world, their own tools, their own prospective. The repository provides a common place to store all this stuff, to link it all together, and with a visual editor to have a common tool that lets you actually go through and answer this kind of questions.”

Now you’re talking.

And if Oslo is not the new blood of DSI, then what is? The DSI story is getting dated, SML is fading in our memories and of the three parts that supposedly compose DSI (”virtualized infrastructure, design for operations, and knowledge-driven management”), only virtualization is actually represented on the list of technologies on the DSI home page. Has DSI turned into just allowing System Center to manage a hypervisor? I still hold hopes that the Oslo data is going to spice things up there. It would be good for the industry at large, not just Microsoft.

I won’t be at the PDC but it will be interesting to see what filters out of these sessions. The first session in the list adds management of hybrid application systems (hybrid as in “cloud/on-premise combination” or “software+services” as Microsoft calls it), to the long “can do” list for Oslo. Impressive, if there is some meat behind the abstract. I think this task is often overlooked in discussions around management aspects of Cloud computing (see “the new, interesting thing is going to be the IT infrastructure to manage your usage of utility computing services as well as their interactions with your in-house software” in this previous entry).

Yes, I am reading way too much into session abstracts, but while I am at it I can’t help noticing that there is a lot of SQL and very little XML/XSD/XPath mentioned there. Even though one of the presenters is Gudge, the only person I have ever met who fully understands XSD (actually even he doesn’t, I’ve seen him in the WS-I days have to refer to… his book).

Even though I am sure we’ll be told that SML can be built on top of Oslo, the SQL orientation won’t make that so easy (I want to see how to build XSD+Schematron validation on top of a relational store using Oslo’s drag and drop development tool). And it puts Microsoft on a different architectural direction from IBM, who, as far as I can tell, thinks that the world is a big XML document. Neither is the most appropriate for IT management models. I prefer a graph model and associated graph queries along the lines of SPARQL or CMDBf.

But that’s just late-night idle speculations on my part (aka “blogging”). Let’s see what comes out in October.

[UPDATED 2008/9/10: Interesting timing. Microsoft is joining OMG, home of UML and BPMN. Coming next: a submission of a "new version" of UML and BPMN that happens to contain the extensions and tweaks that Microsoft made to them in the process of implementing Oslo. This, BTW, is the final nail in the SML coffin (SML isn't even mentioned in the press release).]

It started with a checkpoint on “the state of SOA monitoring and management” by Doug McClure. A good set of questions and a good list of “usual suspects” (but how much did Actional pay to be listed twice?).

Then came this good article from AMIS’ Lucas Jellema reporting on what he learned during a recent Oracle SOA Partner event. He pokes fun at Oracle/BEA for conveniently tweaking their “this is what you need” story to align with the “this is what we offer” part (I am shocked, SHOCKED to hear that a vendor would do that, let alone my employer). But the real focus of his article is to describe the importance of design-time SOA governance (integrated with the other parts of the lifecycle). He does a good job at describing some of the value of the consolidated Oracle/BEA offering.

I couldn’t help smiling when I read this paragraph:

“It struck me that most of what applies in terms of Governance to SOA assets, also applies to other assets in any software engineering process. Trying to manage reusable components for example or even implementing a good maintenance approach for a non-SOA application is a tremendous challenge, that has many parallels with SOA Governance. And to some extent could benefit from applying a tooling infrastructure such as provided by the Enterprise Repository… Well, just a thought for now. I need to know more about the ER before jumping to conclusions.”

If my memory serves me right, the original Flashline product that BEA acquired (what became the Enterprise Repository) was just that, a generic metadata repository for software assets, not something SOA-specific. It’s ironic to see Lucas look at it now and think “hey, maybe this SOA repository can be used for non-SOA apps”. Back to the future. And BTW, Lucas is right about this applicability, as Michael Stamback soon confirmed.

Still in Oracle-land, a few days later came the news that Oracle is acquiring ClearApp. Doug’s post was more about runtime governance (which he calls monitoring/management, and I tend to agree with him even though this is fighting the tide) than design-time governance. In that sense, the ClearApp announcement is more relevant to his questions than Lucas’ post. The ClearApp capabilities fit squarely with Doug’s request for “providing the right level of business visibility into the SOA environment and more importantly the e2e business services, applications, transactions, processes and activities”, as I tried to illustrate before.

More recently, Software AG announced an OEM partnership with Actional (part of Progress) to bring runtime data to its CentraSite registry (which, I assume, comes from the Infravio acquisition by WebMethods before it itself was swallowed by Software AG).

Actional’s Dan Foody of course applauds and uses the opportunity to dispel some FUD (”Actional is tightly tied with Sonic”) and also generate some new FUD (”no vendor had even a half decent offering on both sides [design-time and runtime] of the fence”).

Neil Macehiter has a more neutral commentary on the Software AG news. His analysis ends with some questions about what this means for Amberpoint. Maybe it’s time to restart the “Microsoft might acquire Amberpoint” rumor.

Speaking of Microsoft, the drum roll is getting louder in anticipation for Oslo making its debut at the upcoming PDC. That’s a topic for another post though.

This Oslo detour is a little bit off topic, but not so much. The way Don Box and team envision that giant software model shaping up they probably picture what’s called today “SOA Governance” as just a small application that an intern can build in a week on top of the Oslo repository. Or I am exaggerating?

Unlike Dan Foody I like the approach of keeping SOA Governance closely integrated with the development and IT management infrastructures. At the cost of quoting myself (if I don’t, who will?) “it’s not just about managing Web services or Web sites, it’s about managing the whole SOA application”.

[UPDATED 2008/9/23: It looks like the relationship between CentraSite and Infravio is a little bit more complex than I assumed.]

Oracle (and more specifically the middleware and applications management part of Oracle Enterprise Manager) has just acquired ClearApp. The company is based in Mountain View (California) and their QuickVision product is a very advanced management tool for composite applications, especially BPEL-based and Portal-based applications.

More information about the acquisition is available from this page and the press release. Information about the QuickVision product can be found on the ClearApp site.

QuickVision is a very complementary addition to our existing products and the acquisitions that we have made over the last year in the application management domain. Let’s take a performance management use case to see how they relate to one another conceptually (this is not an integration roadmap, just a comparison of the features of the existing products): Oracle Real User Experience Insight (from the Moniforce acquisition) will tell you that your users are seeing a performance degradation for a specific function of your Web application. If this is a stand-alone Java application, you can go straight into the Enterprise Manager App Server Diagnostic Pack to start from a URL and analyze where processing time is spent (servlet, JSP, EJB, JDBC…). AD4J (from the Auptyma acquisition) provides deep insight into the JVM. It will give you the line number and call stack of the slow methods. For example, it might lead you to a specific database call that is taking a long time to return. You can then follow the trail deep into the database using the Oracle Database Diagnostic and Tuning packs.

But if your application is a composite application (for example one that makes use of a BPEL process to orchestrate services deployed on different application servers), then you would have a hard time finding which application server to focus on. The QuickVision product fills that gap, taking a BPEL process from its invocation point into all its successive steps and into the code that the different steps invoke. So you can see if the problem is within the BPEL execution (e.g. you loop too many times) or inside an invoked Web service. In that case, QuickVision will lead you to the class that implements that service, at which point you have all the context that you need to fire off AD4J and do a fine-grained analysis of the problematic Java code as described above.

In this scenario (and assuming that the root cause is the slowness of a database query executed by a web services that has been invoked through a BPEL process), the chain of management capabilities goes something like this:

User Experience Insight
    -> QuickVision
        -> App Server Diagnostic Pack
            -> Database management packs

A variation on this would be if the service monitored was a SOAP service as opposed to a Web page. Oracle Web Services Manager could then be used as an alternative to Real User Experience Insight to alert you that something was amiss with the application performance. The rest of the flow would be the same.

At the end, it’s not just about managing Web services or Web sites, it’s about managing the whole SOA application.

Of course, QuickVision is not limited to performance analysis, even though that’s my favorite feature. For example, I could have picked a dependency analysis scenario.

To my new colleagues joining us from ClearApp, welcome!

[UPDATED 2008/9/9: InfoQ coverage of the acquisition by Dilip Krishnan.]

Let’s put aside for now all the discussions about whether BPEL is an appropriate tool to capture a “true” business process, i.e. to implement the business logic understood by a business analyst (a topic that has been discussed at length already, including here, here, here, here, here, here and around the 5 minute mark of this podcast). Today, let’s look at it as simply another resource in a developer’s toolbox, alongside things like servlets and XML parsers. It’s a tool that can simplify the invocation of remote services (especially asynchronously), the parallelization of tasks, the definition of scoped compensation handlers, the transformation of XML, the encapsulation of key business logic and, most importantly, the reliable implementation of long-lived processes. If you need a few of these features, you might find BPEL a suitable programming tool. Plus, it refreshingly encourages handling of XML as XML (e.g. via XPath) rather than mindless code generation.

In addition to whatever developer productivity benefit you see in BPEL, there are other potential benefits form using it. They are the topic of this post and they relate to application management.

We all know that in an ideal world, no developer would release an application without providing a set of management capabilities that are carefully crafted to reflect the business logic of the application. Such that IT administrators can monitor, configure, optimize and troubleshoot the application in ways that are related to what the application really does (as opposed to generic metrics like memory, CPU and I/O metrics…).

Back in the real world, this is of course rarely the case. Enters BPEL. Just by virtue of using it in a reasonable way, and without any “just for the ops guys” metadata, BPEL provides a management model for the application. Sure it’s not as good as a hand-crafted management model, but at least it’s there. And it has some pretty compelling properties:

• It feeds directly from the metadata used by the runtime, so it is guaranteed to be accurate (unlike metadata that is created specifically for management but has no role in the actual runtime).
• It shows what external services the application depends on. Of course there is no guarantee that all remote invocation will be represented in the BPEL process, but since that’s a strength of BPEL it is reasonable to expect that it provides a good view of application dependencies (to be complemented, of course, by the application infrastructure dependencies like the database and the BPEL engine itself…). Remote invocations are a common point of failure and/or performance problems so they are a first class citizen of an application management model.
• It explicitly captures process instances. No more jumping from one database table to another (assuming you even know where to look) to try to get a sense of the current overall status. The BPEL instances show the number of in-flight transactions in the application. It is also easy to compare the initialization and termination rates to see the trend.
• It provides a horizontal segmentation of the processing tasks (via the BPEL activities) that is a good complement to the vertical segmentation often offered by application management tools (e.g. time spent in the database, time spent waiting on I/O, etc…).
• It makes explicit certain exception conditions.

All these only make use of very basic aspects of BPEL: the enumeration of PartnerLinks, the notion of a process instance, the existence of activities, the fault/compensation/termination handlers. A fair amount of visibility into the health of the application can be derived form this alone. I am not making fancy assumptions about the management tool being able to make sense of the routing logic in the process or of the correlation rules. I am not assuming that the BPEL engine provides ways to control individual process instances. I am not assuming that the name attributes of certain elements (e.g. PartnerLink, variable) convey semantics that could help the administrator understand some of the semantics of the application.

At the end, it’s not about managing BPEL, it’s about managing an application that uses BPEL.

My point is not to push everyone to write any application as a BPEL process (or a set of them) as a way to get a great management infrastructure for free. But if BPEL is a potential choice for the application, then it’s worth considering those extra benefits in the “pros and cons” analysis. And if you have already decided to use BPEL, it may be worth looking into what management dividends you can harvest from this choice. Of course your mileage may vary depending on how manageable your BPEL infrastructure is. Hint hint…

A few related links. Todd Biske has also written about the management value of BPEL, here and here. A similar analysis can be applied to SCA, but at this point in time there are many more applications out there that use BPEL than SCA, making the former more relevant. I briefly described the SCA side of the equation in an earlier exchange with David Chappell. That discussion is summarized here (including a pointer to David’s original piece). In an earlier post, I touched on the manageability potenial of other sources of application metadata, like OGSi and Spring (in addition to SCA and BPEL). Jean-Jacques Dubray provided additional context at InfoQ.

[UPDATED 2008/9/2: Based on this announcement, I can add one more hint.]

Forrester’s Jean-Pierre Garbani wrote a short report last month about the current offering and future plans of Oracle’s IT management group (where I work).

As the report points out, Oracle’s IT management products don’t always enjoy a level of industry attention commensurate with the value they deliver. This report will hopefully help fix this.

Forrester: “Oracle Focuses On Business Value”.

One of the most repetitive tasks when I was evangelizing WSDM was to explain the difference between the MUWS and MOWS specifications (the sum of which composes the entire WSDM body of work). MUWS (management using web services) describes how to use Web services to expose manageability capabilities of potentially any resource (a server, an application, a toaster…). MOWS (management of web services) defines a monitoring and control model for resources that are Web services themselves (so you can measure the number of messages received for example).

I ended up sounding like a cow when I was presenting. A retarded cow even, since my French accent forced me to say it slowly so people could hear the difference.

In retrospect, we should not have tried to tackle both in the same group. And not just because my dignity was bruised. It was a distraction inside the working group, and a source of confusion outside of it. We should have focused on MUWS (as WS-Management did) and possibly created a protocol-independent monitoring/control model for Web services separately. Something that, BTW, is still missing today.

I am being reminded of this MUWS vs. MOWS state of affair these days, when the topics of SaaS and IT management meet, often under the term “SaaS management”. By that, some people mean “delivering IT management as a hosted service, rather than running the management software in the same datacenter as the application”. Other mean “managing, using an on-premise deployment of the management software, a business application that is being delivered as a service (e.g. Oracle CRM On Demand), along with other local IT resources”. The latter is what I was talking about in this post. And sometimes it’s both at the same time (the business application is delivered as a service along with a hosted management console for status/issues/requests…). Not to mention the extra dimension of providing IT management to the administrators in charge of running a multi-tenant application in a SaaS scenario (instead of meeting the needs of their customer’s administrators).

All of these scenarios are valid. So far, we don’t have good names for them. And the MUWS/MOWS experience shows that good names matter. IMaaS (IT Management as a Service) and MoSaaS (Management of Software as a Service) won’t cut it.

[UPDATED 2008/6/23: This seems to be an example of MoSaaS (or rather MoIaaS) delivered through IMaaS. I am subjecting you to such an awful-sounding sentence as a way drive home the need for better names. The real value of course will come when these capabilities are delivered alongside (and integrated with) all your IT management capabilities. John has a nice analysis that lets some air out of the fluff.]