Another IT event standard? I’ll believe it when I CEE it.
by William (@vambenepe on Twitter)Looks like there is yet another attempt to standardize IT events. It’s called the Common Event Expression (CEE). My cynicism would have prevented me from paying much attention to it (how many failed attempts at this do we really need?) if I hadn’t noticed an “event taxonomy” as the first deliverable listed on the home page. These days I am a sucker for the T word. So I dug around a bit and found out that they have a publicly-archived mailing list on which we can see a working draft of a CEE white paper. It looks pretty polished but it is nonetheless a working draft and I am keeping this in mind when reading it (it wouldn’t be fair to hold the group to something they haven’t yet agreed to release).
The first reassuring thing I see (in the “prior efforts” section) is that they are indeed very aware of all the proprietary log formats and all the (mostly failed) past standardization attempts. They are going into this open-eyed (read the “why should we attempt yet another log standard event” section and see if it convinces you). I should disclose that I have some history with one of these proprietary standards (and failed standardization attempts) that probably contributes to my cynicism on the topic. It took place when IBM tried to push their proprietary CBE format into WSDM, which they partially succeeded in doing (as the WSDM Event Format). This all became a moot point when WSDM stalled, but I had become pretty familiar with CBE in the process.
The major advance in CEE is that, unlike previous efforts, it separates the semantics (which they propose to capture in a taxonomy) from the representation. The paper is a bit sloppy at times (e.g. “while the syntax is unique, it can be expressed and transmitted in a number of different ways” uses, I think, “syntax” to mean “semantics”) but that’s the sense I get. That’s nice but I am not sure it goes far enough.
The best part about having a blog is that you get to give unsolicited advice, and that’s what I am about to do. If I wanted to bring real progress to the world of standardized IT logging, I would leave aside the representation part and focus on ontologies. At two levels: first, I would identify a framework for capturing ontologies. I say “identify”, not “invent”, because it has already been invented and implemented. It’s just a matter of selecting relevant parts and explaining how they apply to expressing the semantics of IT events. Then I would define a few ontologies that are applicable to IT events. Yes, plural. There isn’t one ontology for IT events. It depends both on what the events are about (networking, applications, sensors…) and what they are used for (security audit, performance analysis, change management…).
The thing about logs is that when you collect them you don’t necessarily know what they are going to be used for. Which is why you need to collect them in a way that is as close to what really happened as possible. Any transformation towards a more abstracted/general representation looses some information that may turn out to be needed. For example, messages often have several potential ID fields (transport-level, header, application logic…) and if you pick one of them to map it to the canonical messageId field you may loose the others. Let logs be captured in non-standard ways, focus on creating flexible means to attach and process common semantics on top of them.
Should I be optimistic? I look at this proposed list of CEE fields and I think “nope, they’re just going to produce another CBE” (the name similarity doesn’t help). Then I read “by eliminating subjective information, such as perceived impact or importance, sometimes seen in current log messages…” in the white paper draft and I want to kiss (metaphorically, at least until I see a photo) whoever wrote this. Because it shows an understanding of the difference between the base facts and the domain-specific interpretations. Interpretations are useful of course, but should be separated (and ideally automatically mapped to the base facts using ontology-driven rules). I especially like this example because it illustrates one of the points I tried to make during the WSDM/CBE discussions, that severity is relative. It changes based on time (e.g. a malfunction in an order-booking system might be critical towards the end of the quarter but not at the beginning) and based on the perspective of the event consumer (e.g. the disappearance of a $5 cable is trivial from an asset management perspective but critical from an operations perspective if that cable connects your production DB to the network). Not only does CBE (and, to be fair, several other log formats) consider the severity to be intrinsic to the event, it also goes out of its way to say that “it is not mutable once it is set”. Glad to see that the CEE people have a better understanding.
Another sentence that gives me both hope and fear is “another, similar approach would be to define a pseudo-language with subjects, objects, verbs, etc along with a finite set of words”. That’s on the right tracks, but why re-invent? Doesn’t it sound a lot like subject/predicate/object? CEE is hosted by MITRE which has plenty of semantic web expertise. Why not take these guys out to lunch one day and have a chat?
More thoughts on CEE (and its relationship with XDAS) on the Burton Group blog.
Let’s finish on a hopeful note. The “CEE roadmap” sees three phases of adoption for the taxonomy work. The second one is “publish a taxonomy and talk to software vendors for adoption”. The third one is “increase adoption of taxonomy across various logs; have vendors map all new log messages to a taxonomy”. Wouldn’t it be beautiful if it was that simple and free of politics? I wonder if there is a chapter about software standards in The Audacity of Hope.