Khanderao on Emerging And Integration Technologies

Thursday, May 24, 2007

Conversations in SCA (Service Component Architecture)

Services often interact multiple times to complete a business activity. Such interactions are said to be part of a conversation. In the Database world, a series of database activities are grouped as a part of a transaction. The notion of transaction was extended to Distributed Computing too. In the world of distributed computing, many frameworks and services came up to support distributed transactions. When webservices evolved, standards like WS-Transaction and WS-Transaction Management (WS-TXM) tried to solve the transaction problem. However, a conversation is not a transaction. It does satisfy ACID test of transactions. Rather a conversation is just a way of identifying a series of interactions as a part of one larger activity. In other words, it is like correlating various service invocations related to a single business activity. Hence, conversations and transactions are handling with different constructs in SCA.

Multiple efforts have already been made to standardize conversations in Web services. A long time back, in 2002, HP submitted a specification to define a Web Service Conversation language (WSCL), which was a supposed to be supplementary to WSDL. However, it did not receive any support. Last few years parallel efforts were made at OASIS to standardize two competing specifications, WS-Context and WS-Coordination. Both of them at a very higher level defined notions of context and frameworks to manage them. ebXML supported its own way of conversational trading interactions. BEA implemented annotations based conversations for JWS. SCA’s proposal can be viewed as evolved from annotation based conversation and additional declaration of the conversation intent in policies.

The concept of conversation in SCA is very simple. A conversation may start with calling a method of conversational service (annotated as @Conversational). The conversation continues till a service reference created at start of the conversation is being used. Finally, the conversation ends with invoking a method annotated as @EndsConversation or the conversation timeouts due to being idle for a long time or continued more than a maximum age set for the conversational service. Any further attempt to participate in the “ended” conversation results into an exception, namely, ConversationEndedException.

To make this working in SCA, in addition to annotations in WSDL or Java sources, a conversational intent needs to be declared in policies for the conversational interfaces. Policy interceptors and binding components would then handle the work of populating the conversational context and interact with conversation framework. Thus the developers of services would be spared from the overload of managing conversation.

However, SCA does specify any implementation framework for managing conversation. Rightly so it does not even define a format for conversation identifier. SCA conversation intended to be complementary to various existing standards. Why re-invent the wheel? So one may use a sequence of WS-RM, or correlation id of JMS, context of WS-Context or WS-C, or MessageID of WS-Addressing as a conversation indetifier and use policy/binding implementations based on WS-Addressing, WS-RM, WS-Coordination, WS-Context for conversations.

Like BPEL, SCA assumes that conversations may also be long running. In case of BPEL, a conversation is maintained by using a concept of correlationSets, which can be mapped to a payload. Oracle’s BPEL PM additionally supports WS-addressing based correlations. BPEL does not support explicit end or aging of the conversations. SCA containers would need a framework to handle end of conversations. Meanwhile, a lightweight conversations based only on conversation ID may be sufficient for many applications.

I am attaching a simple diagram representing a conversation between two services. It is based on a business interaction to configure and purchase a PC.


Add to Technorati Favorites

Save This Page on

Monday, May 14, 2007

CEP and ECA / ESB routing services

Event Driven Architecture can be broadly classified into two distinct architectures: CEPs and ECA. Here are some basic distinctions. (some people classify into three: ECA/Simple events, Stream based and CEP thus separating streambased and CEP).

Streambased Complex Event Processing (CEP) systems process high volume of stream-based events received in a defined time window. On contrary, rule based or condition based systems respond to external situation/triggers, called as events, and typically ECAs deal with each event individually. ESBs routing service is an example of ECA.

Streams based systems typically process/joins a series of events to match a pattern or deduce an inference or predict a trend. In contrast, rule based ECAs typically follow a decision tree/table/if-then-else rules to forward the event some destinations or to trigger some actions. ECAs often deal with transformation of data and fan-outs.

Stream based system need to be optimized to give high performing required for a high volume of incoming stream of events.

In summary, stream based event processors try to detect a pattern from incoming data in a given TIME frame. Such processors most commonly do not "act" on "AN" event. They "process" a series of incoming data received in a given time frame. In other words, the stream based event processors "identifies situations" or generates events. (however, Some stream based systems work on "individual" event by setting the time window very small or via an attribute dictating the system to act on individual events as against a series of events.)

In contrast, rule based or ECAs “act” on individual events.

Labels: , ,

Add to Technorati Favorites

Save This Page on

Sunday, May 13, 2007

Complex Event Processing (CEP)

In his keynote speech at Java One, Thomas Kurian, Oracle’s vice President, covered three main facets of Oracle’s Fusion Middleware: Grid Computing, SCA based SOA and EDA.

While I have already written on SCA and SOA, in this post I will quickly introduce EDA in general and Complex Event Processing is specific.

While Oracle already offering EDA based technologies for last few years in the form of RFID- EDA, event based monitoring in BAM, and event-condition-action (ECA) in a form of ESB and Business Event System(BES), CEP is distinctly newer and an important addition.

In their research paper based on their work in Stanford University, David Luckham and Brian Frasca, defined Complex event processing (CEP) as a processing technology for extracting and analyzing information from any kid of distributed message-based system. In their work on this topic, they demonstrated use of CEP for: Casual event histories, Event Patterns, Event Filtering and Event Aggregation. They demonstrated these use cases using Rapide toolsets. A much more work in this field with "different" approaches have been done at various academic including Berkley, UC-LA, as well as industrial institutes.

In my opinion some basic attributes of a stream based CEP systems are:

Streaming related attributes 1. A very high volume 2. a stream based events 3. in a period of time

and processing related attributes (ability to apply processing logics/functions)1. correlations, 2. statistical/aggregation/set processing, 3. pattern matching or 4. predictive Analysis

Today there are many applications of CEP in enterprises from a low level monitoring to intelligent processing. Some practical and familiar use cases can be listed as:

Medical pandemic detection

Fraud detection

Intrusion detection

Algorithmic trading

Traffic hotspots detection

RFID based shipmen/arrival/sales patterns

Stock trading patterns

And many more

A couple of years back, Oracle too joined the efforts and made a significant value addition in this field. A colleague and a good friend of mine, Shailendra Mishra, is leading the work. Oracle now jointly contributing with academicians and other industries to develop SQL like standard language for a stream based event processing. This work is an extension based on academic efforts to develop a language for Continuos Query Language(CQL). When I was working with him to develop a demo for JavaOne, he also mentioned that how easily he could solve the algorithmic trading problem using our tool. Though CEP is already being used for the algorithmic trading, our distinction is in an easy and expressive(declarative) way to solve the problem. Moreover, performance and scalability are two more important aspects. I would say we have done a great work! And we would need to continue to do more.

Moreover, a much can be written and needs to be written to cover the depth of the topic, however, for this entry in todays blog, I will take a pause...

Meanwhile here are some excepts from our PR on this topic:

Complex Event Processing for EDA

Organizations are increasingly adopting EDA as a platform to effectively manage the increasing number of events generated from IT systems, business processes and physical sensors such as RFID. With this in mind, Oracle is developing new Complex Event Processing (CEP) capabilities that support high-volume, programmatic analysis of events to identify patterns and correlations across multiple heterogeneous event sources. These capabilities will complement Oracle's existing EDA offerings, such as Oracle Business Activity Monitoring, that provides real-time operational dashboards for tracking business key performance indicators, multi-channel alerting, and invoking automated or manual response actions.


Add to Technorati Favorites

Save This Page on

Tuesday, May 08, 2007

JSR-312 launched to provide JBI 2.0 : (Too?) ambitious to deliver in the given time fulfill all goals?

A technical committee (experts group) has been formed for JSR-312 to define the next version of JBI, version 2.0. Though software vendors like Sun, Borland, Iona, Tibco, Redhat, Webmethods (now Software AG) are participating in writing this specification, most of the major Application Server as well as ESB vendors are missing from the list. BEA and IBM were not in the camp of the supporters of JBI 1.0 too. Some other vendors from the JBI 1.0 are also dropped out because of their focus on SCA (now OCSA). Without a critical support from key vendors, a survial of such technologies becomes difficult(if not irrelevant). However, let me not prematurely declare a death. I would be happy to see some good results, like bridging gaps between various technologies, coming out of this JSR and such efforts.

I noticed many desired features in the wish list of this JSR. As noted in the JSR draft, the JBI 2.0 intends to be a target runtime for SCA. I had mentioned a possibility of such architecture in my previous posts. However, OSGI may too target to provide a runtime for SCA. There are some overlaps between OSGI’s Admin wire - DS specifications and metadata with SCA. Additionally OSGI provides features like lifecycle management, class loading, etc. Hence, the die-hard supporters of OSGI see OSGI to be evolving not as a runtime for SCA but replacing a need of SCA. However, OSGI's DS specifications still need to be enhanced to support enterprise features in multi-JVM environment. Interestingly JBI's JSR too mentions OSGi as “Enhancements to support full compatibility with OSGi, without necessarily requiring OSGi”. Since the Experts group is yet to meet and flush out the details, I do not give more information.

Coming back to the JBI2.0, it seems that the specification may add much wanted integration features like XSL, integrations with orchestrations (BPEL), Rules, etc to keep whatever limited i momentum it has in the market of ESB vendors where JBI1.0 got some traction with atleast few of the implementations supporting it. However, the scope and lack of technical clarity to provide a platform for some conflicting and some overlapping standards, (JBI1.0, SCA, OSGI, WS-CDL, BPEL,) and goals concerns me about a feasibility of this JSR to provide a “simple architecture in a timely manner” (as stated in JSR). I hope that in F2F meeting the expert group can have a clarity of realistic goals and a higher level architecture. If I could not participate in this JSR, I would at least keep a close watch on its technical direction.

JBI 2.0 would be targeted both for J2SE and J2EE. Leaders of JSR-312 and JSR-313 are yet to align their plans. I already raised this concern to both of both to Ron+Peter and Bill Shannon (leaders of the two JSRs)

Labels: , , , , , ,

Add to Technorati Favorites

Save This Page on