Integration technologies for Cloud

2010 is seeing Cloud computing and mobile computing at their 'tipping points'. There are many startups being launched in these areas. With many applications being SaaS based hosted on Cloud, the next requirement would be how to connect them securely and reliably. What could be the right technologies for the same. This need would definitely turn into Cloud based Integration technologies - SOA technologies.

Larry J. Ellison at Oracle Open World 2009

(This is my near real-time blog. Most of the content was keyed in real time and later was little bit polished .)

Wednesday was the Larry's keynote day. Today at 2:45pm it was scheduled in the Mascone center where the Oracle Open world is being held. In the beginning, Charles Philip introduced the Diamond sponsor of the event, Infosys and its CEO Mr. S. Gopalkrishnan. His speech was focused on IT led innovation. It had a lot of good content but the presentation needed to be improved. Particularly such a large audience expects some charisma, some laughter and some real life incidents connecting to the crowd. Anyway, after the Infosys' CEO's speech on Innovation, Charlse Philip invited Larry. In his typical black sports jacket and black (crew style) sweat/T shirt, Larry appeared on the stage. Right in the beginning, LJE mentioned the topics he was planning to cover starting with Linux to Fusion Apps. Here are some of the snippets from his presentation.

It seems that flow of his presentation was from OS to dedicated box to monitoring to Fusion Apps.

So LJE started with Linux. Remember 2 years back at OOW he had introduced Oracle Linux and last year he had introduced VM. This year he unified the vision with his comment. In LJE's view, the Virtual VM and OS should blend together for easier to deploy, patch etc. Oracle VM would also support other OSs (like Windows)

Then Larry moved on to Exadata Version 2
Running Oracle enterprise Linux, Sun Oracle Database Machine . He specifically mentioned that this machine is different than the Spark based machine, demonstrated on Sunday, which is 16 times faster than similar from IBM. The customers are getting performance improvement with 10s of times not just in 10s of percentages. Exadata2 is very first database machine aimed for high performance OLTP and not for warehousing like others.
Exadata2 does random IO very rapidly. It is fastest and fault tolerant and consuming less power. 1million disk i/os per sec. Despite 72 racks, IBMS machine is not fault tolerant but very costly. For same money, Exadata is 4 times faster.

Somewhere here we saw "Kalifornia"s governator Mr. Arnold appearing on the stage! He covered a lot on innovation and technology. He mentioned how he could do so much in movies because of the technology. He then moved to clean tech and environment. The only joke that he could create a good laughter was about hands-free cell phone and his wife Maria getting three times caught breaking the hands-free-cellphone law. Anyway, he ended with praising role of Oracle and Sun in innovation, business and employment to California.


Afterward, Larry launched a crusade on IBM with the performance edge of the Exadata. During the course, he also elaborated the key contributors in improving the performance. He emphasized that it is faster than fancy-specialized In-Memory DBs. It is faster because of Grid computing, compression the data, faster movement of data, and with the usage of fast flash drives.
Another advantage of Exadata2 that he mentioned was it has simplified deployment because everything pre-cofigured.
Costs 110,000 - 350,000 - 600,000 ... Start small and grow.

In Summary, Exadata is positioned as:
The Fastest for Warehousing and OLTP, best cost/performance and Fault tolerant & scalable.

This is where Larry offered his $10 million challenge to anyone including IBM to run any apps on similar IBM box with more than half the speed that of Exadata! Read again $10 million!!

Then LJE moved on to monitoring but introduced beyond just monitoring. He introduced the concept of central Global Configuration DB that can be used for Proactive problem prevention. It can be used for Recommends patches, health checks etc. Such system would proactively notify customers based on their configuration about problems and solutions. Then help in problem resolution with automated download, validation and merge patching.

He also elaborated the usage of Business Service Metrics in EM for Detect underlying technical infrastructure failures and then use Correlation Engine to further diagnose the issue. Richard demonstrated the real time problem detection, patching and merging.

Larry's final presentation was the latest and greatest introduction to Fusion Applications. he high-lighted that Fusion apps are:
1. SaaS ready / Cloud Ready
2. Service Oriented Architecture. Because of SOA, it can be easily connected to other apps.
3. Fusion Apps: Replacement Applications and new apps are add on to Apps Unlimited!

This last aspect is a great story to motivate existing customers to upgrade or selectively adapt.

Anyway, during his presentation, Larry mentioned following applications: Talent Management., DOO, Incentive Compensation, Territory management, GRC, etc ... all these apps I provided consultancy to!!!!

While covering details, Larry emphasis following design Principals of Fusion Apps.
1. Standards based middle-ware
2. Embedded business intelligence
3. Modern and consistent UI with integrated collaboration
4. SOA
5. SaaS
Now pause at the third bullet:SOA! Just remember that is the key of the co-exist with apps-unlimited or with any other apps. Isn't it the most important aspect of the Fusion Apps? Oh, I was probably over-emphasizing the area where I played role in SOA-BPM!

Larry then invited Steve Miranda for demonstration of Fusion Apps(I like working with him. A very nice-pleasant personality and good leader). Chris Leon joined Steve for the demos. They demonstrated two scenarios. Both demonstrated a very new way of building apps. Fusion Apps uses BI natively. It also provides real-time collaboration. And I am proud to be one of the key contributor in SOAfying the Fusion Apps!

The overall show was almost 2hrs 15mins and it had engaged all the crowd. Larry does a great job especially without slides. I was hoping some more fire-shots but I guess this was enough. Anyway, Oracle already announced more than half a dozen new releases this week. That itself is overwhelming.

Typically keynote speeches are high on messages and less on details. The audience was looking for a direction and new launches. That way, Fusion Apps and EM based auto patching were the new launches. The Linux-VM and Exadata was the direction or continuation. These two in general and later in specifically suitable for setting a stage for Sun merger. Linux and VM direction was a message of continuity and consistency with previous open worlds.

Analysts also analyze Larry's speeches for his positioning with respect to the competitors. Though RedHat (Linux), Vmware(VM), SAP (Fusion Apps) are direct competitors in those respective markets, Larry did not spend time on those but focused on IBM. It was surprising that he spared SAP too. Isn't it interesting? Is SAP out of picture? Another observation, he did not comment much on cloud computing. However, he respectfully added SaaS in the context of Fusion Apps. In my opinion, such ommissions and commissions give a sense of direction for Oracle. Does it? Especially on the backdrop in Larry's interview a couple of weeks back where he ridiculed the hype of cloud-computing and then affirmed the presence of the cloud-computing. In the contrast, it was missing from today's keynote but SaaS and Linux-VM was added on top of Exadata. BTW I also noticed that Larry quickly adapts to the changes in the environment. So it won't be surprising that he would change his pitch if the environment changes.

Talking At OOW 2009 Today on SOA Suite Best Practices Based on Development Experience of Fusion Apps

Session Information

ID#: S308799
Title: Oracle SOA Suite 11g Best Practices Based on Oracle Fusion Applications Development Experiences
Date: 12-OCT-09
Time: 17:30-18:30
Venue: Hilton Hotel
Room: Yosemite B

SOA on Cloud

In today's world of Cloud computing, I would like to make a case for SOA on Cloud. It make sense for providing an agile and scalable infrastructure. SOA on Cloud would bring the setup cost of the customers down. Recently I had a discussions with a large consulting company engaged in providing SOA consultancy and Syetem integration to large enterprises. The executives really liked the idea of SOA on Cloud. That would immediately bring their cost down. In their opinion, at each client they spend 2-3 weeks in getting environment procured, set up etc before they could start SOA setup. With SOA on the cloud, they can immediately avoid all the setup time and start doing the real work. Obviously the saved time is a saving on billable hours.

Speaking at Oracle OpenWold 2009

I will be speaking at OOW 2009 on Oct 12th. The topic will be:
Oracle SOA Suite 11g Best Practices Based on Oracle Fusion Applications Development Experiences

AS11 SOA Tips: Configuring Component (Process) level properties

Oracle's SOA Suite AS11 is based on SCA (Service Component Architecture). Hence the configuration of process level properties is also done as per SCA standard. To be specifically as component properties in composite.xml

Here is an example:





For details refer to FMW SOA Documentation Appendix C

Why is it important?
The Component Properties can be configured in the production environment via Enterprise Manager and they can be accessible to the running component or component container (or service engine like BPEL)

AS11 SOA : Tracing events using event logger

If you are publishing events in AS11 and would like to trace it : A quick check to see whether event has been published or not would be using event logger.

http://:port/soa-infra/events/edn-db-log

Of course, you can also use Enterprise Manager (em) to trace the event. There are multiple ways to trace them in EM.

1. Through instance tracking on the subscribing composite,
2. Business Events page in EM ( Right click on soa-infra -> select Business Events)
3. Looking into log files ( once again via EM)

Event vs Services to initiate a BPEL process in AS11

Since As11 supports both events and services to initiate a SOA composite ( may be having a BPEL process within it), there is always a question around when to use events and when to use service invocation to start the process.

The decision can be based on the following understanding of the events and service patterns.

Events

• Events are suitable for a decoupled integration.
  
• Events are one way messaging (no request response pattern and correlation)
  
• The publisher does not know about the subscribers (there might not be subscribers too)
• The publisher is not blocked for the execution of the subscriptions
• Events support 1:n fan out (multiple subscribers for the same event)
  

Service invocations

• Service invocation is a tight integrations with the service provider
• Service invocation supports request response patterns (sync or async)
• Synchronous service invocation is blocked till the execution of the service completes.
• Async Request response pattern is often supported via WS-adderessing and correlation

Speaking at SOA World 2008

I will be speaking at SOA World 2008 next week. Here is the link to my session:

http://soaworld2008.com/general/session1108.htm?id=145

See you there if you are around.

I will be speaking on BPEL Lifecycle issues and solutions.

SDBP 2008: My Second conference in October

I will be joining Dave Chappell and Clemens Utschig at Software Development Best Practices Conference in Boston. We will be presenting some thoughts in a SOA track on Next Generation SOA. See you there... Khanderao

Quick recap on ESB, BPEL and BPM

I came across repeated queries around ESB, BPEL and BPM. Oracle's SOA suite AS11 provide all of these as a comprehensive and well integrated product suite. If someone wants to logical group the components separately to reposition them in these categories then it could be like:

ESB Enterprise Service Bus
Fabric, Mediator, Binding Components, Adapters, MDS, Policy framework and a registry
ESB essential deals with message routing and mediating between different application services. The mediation could be different transport protocols, different handshake protocols, different standards, etc. At the heart of ESB, there is a message router and transformer. ESB can be service oriented or message (in a form of an event) driven. SOA suite AS11 supports both Service and Event patterns.

BPEL Business Process Orchestration
BPEL is a service engine in SOA suite.
BPEL does process orchestration of web services. Oracle's BPEL comes up with Human task service to support workflow patterns. It also uses Oracle's Business Rules to inject dynamic decision points in a process. Though BPEL can implement most of the patterns of ESB (VETRO), it would not be good for light weight bridging between services to implement service virtualization. BPEL's key strengths is in providing a standard language for process execution with key functionalities like: correlating async interactions, providing a flexible flows, compensations for the long running processes, eventHandlers for out of the band interactions, and wait for timeout/wait activitities etc.

BPM Business Process Management
BPA for Business Process Modelling
BPM runtime for BPMN/XPDL (on roadmap after the acquisition)
Human Workflow and Advance Approval Management
BAM Business Activity Monitor for real time dashboard
Rules for dynamic busines
Web Center for Collaboration


Governance
Registry
Repository
Policy Manager
Console

Migration of Oracle's ESB from 10.1.3x to 11

With the announcement of Oracle SOA Suite AS11 aka Oracle Fusion Middleware (FMW), many folks asked about the future of ESB 10.1.3.

ESB typically provides an infrastructure for standards based integration between systems supporting different protocols. At the heart of a ESB is a routing engine and a bus. On the edges it would have adapters supporting various protocols. Such framework supports policies like securities and provides a runtime visibility via a console. The FMW AS11 does exactly same and much modular way as.

FMW AS11 does all of these and more! And it is based on emerging Service Component Architecture(SCA). SCA provides a much missing standard for ESB to assemble components- services interacting together.

In Oracle Middleware version 10.1.3, ESB and BPEL are separate runtime. With FMW AS11, BPEL and Routing Service of ESB are service engines in a much modular architecture architecture based on SCA. Mediator is a new incarnation of ESB's routing service with a lot many new functionalities. Since the basic UI of the routing service is same as 10.1.3, those who are using ESB 10.1.3 you would be instantly be able to use it.

In addition to ESB 10.1.3 functionalities, you would be pleasantly surprised to see many new features that I could get added while re-architecting ESB on a new SCA based architecture.

Here are few of them:
1. Supporting Events natively: FMW now supports both Service and Event paradigms. Mediator is a rendezvous of both service and events.
2. Mediator supports more MEP patterns including dearly missing Async Request Response.
3. Mediator supports multi-part messaging and rpc/doc style
4. Mediator provides Schematron based custom validation framework
5. Mediator provides a rules based dynamic routing framework
6. Mediator provides a better fault handling and fault policies
and more such functionalities.

Though mediator's model has been changed, you can migrate 10.1.3 project easily in AS11 by opening them in JDeveloper.

Essentially, ESB in FMW AS11 would be Mediator + SCA based service Infrastructure + Events + Policy Framework + Adapters along with common infrastructure components like MDS(Meta Data service) etc.

Following picture (presented during OOW'2007) is the best way to represent ESB and SOA suite in AS11.

Oracle SOA AS11 Preview: Config and logs

Oracle SOA AS11 Preview version has embedded oc4j Application Server hosting SOA suite.

Configuration:
You may need to change some of the configurations of the server e.g. jms, datasources, logging levels etc.

The config files are under
\system11.1.1.0.22.47.94\o.j2ee\embedded-oc4j\config
Typically you may need to change logging levels, datasource settings, or transaction timeout etc. These can be done using config files in this directory

Logging:
If you need to change the logging levels, you need to change j2ee-logging.xml from the config directory. You can increase the logging levels in this file from NOTIFICATION:1 to FINE to increase the logging.

Log Directory: diagnostic.log is under
userdir\system11.1.1.0.22.47.94\o.j2ee\embedded-oc4j\log\soa

BPM: Close loop integration between Modeling, Execution and Monitoring of Business Process

Life-cycle issues in BPM:
It is well understood that Business analysts need a business friendly tool to capture requirement, model a process and / document a process. Many modeling tools are already available and are key in any BPM strategy. The business analysts do not implement those processes. Especially it is true for "executable processes" like BPEL. IT developers or system integrators often take a business requirement and turn them into an executable business process. There is always a chance of a communication gap. Off course, a proper review would reduce chances of such reviews. However, if a business analyst needs to review BPEL process, he would neither be comfortable nor be capable of. In such scenario, it is definitely useful to have an automatic generation of BPEL process and many modeling tools generate BPEL processes.

However, the key of modeling is to be higher level and capture key business activities. A modeler can capture some key business activities, however, none of these would be or should be complete. Hence, IT developer / SI would need to add details or implement specific business as well as technical activities to make the process complete. They may add data transformations, validations, error handling, or implement specific handshaking required by internal/external partners. Some of these, details may need to surfaced to the model. Otherwise, over a period of time, the model start getting out of sync with the implemented process. Thus, round tripping is important.

While a process can be modified at the time of or after implementation by IT/SIs, the business analyst may also need to change the model to reflect a changing business need. Thus there would be changes in both the original model as well as the generated business process. This is a very valid case in a life cycle of a business process. So the key is to support merging the changes.

Oracle BPM:
In the month of September, Oracle announced a next updated version of its BPA (Business Process Analysis) Suite which supports closed loop integration between modeling and execution models. Oracle being a vendor providing a comprehensive solution in BPM-Integration space, has offered such tighter integration between various tools and by providing a standards based (BPMN) business friendly modeler, a standards(BPEL) based process designer, an execution runtime as well as monitoring environment (SOA Suite).

This new version supports thus provides "a closed-loop engineering and bi-directional sync'ing capabilities, enabling business analysts and developers to closely collaborate throughout the entire BPM life cycle using the best tools for their specific needs".

Using Oracle BPM Suite: Modeler and IT/SI experience
Using these products, a modeler can model a process, and then publish it. Using JDev based BPEL designer, an IT developer/SI can explore this published model and start with its "blue print" which is a common model between BPMN and BPEL. The BPEL process can further be refined with implementation details and would be published. If the modeler changes the process model and republishes it, the IT/SI developer would get a visual notification in his/her JDev. The developer can sync up the process, see the differences, merge the changes (comparing the new versus existing process) and implement the details for the newly added business activities related.

Seamless Integration with Monitoring:
Oracle's BPM suite also helps in monitoring. The modeler can specify (select) monitoring points, the generated BPEL would carry all the "sensor instrumentation" without a need to add them manually. These sensor data can be fed to Oracle's BAM (Business Activity Monitoring) module which can provide a real time dashboard.

The execution can potentially be analyzed and the analysis can be used in further refining the business process model, thus completing the circle of life cycle activities.

Oracle Honored with a Tech Innovator Award in the Business Enterprise

Oracle Honored with a Tech Innovator Award in the Business Enterprise Category by CMP Channel’s VARBusiness
Component of Oracle® Fusion Middleware Receives Industry Accolade

REDWOOD SHORES, Calif., 09-OCT-2007 05:05 AM Oracle today announced that it has received a CMP Channel VARBusiness 2007 Tech Innovator award in the Business Enterprise category for the Oracle SOA Suite, a component of Oracle Fusion Middleware. The fifth annual Tech Innovators Awards celebrate the most innovative products, people and personalities in the industry and highlight emerging products and technologies currently sold through the IT Channel.


Oracle SOA Suite is a comprehensive, standards-based set of middleware products used to easily build, deploy and manage Service-Oriented Architectures (SOA). The suite's hot-pluggable capabilities enable customers to take advantage of the many benefits SOAs provide, while leveraging their current middleware technologies and avoiding a costly rip and replace project. In addition, Oracle SOA Suite enables high levels of service quality with a unique Grid architecture for advanced scalability and performance using low-cost commodity hardware. Using the Suite, organizations can seamlessly connect, extend and evolve their existing IT systems to rapidly deliver new business services. Oracle SOA Suite is comprised of the best-in-class SOA components of Oracle Fusion Middleware and supports Oracle and non-Oracle application servers and messaging buses, including IBM WebSphere, BEA WebLogic and JBoss Application Servers.

SOA on Grid and relocatable BPEL processes

Dave Chappell, Author of ESB book and SOA Chief Strategist with whom I am privileged to work, and Dave Berry, ESB PM and colleague, jointly published an article covering SOA on Grid a potential next generation to provide a robust, scalable, reliable, high performing and highly available infrastructure for SOA.

It most of the obvious target use cases like:
Large payload
Active-Active clustering
Load Balancing
In memory persistence caching of DB (asynchronous write)
Performance improvement of BPEL PDS - Dehydration (process data/state storage)

But most importantly, the article introduced one important concept, we had brainstormed earlier, that is re-locatable BPEL process to minimize a data serialization. Why bring a large data across a network when a process can itself continue from a node on/closer to the node having data. e.g. Apps adapters.

Certainly doing such agile services would provide a stateful however relocatable services resulting into high performance and high availability.

BPEL processes in PeopleSoft Applications using Peopletools 8.48 (and later)

In last few days, many customers, professional service engineers and integration specialists asked about using BPEL with PeopleSoft Applications. So I am giving some high level overview of Peopletools 8.48 functionalities for integrating with BPEL.

In 2005, when I was the Enterprise Architect of Peopletools (a development and runtime platform of Peoplesoft Applications), one of the important projects we did was to add many features to enable SOA in a better way. Peopletools 8.48 provided a better Service designer to consume or provide webservices. Many new features like WS-Addressing support, WS-SE, etc aligned Peopletools 8.48 to seamlessly work with BPEL. A much better than earlier version of Peopletools 8.46 and 8.47! (both these version are also certified with BPEL 10.1.2).

Many additional features like a support to WS-A headers & correlation id, security credentials, WSIL lookup, and others, allow PeopleSoft Applications and BPEL to discover each other’s services and invoke them other synchronously or asynchronously in a secured way.

We also added some features specifically for BPEL e.g. Partnerlinks in generated WSDLs, a utility Application classes, BPELUtil, IBUtil etc. With partnerlink support, the PeopleSoft Services are easier for BPEL to consume.

Using newly introduced BPELUtil, Peoplecode developers can directly launch a BPEL process. IBUtil class provides APIs to track the process. It also provides some more utility methods to get BPEL console URL, domain, etc. In fact, Peoplesoft CRM developed a process monitor for CRM processes using these APIs in release 9.0. Additionally, BPEL console can be launched within Peoplesoft portal without re-login.

Consuming a BPEL process was made very easy in Peopletools. One can consume the service by discovering from BPEL’s WSIL, UDDI, or directly importing using wsdl URL, or from a WSDL file. A copy of the WSDL is stored in WSDL repository table. Using the service designer, the developer can select port type, operations and messages. The developer can also add some handlers and routings.

Easy Coding: The consumed BPEL process can be launched (invoked) using a very few Peoplecode statements.

Creating a Message

&payload = CreateXmlDoc(&customer);

&msg = CreateMessage(Operation.PROCESS, %IntBroker_Request);

&msg.SetXmlDoc(&payload);

Invoking a BPEL process

&response = &bpelProcess.LaunchSyncBPELProcess(&OPERATION, &msg, "", "");

Processing a response

If All(&response) Then

&responseString = &response.GenXMLString();

WinMessage(&responseString);

Else

WinMessage("Error: No reply ");

Providing Services:

Peoplesoft Component Interfaces (CI)s and Peoplecode App Classes can be easily exposed as web services. The Service Designer helps to assemble the service by defining service operations and map them one or more CIs or App Classes or Peoplecode functions (handlers). Using the Schema designer one can design Peoplsoft’s rowset or non-rowset based message or import pure XML message as a schema. The service designer abstracts the internal names of handlers and messages allowing developers to give appropriate names. After assembling a service, the developer can publish the service as a WSDL to a WSIL or UDDI repository. The WSDL is also saved in an internal WSDL repository and available for query or export.

Routing and transformations:

Many times the external service may expect or send a message, which could be different than the internal message. The Service designer allows developing and using transformations of these messages using a graphical mapper.

Peopletools supports following Request Response Patterns:

1. One way notification (Fire and Forget)
2. Synchronous Request Response
3. Asynchronous Request Response

The Async Request response is supported using WS-Addressing headers. The external WS-Addressing headers, like correlation message id, propagated to Peoplecode based application (via IBInfo).

Security: In the world of integrations, Security is very important. Different framework handles security differently. PT8.48 implements a waterfall security model. A Username token and pass code token can be set at Service operation or Node or may have a default. It can be sent encrypted, digitally signed or text. An application programmer can override these using Security override API to provide a different token. This security credentials are sent via WS-SE.

Using the Integration framework, Enterprise Applications like CRM, HCM developed many Business Processes with BPEL.

SDO
JSR-235

(Peoplesoft is a Trademark of Oracle Corp. To differentiate with other ERP applications offered by Oracle, I am specifically using Peoplesoft name.)