Lightweight ESBs, compared to traditional ESBs as their name suggests, are an emerging type of simplified integration solutions that focuses on recurring needs. They do not bring additional integration features, and intend to address integration issues in a new way instead. Well-known solutions in this category can be counted on the fingers of one hand: ApacheCamel, MuleESB and SpringIntegration for now (note the impartial alphabetical order...). The goal here is not to compare the three of them but to appreciate what for sure they have in common: their efficient and productive nature.

This article consists in a set of questions to address when dealing with applications integration. Many of these questions help indifferently to choose an exchange architecture, compare solutions or design flows. To cover the entire scope of applications integration the checklist breaks down in three parts as follows, focusing on what matters for lightweight ESBs:

• functional requirements, that means requirements for pure integration features,
• non-functional requirements as technical monitoring and deployment, and
• organisational aspects

Functional requirements

Let's start with a little reminder: a reference stack of applications integration.

Don't panic! Off course they are many ways to represent it. This one is willingly simplistic for the proof of concept: this is typically the features covered by any traditional ESB. But who is really using all of these features at a time? Here we are: getting over the pitfalls of complex and heavy Enterprise Integration solutions.

Connectivity and communication

What has to be integrated? This simple question looks stupid but any integration initiative starts with it: validate the requirements concerning connectivity. If standard protocols only are used it is relevant to look towards lightweight ESBs. Nevertheless in the case some applications have to be integrated through a proprietary protocol, lightweight ESBs, and more generally open-source ones, could be very limited. For example, a common strategy is to use the ERP integration module and the protocol that comes with it, for performance, monitoring or errors management reasons. Same logic apply for legacy systems and sector-specific standards.

Things are moving for Cloud integration, as MuleSoft announced its PaaS integration offer. A real lightweight ESB is finally available on the Cloud. Cloud integration patterns and trends were largely addressed in a previous article.

It is well known that asynchronous flows are a preferred solution in many applications integration cases. Actually each time it is possible. Message Oriented Middlewares (MOM) such as JMS or AMQP are powerful on technical aspects but are very poor on all functional requirements. To bypass these limits, asynchronous messaging can be combined with a lightweight ESB to get the best of both worlds. Lightweight ESBs can interact with a distant Message Broker...

...or embed it.

Having such a choice is useful to tweak the robustness and availability of the entire solution.

Routing and other messaging patterns

Messaging patterns have to be chosen in view of business flows to implement. I invite you to identify the required patterns flow-by-flow against the complete list of Enterprise Integration Patterns. Although some patterns are overly complex most of them are fundamental to any integration solution.

The three lightweight ESBs mentioned before are more or less strict implementations of EIP. More or less because not all patterns are supported. Note that to fill the gaps some higher level patterns can be home made from lower level patterns at the cost of ... development.

As underlined MOM are very poor concerning routing and transformation capabilities. Traditional ESBs require explicit configuration at various layers. This is where lightweight ESBs take place: bringing simplicity and efficiency by configuring only the required messaging patterns. Whether based on a Spring-XML or on a DSL configuration, lightweight ESBs rely on convention over configuration: for example an implicit protocol translator, an implicit router that does nothing special, etc.

Transformation

You may often ear "no business in the integration middleware". When it comes to data translation most organizations have well-defined rules on what to put in the integration middleware and what not. The only rule is that there are no rules! It depends on too many factors to be cast in stone and the point should be addressed systematically.

Lightweight ESBs have the same mapping capabilities as traditional ones. Therefore same issues apply: whether to put the mapping logic on applications side or on the middleware side, whether to use Canonical Data Model or not, etc.

Orchestration and more

The term orchestration actually regroups two notions:

• technical orchestrations derive from the technical design of the flow (complex routing, errors management, etc.). Technical orchestrations only consist of Pipes & Filters or basic forms of Process Manager.
• business orchestrations are top-down implementations of business processes. Technically they use both Pipes & Filters and Process Manager too, but can involve more systems and potentially occur through a long period of time.

Lightweight ESBs address technical orchestrations very well. However, only simple business orchestrations are supported as they do not provide any advanced mechanism for message correlation, hibernation, compensation nor business activity monitoring.

Orchestration is one of the most influential criteria when comparing lightweight with traditional ESBs. Once again it depends on the business value to put in the middleware layer.

The lack of a business rules engine embedded in the integration solution is not penalizing as it intends to address different scenarios. Business rules engines schould be placed closer to the business logic with the most value, that means into applications.

Some integration stacks include a Complex Event Processing (CEP) engine too. However, CEP addresses different integration scenarios. The precious time saved by the fast patterns detection (events recognition) of CEP would be nibbled by the complex events collection and notification through the various integration layers. CEP solutions provide their own API for message input and offer other forms of Message Router. That is why CEP do not require to be coupled to an ESB.

Finally, lightweight ESBs do not offer business monitoring across multiple instances of a same flow definition. At most, what can be done is defining basic SLA such as timeout and overload protection. This is consistent with the type of orchestrations addressed.

Non-functional requirements

Errors management

Although lightweight ESBs support the detection of errors (Dead Letter Channel) and anomalies (Invalid Message Channel), they do not offer any framework for resubmitting messages. This implies either resubmitting the message from the source system or performing a corrective action on the target system.

That is mainly due to the fact that there is no native Message Store on the way. This is where lightweight ESBs still have gaps to fill. An alternative is to tool up with a custom error management to reproduce more or less what traditional solutions provide natively:

The goal here is to tool up strictly with what is necessary for the errors management strategy.

Deployment

Time is coming to introduce another term: the integration framework. Both lightweight ESB and integration framework notions are very close. Integration framework could be considered as a deployment mode of a lightweight ESB. There is at least a big overlap between them.

Once embedded in an application it takes the role of an Emissary_._ Such solutions are particularly powerfull to equip a Nova (a business domain central application) with mechanisms such as Messaging Gateway and Service Activator. The integration framework then acts as the integration module of a software package. This is typically the positioning of Spring Integration.

Robustness and availability are greatly eased by embedding a lightweight ESB into an application: the integration system is not a bottleneck neither a single point of failure (SPOF) anymore. And over all there is no additional system in the infrastructure.

One limitation with embedding the middleware into an application could be the impossibility to implement flows with a many-to-many cardinality (ie. N => P flows). But in real life flows with a many-to-many cardinality are very uncommon... when they do not result from a design problem.

Lightweight ESBs can be indifferently embedded into an application or deployed as a standalone middleware. They support a large variety of light and heavy deployment technologies. The most pertinent way to deploy lightweight ESBs is using Spring. Avoiding dependency to an heavy container as JBI preserve their flexibility and ease of use.

Performance

The world (of ESBs...) is divided in two categories:

• ones with XML as the internal representation of messages, with systematic XML serialization and deserialization,
• ones with a custom binary internal representation or, depending on the context, no particular representation.

The three lightweight ESBs aforementioned belong to the second group. That means they dot not suffer from serialization overhead. Added to the fact that their internal architecture consists of less layers and that these layers are more integrated together, they offer good performances. That being said there is not much surprises: performances are ones expectable from a JEE solution.

Organisational aspects

Architectural initiative

The term lightweight ESB itself is a paradox as it is more efficient as a local services bus. Those solutions are perfect for a tactical integration project at the scale of a functional domain. The integration team is all the more reactive since it is interacting with fewer projects.

It is recommended not to define peremptory rules but to diffuse best practices instead. A common shortcut is to expand the integration middleware to the entire organisation at any cost. Often by twisting it to host any kind of flows, or reusing flows that don't have the same business meaning. Moreover the reuse rate of existing flows is significant inside a functional domain but decreases considerably beyond.

Rightly or wrongly, the choice of an ESB is often made at Enterprise level, for strong infrastructure rationalization purpose. Even if the ESB is centralized, try at least not to centralize the integration teams too much. Keep teams autonomous and reactive instead.

Technical competences

Any JEE (Spring ideally) developer can do the job. This is a major point regarding the sourcing strategy: in most cases the deployment of a commercial ESB lead to outsource the expertise. Such alternative solutions are more affordable. The cost of access to the technology is lowered: developers can use their preferred development environment and capitalize on their JEE knowledge, without learning all the specificities of a vendor solution.

Cost allocation and decision cycles

The large scale deployment of ESBs always bring the same problem of cost allocation. The allocation on an Activity Based Costing principle is very difficult for shared flows. When confining the middleware to a defined perimeter it becomes a lot easier to identify entities in charge:  functional domains themselves.

When a functional domain is responsible for its integration middleware, the time necessary to select a solution is reduced, as less actors are involved in the decision. The choice of an ESB for the entire Enterprise can take months and months. Since purchasing department and technical direction are not in the loop, a short decision cycle increases the time-to-market.

Conclusion

The main lack with lightweight ESBs is the impossibility to implement complex, long-living, business processes. That makes them inappropriate for an Enterprise top-down BPM approach. This limitation apart these light solutions cover all recurring integration requirements, in a more simple way. This is possible because they are built around integration best practices and conventions. If you can't address your issue with Enterprise Integration Patterns, I invite you to double check your requirements. Also note that high level BPM scenarios are marginal in comparison to basic integration. Hopefully, there are more lightweight solutions to come, certainly among commercial ones. The reengineering effort to migrate traditional ESBs to the Cloud already lead vendors to simplify their solutions.

ESBs are often deployed to integrate one and only one application (a shared repository, an ERP, a Nova) with the rest of the Information System. If integration middleware and application are tight coupled anyway, why not deploy both of them as a whole ? Lightweight ESBs can either be embedded into an application or deployed standalone. This is a precious option.

Beyond these technical aspects, the use of a lightweight ESB can bring many organisational benefits, in which:

• Flexibility : since integration projects are here to fill the gaps between systems, they have to go through changes faster. The flexibility of a lightweight ESB make it a good candidate for agile projects.
• Cost transparency: most organizations are unable to identify and allocate costs associated with an integration project. Deploying lightweight ESBs locally make departments aware of their responsibilities. The ESB is not just a necessary IT cost anymore.
• (Re)internalization: it is much easier for a developer to become familiar with a lightweight ESB, giving the possibility to (re)internalize such technical competences.

Lightweight ESBs offer a good trade off between the point-to-point spaghetti-ware and the over-centralized ESB. They are best suited for local initiatives at the scale of a functional domain or a business area. By keeping it tactical, the middleware does not turn into an administrative and technical bottleneck. Ultimately a motto for modern integration could be: make local successes!