This is the third in a series of articles on the emerging world of microservices PaaS and its relevance to Enterprise Application Integration. The other articles are:
This article discusses the current iPaaS market and compares and contrasts common mediation approaches for iPaaS and EAI. I’ll be considering briefly how this relates to microservices PaaS.
In 2011, Gartner introduced two new terms to the emergent cloud market. A Cloud-Enabled Integration Platform (CEIP) is an integration technology which supports and exploits private cloud hosting. An Integration Platform-as-a-Service (iPaaS) is a cloud-hosted integration platform that provides “a suite of cloud services enabling development, execution and governance of integration flows connecting any combination of on-premises and cloud-based processes, services, applications and data within individual, or across multiple, organizations”. Gartner described an iPaaS as combining “integration and governance and other capabilities…as a single, integrated infrastructure”.
At the time, Gartner was criticised for defining an iPaaS along similar lines to large on-premises integration platforms that handle mission-critical workloads within the enterprise. They seemed to envisage iPaaS as a cloud-hosted equivalent of the existing on-premises technologies, offering a correspondingly extensive list of functionality. This vision was deeply problematic for an emergent market. Apart from the huge amount of engineering this would entail, the very idea strikes at the heart of one of the central benefits of the cloud. The chief characteristic of the cloud is not technology innovation. It did not emerge from some break-through in computer science or disruptive new technology capability. The cloud, both public and private, provides familiar platforms and frameworks. It introduces greater elasticity and it can reduce costs and timescales across the application lifecycle, especially at the start of that cycle. However, one of its most disruptive characteristics, especially in the public cloud, is not technological, per se, but rather its ability to ‘democratise’ technology.
Democratisation means making technology more readily accessible to a wider group of consumers. The public cloud, by its very nature, demands this accessibility. It employs a direct cost model that requires customers to pay for what they use. The commercial proposition is to provide the widest group of customers with rich capabilities they can easily consume at a lower cost than any alternative. Public cloud vendors, therefore, have no choice. They have to democratise technology in order to build a viable commercial offering.
Democratisation is a challenge for enterprise application integration. EAI has a reputation for being difficult. The tools and products that support EAI are widely seen as complex and difficult to use. The major enterprise-level integration suites certainly contain rich technology sets. They engender a steep learning curve. However the complexity is not a simple matter of tooling. There is inherent complexity that arises from business processes themselves and from the siloed, proprietary implementation of so many applications and services. When developers experience the steep learning curve associated with integration technology, this is often as much to do with gaining an understanding of EAI patterns, approaches and techniques as it is about mastering a particular tool set.
Since Gartner introduced the term in 2011, there has been an explosion of iPaaS offerings. I had no trouble, recently, identifying over forty such services in the marketplace. My list is not exhaustive. Few of these correspond to Gartner’s original vision of cloud-based equivalents to their on-premises cousins. Instead, the market has voted with its feet by rejecting the long shopping list of functionality in favour of lightweight integration offerings with a focus on rapid development and integration of SaaS services and APIs. Gartner has had to follow this line, and now happily ranks these services in its ‘magic quadrant’.
There are, of course, many variations on a theme. There is no precise way to categorise the different approaches to iPaaS. However, we can suggest four broad categories:
When marketing iPaaS capabilities, vendors generally emphasise some or all of the following characteristics:
These characteristics play to the strength of the cloud as an environment that democratizes integration. Developers and ‘power users’ create their integrations rapidly without the need for in-depth development skills. They employ simple browser-based graphic tools rather than complex development environments. They use connectors that have a built-in understanding of their target applications and services. These connectors reduce mediation wherever possible to a set of configurable options. The solutions scale automatically to handle workloads of arbitrary size and complexity.
iPaaS is a forward-looking approach. It envisages an ever-increasing dependency on cloud-hosted SaaS services, integrated primarily via REST APIs. Most iPaaS services support JSON data representation as well as XML. Indeed, the ascendancy of REST and JSON as the basis for cloud-based service interchange is one of the chief enablers of iPaaS. They act as the cement for its foundations, providing a base level of standardisation which offers significant benefits.
This isn’t just a matter of adopting a common approach based on a minimum set of verbs, uniform resource location and human-readable data representation. The stateless and minimalist qualities of REST force SaaS development teams to think carefully about the consumption of their services by applications which are unknown at the time of development. Most SaaS services are built from the ground-up for integration. They implement APIs that are designed to be consumed by unknown clients. Where possible, they avoid the sorts of complexity that regularly arise in enterprise application integration. Where necessary, they constrain that complexity through the use of RESTful APIs and well-understood web service patterns.
One of the more striking characteristics of the evolving iPaaS world is the emphasis placed on libraries of connectors. Each connector handles mediation between some service or application and the iPaaS integration service itself. We saw in a previous post that mediation between an ‘inner’ and ‘outer’ world is a key feature of hexagonal architecture and integration services. In hexagonal architecture, connectors are adapters which may drive, or be driven by, the logic contained in a given iPaaS service instance.
Most iPaaS services provide connectors that support specific applications, services and APIs. Connector design emphasises configuration over development. This is especially true of API-centric connectors. Typically, these components will implement detailed support for API-specific methods. For RESTful services, these methods are centred on the use of the standard HTTP verbs and URL resources.
Most iPaaS services provide support for major SaaS services such as , Dropbox, Google Apps, Dynamics CRM and Workday. There may be connectors for established ERP software such as SAP and JD Edwards, social networks like Twitter, Facebook and LinkedIn. There may be support for relational and NoSQL databases, application software, vertical industry standards like SWIFT, HL7 and HIPAA, EDI standards (EDIFACT, X12, TRADACOMS, etc.,) and much more.
As well as API-centric connectors, most iPaaS services support a library of protocol adapters. Protocol adapters handle specific transport and application protocols such as FTPS, SFTP, SMTP, POP3 and AMQP. These adapters extend the range of integration scenarios that the service can address.
Connectors play a crucial role in message mediation. However, they are only part of the story. Mediation involves the processing of messages as they pass between systems, applications and services. Connectors and protocol adapters handle one end of this interchange. In terms of hexagonal architecture, they mediate directly with the external world. However, the ports and adapters of hexagonal architecture connect this external world to the application domain. Mediation needs to handle domain-specific concerns as well as connections and protocols.
Mediation frameworks have traditionally been modelled on the upper levels of the OSI Reference Model. Even where this is not explicitly the case, the OSI model still provides a useful yardstick against which mediation frameworks can be measured. Of course, we need to interpret any correspondence to the OSI model in a generous and abstract way, rather than a highly literal fashion. The OSI model is, after all, conceptual in nature.
The OSI reference model plays a foundational role in networking. It may seem strange to claim it is also central to integration. Consider, however, that networks connect different applications and services, but are not sufficient to mediate between those systems. For example, consider encryption and decryption. The network stack handles transport-level encryption (e.g., SSL/TLS) in layer 6 (the Presentation layer) of the OSI model. However, a network stack isn’t designed to handle application-specific encryption of data fields at the message level. Similarly, the network stack will handle sessions at the transport level (e.g., socket-based sessions) in layer 5 (the Session layer) in the OSI reference model. However, it won’t know how to link session management to higher-level transaction semantics, error handling, recovery mechanisms or message throttling policies.
We can think of mediation as a set of extensions to the upper layers of the OSI model. The top three layers (Session, Presentation and Application) act as a logical unit concerned with mediation between source and destination systems. The lower layers handle networked transmission of data across a physical medium. As a unit, the upper three layers interface directly with the Transport layer (layer 4). For integration purposes, we extend all four upper levels. This is because, in integration, we will need to cater, at a minimum, for protocol adaptation, including adapters based on transport-level protocols such as TCP and UDP.
With these concepts in mind, we can sketch the logical approach to mediation taken by integration tools and services. The following diagram illustrates the concerns that may be addressed by different layers of the mediation framework. In this model we can think of the connector as the bridge between the network stack and the mediation layers.
The list is by no means exhaustive, but does serve to illustrate that mediation is non-trivial. Many of the current iPaaS services do not attempt to address all of the concerns outlined above. This is often true of the Cloud-Only category identified earlier, and also of the cloud-based extensions in the Low-Fidelity Hybrid category. The Hi-Fidelity Hybrid category exhibits more variation. Some services provide rich EAI functionality that is reproduced in the cloud. Others are similar to Cloud-Only services, but provide simple on-premises agents or daemons to allow hosting of workloads behind corporate firewalls.
The richest mediation frameworks are generally associated with the CEIP category. This, in turn, reflects the fact that many products in this category are fully functional EAI products. I am, of course, generalising. There is a great deal of variation. However, the iPaaS market values simplicity, whereas the EAI market has tended to value rich, generic functionality and accept the corresponding complexity.
One example of this is batch processing. I haven’t investigated all the iPaaS offerings in depth, but I have yet to find any which offers the level of batch control I am used to in the EAI world. This is hardly surprising. With its emphasis on SaaS and REST APIs, iPaaS plays to a market in which batching, if used at all, is handled by external services through well-understood patterns and standardised interfaces. In the EAI world, message batching is fairly common, but is associated with a host of complex issues that are often handled inadequately by applications and services. EAI tools, therefore, integrate batch management deeply into their mediation frameworks alongside transaction control, persistence, recovery, failed message handling and other capabilities.
Another area where EAI tools are often richer is transformation. Almost all integration products implement a user interface to support the ‘mapper’ paradigm. However, these tools are not all equal. There is considerably variation among iPaaS services. Some provide transformation tools that rival those of EAI products. Other implement very rudimentary tools that can only handle simple maps. A surprising number of iPaaS mapping tools depend on textual scripting, rather than configurable component functionality, to do the heavy lifting. Again, I have not investigated all the iPaaS tools in depth, but I have yet to find mapping tools that are designed to handle the extremely large message types and complex mapping requirements that are sometimes imposed by vertical industry messaging standards or complex enterprise suites.
Most iPaaS tools emphasise the use of configurable connectors to speed up development and aid rapid integration. To this end, they provide connector libraries, many of which are sizeable. This is sometimes claimed to be an advantage over EAI products. The truth, of course, is more nuanced than the sales literature suggests, and it is worth considering some of the issues that arise.
Connectors can be broadly, and non-exhaustively, categorised as follows:
There is no general difference between iPaaS and EAI connectors. In both worlds, they provide broadly comparable capabilities with similar configuration-first approaches. However, the established EAI tools focus on on-premises application integration and predate the emergence of SaaS. Hence, they generally offer fewer API adapters out of the box. This is in contrast to the iPaaS world which emphasises integration via web APIs. This is one of the reasons why iPaaS products often provide larger connector libraries. However, it is not the whole story.
As well as web APIs, iPaaS connector libraries generally provide richer support for NoSQL data management systems and for distributed caching. EAI tools tend to focus more on rich support for a small number of major relational database systems.
Some iPaaS libraries provide multiple application adapters in place of a single, richer adapter. This is often the case for adapters for large ERP and CRM suites. I suspect this reflects a desire to make each adapter as easy to use as possible. Again, I can only generalise, but I think it is fair to say that EAI tools have a greater tendency towards providing a smaller number of larger, more complex application adapters.
Another area in which iPaaS services tend to differ to EAI products is the access to the libraries. With EAI tools, the emphasis has generally been on ‘out of the box’ adapters. Some EAI products have spawned sizeable ecosystems of third party and community connectors. However, these ecosystems may have no centralised catalogue or market place through which they can be discovered, purchased or downloaded. They are licenced by each vendor and supported separately to the EAI product.
The iPaaS world is striving to implement a different model, albeit with partial success. Adapters are often easily discoverable via web sites. With so many new entrants in the market, few vendors are in a position to build sizeable ecosystems of third-party and community adapters. However, some are having a degree of success. Others must invest in providing sizeable libraries of proprietary connectors. Those vendors that are establishing a wider ecosystem take care to ensure that adapters are easily discoverable. They may provide a market place for third party connectors with the possibility of generating revenue streams. Some vendors link adapter usage directly to service cost, providing greater access or usage to customer who pay higher premiums.
The centralisation and monetisation of connector ecosystems is not, by any means, a definitive characteristic of today’s iPaaS markets, but it does, I believe, point to the future, especially in the context of emerging microsystem PaaS platforms.
Connectors are configurable components which handle the interchange with external applications and systems. In hexagonal architecture, this is only part of what an ‘adapter’ must achieve. EAI tools and iPaaS services, alike, provide additional support for implementing mediation.
There is a great deal of variation in terms of implementation, but some broad themes emerge. We can sketch the outline of three such approaches. It is tempting to associate these neatly with EAI, ESB and iPaaS, and there is indeed a degree of correlation. In reality, however, this delineation is not clear.
In this model, mediation is handled through some form of pipeline processing. The pipeline is a construct that handles the sequential processing of messages emitted by or sent to a connector. We can think of these pipelines as conceptual extensions to the message channel that terminates at some message endpoint. In reality, pipelines are specialised forms of sequential workflow. The diagram below illustrates one possible arrangement for two-way interchange.
Pipelines are often invoked in the context of some configurable container or service which handles additional transport, session and application concerns, leaving the pipelines free to concentrate primarily on presentation concerns. The container or service may host or manage connectors.
Pipelines are generic constructs. They may support a flexible template approach for defining sequential processing stages, or the stages may be pre-determined. Because of their generic nature, they generally require careful attention to be given to the definition of message schemas. Some pipelines only allow configuration of built-in functionality. Others allow developers to plug a sequence of mediation components into each pipeline. Pipelines may support the use of message maps to transform messages. In an earlier post I described how messages are the primitives in EAI, and how this is linked to the perceived value of individual messages to the business. Pipeline processing corresponds well to this emphasis in EAI.
One of the main advantages of this model is that it maintains a clean distinction between mediation and application logic, in accordance with hexagonal architecture. The constrained nature of pipelines discourages developers from using mediation to implement automated business process logic and other application services. Pipelines don’t model complex flow control and are designed for per-message processing.
The container model provides a tightly-coupled and monolithic approach to handling mediation. Although it may make use of pluggable and re-usable components, the container is complex and often provides considerably more functionality than is required for any single interchange. The complexity of the mediation framework suits more centralised architectures. It may require more complex configuration and deployment than other models. It tends to emphasis development of schemas, maps and mediation components using specialised tooling.
Monolithic mediation containers do have advantages when tackling complexity. For example, it makes it very easy to implement robust transactional control over entire batches of messages, ensuring that the entire batch can succeed or fail as a unit, and that individual messages that fail can, if required, be routed to a dead letter queue or error handler as part of the single transaction co-ordinated with any external transactions as required. As we have seen, this kind of complexity is often encountered in EAI.
This approach emphasises mediation through the loosely-coupled composition of fine-grained mediation services. A decade ago, this was strongly advocated as a better alternative to pipeline processing by the ESB community, leading to controversy in the wider EAI community. Instead of centralised, monolithic, heavy-duty mediation frameworks, this approach employs a lightweight, highly distributed service container model. The monolithic mediation pipeline is refactored as a collection of independent services that can be hosted freely across the distributed environment and scaled horizontally across as many instances as required.
The approach supports easier evolution of mediation. As the business landscape changes, organisations must evolve their investment in mediation to handle new requirements quickly and cost-effectively. This is facilitated by an approach in which each mediation component is separately deployable and versionable as a discrete fine-grained service. It may provide the freedom to implement mediation functionality using a mixture of languages, tools and frameworks. However, this potential advantage is often compromised by the need to adopt a unified service container technology which, in turn, builds in dependency on a constrained set of languages, runtimes and tools.
A lightweight container model may not implement the same richness as a heavy-duty mediation container. In any case, some issues, such as robust transactional control over entire batches, is very difficult to implement in a highly distributed fashion. There is still a need to co-ordinate the invocation of the discrete services and to provide monitoring, tracking and diagnostics across an entire composition of fine-grained services.
Over the years, it’s fair to say that the original ESB vision for EAI has only been partly realised. Several ESB frameworks and products provide EAI functionality. Indeed, it has become difficult to think of ESB independently of EAI. There has been a tendency to build complex container models that mirror much of the complexity and even the monolithic nature of EAI tools. The configuration and deployment models required for EAI can be even more complex than those used in more centralised models. Like EAI products, the containers support the implementation of coarse-grained mediation services.
From a commercial perspective, ESB tools tend to licence their technology differently to EAI tools, allowing affordable distribution of solutions widely across the enterprise. This is often the major differentiator for on-premises products. ESB products favour wide distribution of mediation services. EAI products favour constrained distribution of mediation services over a small number of centralised servers.
We’ve seen that there is a tendency to move towards more monolithic, coarse-grained mediation services to handle the complexities of mediation, especially in EAI scenarios. The third approach acknowledges and embraces this. The motivation is to provide integration tools that simplify the composition of coarse-grained mediation services by moving away from configurable frameworks and the composition of services. Instead, entire interchanges of arbitrary complexity and granularity are composed graphically as single services. These are very coarse-grained services that combine three elements:
Workflow mediation has been supported in many different forms for a long time. It is commonly used in ETL (Extract, Transform and Load) tools. Some of the ESB products have adopted this approach instead of fine-grained, highly decoupled mediation services, although they also support the implementation of fine-grained services where required.
EAI products have tended to take a slightly different approach. They may implement workflow tools which can be used for mediation purposes, but cleanly separate them from mediation frameworks which are often licenced as different modules or products. The primary purpose of these tools is to implement service orchestration and automated business processes in the application domain. They may be based on BPEL or executable BPMN. These orchestration tools may have limited or no support for connectors, but can generally interact with web services, although this may be restricted to SOAP. They may interact directly with a proprietary message fabric (e.g., queue-based topics and subscriptions) which support a separate mediation framework, and which may also be licenced separately.
Most iPaaS services implement workflow mediation as their central approach. Many provide browser-based workflow composition tools, although some still require specialised IDEs. Workflow mediation favours configuration over development. It supports rapid implementation and deployment using graphic tools. In some cases, it may enable power users as well as developers. However, this comes at a price. We have seen that many iPaaS services provide simplification by avoiding some of the inherent complexity that regularly arises in EAI. We can also see that the workflow approach constitutes a monolithic, coarse-grained approach to integration.
The problem run deeper than the coarse-grained nature of the services. Workflow mediation fosters an approach that runs contrary to the insight of hexagonal architecture. It makes no clean distinction between the mediation layer and the application domain. Indeed, it is common for iPaaS vendors to promote their workflow mediation as an environment for automating business processes.
In hexagonal architecture, business processes are ‘external’ to the application domain. They are the activities that people, departments and trading partners undertake on a regular basis. They include customer-facing and back office processes as well as processes that cross organisational boundaries.
EAI addresses scenarios where organisations need to achieve the following:
EAI is more than technical integration of different applications. It is the integration of the business itself, the activities it carries out, the processes that support those activities and its interaction with customers and trading partners. Organisations that understand EAI in this way are better equipped to make appropriate decisions about integration technology and mediation approaches.
The ‘inside’ of hexagonal architecture is the locus for business process integration. It handles interchange between the activities and processes that reside on the ‘outside’. The automated processes in the application domain, together with the processes, whether automated or manual, in the external business environment, constitute the concerns of importance and interest to the business. Ports and adapters, together with the external applications, services, data stores, devices and systems, support those business concerns and capabilities.
Messages provide the core abstraction for business process integration. They are the unit of exchange across the mediation boundary. Some messages will be dictated by external applications, systems and services. Others will be defined within the application domain, independent of any specific application. These domain messages (in EAI they are often called ‘canonical’ messages) play a pivotal role when integrating business processes. They protect the investment that organisations make when integrating their processes. They do this by cleanly decoupling the integration logic from the various applications and systems used by the organisation.
Workflow mediation encourages the implementation of monolithic integration services that make no clean distinction between the ‘inside’ of hexagonal architecture and the mediation layer. While this may be acceptable in many scenarios, it does not provide the best approach to addressing enterprise integration requirements. Of course, developers can take care to ensure that they only use these tools for mediation, separating out any business process integration into independently deployable services. Merging automated business process integration logic with the mediation layer should be avoided where possible. It undermines many of the core reasons for investing in integration tooling.
iPaaS has rapidly grown to become a major theme in cloud computing and has spawned many new companies and services. iPaaS tools provide simple, rapid integration of services and applications. They focus on the integration of web APIs alongside other applications and services, and they exploit the elasticity of the public cloud to scale effectively and rapidly as required. However, from a microservices perspective, many of today’s iPaaS services violate core microservice principles and leave much to be desired.
We have already discussed the problem of workflow mediation which is central to many of the current iPaaS offerings. Workflow mediation promotes a monolithic approach which encourages inappropriate and brittle merging of application logic with the mediation layer. Other issues arise. IPaaS tools are proprietary and specialised. They support runtime environments which often limit the choice developers have to select languages and tools. They may be tied to a specific cloud platform or environment, limiting hosting choices. They are, in effect, as ‘centralised’ as EAI tools ever were, forcing developers to depend on a single proprietary platform for services on which their custom solutions depend, but which are not suitable for any purpose beyond integration.
This may seem harsh, given the appeal that iPaaS has in today’s marketplace. My purpose here is not to undermine that market, but to point out that today’s iPaaS offerings are not a panacea for integration problems. They simplify the model, but each simplification introduces additional issues. For each advantage there is a disadvantage. Like any software, iPaaS services represent a set of trade-offs. Organisations need to consider these trade-offs carefully when selecting technologies appropriate to the needs.
I claim no proven ability to foresee the future, but I suspect that the current generation of iPaaS tools will be a brief step in the evolution of cloud-centric integration. The emergence of next-generation microservices PaaS (µPaaS) platforms suggests that the current models will not prevail. A decade ago, ESB tried to move integration forward to a model that accords more closely with microservice principles. It largely failed in this endeavour. iPaaS addresses some of the complexity issues, but fails to tackle the fundamental problems. µPaaS opens up an opportunity to think through these issues afresh. It addresses some of the impediments that have afflicted earlier attempts. It shares the same goals as today’s iPaaS services, but points to a radically different and more democratised approach. These are themes I hope to explore in greater detail in future posts.