Friday, April 21, 2017

About Service Oriented Architecture on an Enterprise

In my opinion, one of the main, if not most important, goals of SOA deployment on an Enterprise is to let to the Enterprise's information systems the ability to make continues improvement and to align technology and business via the application of service-orientation. Ideally, the systems have to be maintained by the IT division of the Enterprise, not by the developers, it's not a normal situation when a new application can be connected to the Enterprise's IT landscape only by a set of consultants who work for the developer of the deployed SOA solution.

The following four conditions may ensure the achievement of stated goals:

  1. Changing the integration strategy from point-to-point to enterprise service bus (ESB).
  2. Ensuring the independence from message formats.
  3. Writing documentation well enough.
  4. Technical and business monitoring of every message lifecycle.

Enterprise service bus leveraging

The lack of point-to-point strategy is low scalability. The number of connections among the systems is described by the following formula:


If you connect together any three information systems, only three connections are needed. The point-to-point strategy wins. But, if there are 10 information systems, 45 potential connections are available, not so few. The connections must be provided on the network level (one application is able to use HTTP, but another one - RMI only, the third one uses JMS, and Nth can only put messages in a table inside a database) as well as on the data formats level (one receiver accepts XML files only, but IDOC is the most popular format for SAP, at the same time SOAP is the Web-Services format and today JSON is getting very popular). In the point-to-point case, enterprise applications should align protocols as well as message formats themselves and the related costs increase. The architects, who work on the project, have to understand the customer's strategy and have in mind how many enterprise systems are going to be connected.

Ensuring the independence from message formats

The message mapping from source formats to the formats accepted by a target system may be encapsulated within ESB. Actually, mapping is a basic capability of ESB. The question is what is the right way to organize mapping. A related term can be found in some literature about ESB, it is "Canonical Data Model". Usually, Canonical Data Model means a common format that provides an additional level of indirection between application's individual data formats. The development of a Canonical Data Model along with the corresponding mappings is a complicated process and managers wish to save money by just skipping it. When can the transformation to and from a Canonical Data Model be skipped without affecting the quality?

Firstly, if most of the recipients are systems designed by the same vendor and there is a native message format for these systems (for example SAP IDOC), the format can be employed as Canonical Data Model. Any transformations to the native message format would be performed only in adapters for source applications.

Secondly, if most of the sources are systems designed by the same vendor and there is a native message format for these systems, the format can be employed as Canonical Data Model. Any transformations to the native message format would be performed only in adapters for target applications.

Thirdly if there is an island of connected systems when the systems are only connected to each other and have no connections to the others, it may be ineffective to transform messages from a source format to canonical and then from canonical to a target one. But this structure can be fragile: if a new requirement arrives, a complex mapping will have to be designed. The decision point for implementing a canonical data model is the project roadmap. If the customer is planning to replace some enterprise applications, using the Canonical Data Model design pattern could reduce the complex of the integration architecture.

Writing documentation well enough

Unfortunately, the total entropy of the universe can only increase over time. The complexity of a large business information systems, which connects over tens endpoints, is significant increasing. The developers will choose to develop new endpoints and processes instead of adaptation existing ones. Actually this fact is a reason why the implemented SOA role is eventually eliminated and a new SOA implementation will be needed upon the new landscape. To avoid this, some technical means should be used (a service registry and repository) along with organizational measures such as the careful and well enough documentation of the integration landscape. The documentation has to include all mandatory information about every service type (utility-, application-, business- and process- services):

  • service name
  • integration style (batch, near-real time)
  • service invocation style (by trigger, Change Data Capture, etc.)
  • message exchange patterns (Request-Reply, One-Way, Pub-Sub, etc.)
  • business domain of messages
  • message formats (XML, CSV, SOAP, IDOC, etc.)
  • communication protocol (HTTP, MQ, JMS, RMI, etc.)
  • service contract definition (WSDL, WADL, Swagger, etc.)
  • security methods (authentication, authorization, cryptography, non-repudiation, etc.)
  • channel-level vs message-level security
  • confidentiality and integrity needs (WS-Security, SSL, JWT, etc.)

Separately it is necessary to visualize the business processes the services participate in, the sequence of service invocation, and the rules of the output information interpretation. The documentation should be organized into an indexed library since the entropy of the documentation is also increasing.

Technical and business monitoring of every message lifecycle

Another benefit SOA provides is the ability to ensure the end-to-end monitoring of message transmittion and processing. The end-to-end monitoring let the operation team achieve the below goals:

Meeting SLA Requirements

An operations team must be immediately notified about SLA violations by the monitoring subsystem. The team should be able to get any information related to the status of each component of the integration solution they maintain. Appropriate actions would be taken: to fix a broken connection, switch to another communication channel, solve performance issues related to a database and so on.

Service usage statistic gathering and analysis

Service usage statistic should be being gathered not only for each service invocation monitoring in order to reveal any unauthorized access or unforeseen use cases for the organization's business systems. The statistic should be being gathered also for the optimization of computing capacity usage by services. If the statistic demonstrates a service isn't used, the analysis may show the service has just been eventually excluded from all business processes. So, the plan of service decommission should be developed and implemented. Other services can be rebalanced among the on-premises or cloud resources consumed by the organization.

Message delivery monitoring and control

This control is necessary to solve problems with lost messages and figure out why some connected business application are in inconsistent states. The message passing process in the bus are broken into several stages as a minimum: registration, transmission, delivery. The development team usually is under pressure, and a number of mistakes may be made. As a result, messages might be stuck in the bus. End-to-end monitoring of every message helps to detect and analyze these issues. The monitoring generates a report providing the following information about a message:

  • message type
  • source system
  • message id in the source system
  • destination system
  • message id in the destination system (if not null)
  • message state (on which state the message is: delivered, processed, etc.)
  • error description (if any error occurs)

Message processing monitoring and control

Successful delivery a message to a business system isn't the last action of a business process. Every message has to be processed inside the system: be parsed, broken into a graph of business object, any corresponding actions should be taken, a response should be yielded as well. Some circumstances might hinder the correct handling of a message: the format is incorrect, the target business system contains inconsistent data, the business logic of message processing is incorrect and so on. If a business system silently swallows each error, no appropriate action won't be able to be taken. The service bus must be notified about results yielded by every message. There are two possible options depending on which message passing are in use.

  • Synchronous message passing. A response is yielded as soon as a message has been handled, so no additional notification to the bus is needed. The bus has to commit the response in a persistent store.

  • Asynchronous message passing. No response might be yielded at all (as example, if the Fire-and-Forget pattern is leveraged). The service bus has to be notified about the result of message processing. There should be an additional channel between business systems and the bus, the notifications would be passed by the channel. If an error occurs, the bus will be notified with the message code and description.

Please, share your opinion about SOA/ESB implementation as well as enterprise application integration. Thank you!

Would you like to give a 'Like'? Please follow me on Twitter!