Wednesday, July 12, 2017

How We Added a Spoon of Agility to a Large Oracle Service Bus Project

Several months ago I already shared how to ensure guaranteed delivery for 5.000.000 messages handled per day by Oracle Service Bus (OSB). In the case, OSB is heavy used to integrate a customer relationship management application (the CRM) in a solid environment with many regional billing systems, provisioning, ERP and ordering management system (OMS), etc. Also, the integration solution is deployed across four ex-standalone corporations, each of which has its own IT infrastructure, and this circumstance makes the project more and more interesting.

The common problem from a manager's point of view is to provide consistency among services deployed on the bus, the main client application (the CRM), and numerous 3rd party applications. Also, there is a time lag in development and end up to production these parts of the infrastructure. And the development team forever must have guaranteed stable and ready to deploy code of the service bus.

The situation is usual for almost all integration projects in the World. The releases of an enterprise service bus must be delivered in a reliable way, no one integration flow (neither the client -> service bus nor the service bus -> 3rd party application communication) must fail after the release is deployed to the production environment.

Let me explain the features of our project.
  1. A one pizza size development team, not more than 5 developers are there.

  2. Hovewer, the team has developed and delivered 13 services and 46 adapters to a wide range of enterprise applications. The services provide very different business logic - from just wrapping a stored procedure into a SOAP-based web service till a complex orchestration employing the Splig-Join component of Oracle Service Bus.

  3. SVN is utilized as a source code management tool.

  4. Oracle Service Bus out of the box provides no capability for service versioning, so it's impossible to expose numerous versions of the same service simultaneously. Although the main service client, CRM application, is developed by a team working for the same company, we are able to deliver releases of the service bus regardless of this capability.

  5. The development process is based on Agile. Sometimes it becomes a reason of inconsistent builds. In fact, this article is about how to eliminate the impact of any agile methodology on the consistency of the enterprise service bus.

The approach used for development and delivery each service bus release is based upon the above features. A service bus release is an update for the production environment applied only after a scrum sprint is completed, regression tests are run, and all found bugs are fixed.

  1. All new code is borne in trunk, a new branch is created from trunk for each release, then the team spends some time to make the code stable (all tests have to run successfully). The bus can be built entirely (all services are ready to be deployed as one deployment unit - a large jar-file, the services must support all contracts provided or consumed by the CRM and the 3rd-party applications) as well as particularly, only required (changed during the sprint) adapter and services get into the deployment unit. The second option is needed because the Oracle Service Bus export mechanism for and earlier takes a lot of time, up to numerous hours, for the export of the entire bus code. It's not a good idea to waste hours just for making an update of two-three services.

  2. Any changes to the code of the service bus and CRM services interfaces must be synchronized with each other, the following simple rules help to solve this problem.
    • The CRM development team accepts an integration task only if any required interfaces are already implemented on the service bus.

    • The changes is actualized on the CRM if the appropriate service on the service bus is developed and tested.

    • We always like to avoid any architecture refactorings if they aren't able to be completed on the service bus and inside the CRM during a sprint. If a refactoring must be performed, for example, the routing logic requires a new parameter from a client, the refactoring is still performed but backward compatibility is taken into account.

    • It is worth noting, we leverage a number of universal across all services components: the canonical data model and routing table. Also, the CRM invokes the canonical services directly without any adapters. In this way, there is a case when a change inside the canonical data model might break the compatibility of service interfaces. The CRM development team should regenerate the code of their service clients before the issue of a release. So far we have had no problems with this approach, there were some issues when the service bus development team forgot to update a WSDL file on the bus.

  3. The synchronization of development and deployment the service bus and 3rd party applications is ensured through the following measures:
    • A new task can be assigned to the service bus development team only if the appropriate service (an Oracle Database package, web-service, or RESTful service) is developed and deployed on the SIT environment.

    • It is considered by default, as soon as an adapter is ready to be deployed on the production environment of the service bus, the appropriate 3rd party application will expose all required integration points. From my experience, that assumption is correct in 90% cases. If some changes were applied but the 3rd party application wasn't ready to support them and the support was delayed, the changes would be dropped in a separate branch. You can tag the code in the SCM with a readable name to find your work quickly when the 3rd party application is ready.

  4. New services are developed on the service bus before they are started being consumed by CRM or the 3rd party software. At this time, the CRM development team might implement a new client for a new service which is under construction on the service bus. In the case, the new client won't be activated and the service won't be deployed. This approach makes very important the principle of service decoupling inside the service bus, services shouldn't depend on every change committed to others, especially if the others aren't ready for deployment. Within our Oracle Service Bus project every proxy services invoke adapters using dynamic routing even if there is exactly one available destination for the service.

  5. Because each time we deploy to the production environment only a subset of services, a detailed changelog is essential. A today assigned release engineer (someone from the service bus development team) should know and understand which services and adapters have been changed and must be deployed. This kind of update process bring some new risks, if a service or adapter ( especially an adapter) is missed, an overall release will fail. In the case, the changelog is a list of scrum user stories containing an integration task. Jira is employed as a task management tool, each issue has a detailed description and each appropriate service and adapter are marked there. A release untested artefacts are included in is considered as untested at all and must not be deployed in the production environment. All changed on the service bus are equal to changed inside the CRM or 3rd party software.

  6. The set of the service bus resources which are ready to be deployed in the production environment should be necessarily tested. If there is a stub for 3rd party application, the corresponding regression tests have to be developed. As a rule, we have a test instance of each application connected to the service bus, so the entire service bus code is covered by regression tests. This point ensures the service bus works as designed if the overall code is deployed, but otherwise, when only a changed subset of the service bus code is deployed, some stable parts might become outdated.

One more thing. If you have no clear understanding how the production supports any proposed changes, a case developed by business analysts covers not every substantial aspect of system behavior or requirements aren't set well, and you know they will be changed many times during the sprint, so there isn't a final solution, it's better just do nothing. Neither outcome is certain.

If you design or manage the development of integration solution or microservices-based application, please share in the comments under the post any approach to release management that has been used. This topic is essential for best in class application delivery and everyone may found a new idea or thought for his/her everyday practice.

Would you like to give a 'Like'? Please follow me on Twitter!