With more and more technology options and vendors on the market, the possible ways of building integrations and API’s has expanded significantly.

A microservice based design pattern offers a number of great benefits in the integration space specifically.

Each microservice performs a single function on a data message.

Each microservice is stateless.

Because microservices are smaller code bases, they can be developed relatively quickly.

Changes to a single micro service don’t require a full rebuild of the solution.

Key to a successful microservice deployment though is a robust architecture and pattern of working. Once agreed, all developed microservices need to follow the same pattern or the ability to manage the solution over time will become problematic.

Key Considerations:

  1. Know how your costs will scale as more messages flow through the system.
    • Depending how your architecture is built, what may work at 1000 messages/day may be cost prohibitive at 100,000 messages per day.
  2. Consider your migration path before you begin building.
    • Change is inevitable. Building features into your solution from day 1 that make migration easier will benefit you in the long run:
      • .e.g. Ability to change URL’s
      • DNS
      • Ability to version manage API’s
      • Realistic downtime windows or redundancy to allow maintenance
      • Retry mechanisms for when external systems fail
  3. Consistent authentication
    • Especially important when multiple different developers are building separate integrations is ensuring there is a mechanism to standardise authentication and authorisation. This will reduce the future oboarding effort for new integrations but also provides a means of validating who has access to your integrations.
  4. Logging and monitoring
    • Establishing a common method of logging the integration functions as well as alerting for abnormal activity reduces the manual intervention required to continuously monitor the integration activity.

Example Deployment Scenario:

A professional services firm is modernising their infrastructure. The application they have been using to track clients and projects is no longer in support and with no viable alternatives in the market, they have opted to use a variety of different solutions and built integrations between the vendor systems.

For the integration approach, they have chosen microservices running on Azure infrastructure. The integrations are all low volume transactions and the serverless pricing model is suitable for their needs.

With the majority of their replacement systems being SaaS based, they have opted to place the integrations behind an APP Gateway. This allows for robust security controls in a centralised manner and detailed logging of all transactional activity to aid in future troubleshooting.

The design pattern chosen for the microservices is as follows:

  • Each external system has an inbound message receiver to receive webhooks.
  • The purpose of the inbound receiver is to validate the authenticity of the message, respond to the sender and pass it onto a processing queue in the form of Azure Service Bus.
  • This de-couples the transfer of information from the downstream processing that may be required.
  • A transform and distribute function listens to the Azure Service Bus and responds when an appropriate message is received.
  • The message is picked up from the queue and the function attempts to process it.
  • If the processing is successful, the message is removed from the queue, else it is placed back on the queue to be retried.
  • The transform and distribute function has a number of key functions:
    • Collect any additional information relating to the request
      • E.g. a webhook from the accounting system may notify of a new contact being created, but not contain the details of the contact in the specific webhook.
      • The transform and distribute function would take this message and call the relevant API to augment the details of the contact into the body of the payload.
      • Finally, the message is placed onto one or more distribution queues for the downstream systems. In the example of a new contact being created, the contact may need to be created in multiple downstream systems.
    • The design pattern allows the message to be distributed to a number of separate queues for each of the systems and processed with specific logic for each downstream system.
    • The Outbound system update functions listen to the distribution queues and enact the required changes.
    • Transaction logs for all of the activity are sent to log analytics.
    • Notifications of relevant activity are pushed to Teams for notification to the relevant business unit.
    • The design pattern (with the inclusion of the API gateway) allows legacy applications to be included and made available in a consolidated manner for external systems to update).
    • Provision is also made for ‘stateful’ integrations with the inclusion of a database for read/write access.
    • This database allows for caching of information from systems that do not expose an API directly. It also allows for stateful responses to integration messages. While this is not an ideal integration pattern for a number of reasons, it is not always possible to make changes to external systems.