API Integration 101

Over the past month, in my current role as Head of Platforms and Integrations, I have had the opportunity to interview over 50 candidates for API Integration roles. Through these interviews, I've noticed a knowledge gap in certain areas which include system design patterns, API user management, integration concepts, and API specifications.

While it's not possible to cover all topics pertaining to APIs and integration in a single article, I'd like to focus on a few key concepts that could pique the interest of individuals who are eager to learn and grow in this field.

Multi-Tenant API Architecture

The diagram above illustrates a multi-tenant system which enables third parties as well as internal systems to use the same services. This architecture is increasingly common in modern API platforms, allowing organizations to serve multiple client types through a unified gateway while maintaining proper separation and security.

OSI Model – The Foundation of All Computer Communication

The OSI (Open Systems Interconnection) model can be best described as the universal language for computer networking and it is a theoretical way of describing how different layers of a network communicate with each other. There are seven (7) layers in the OSI model which have their specific functions and protocols however within this article, we will focus on two layers i.e the application & network layer.

Application Layer

This is the layer closest to the end user. It provides network services directly to applications, such as web browsers and email clients. API security mechanisms like OAuth and API keys operate at this layer.

Network Layer

This layer handles the routing of data packets between different networks. Security considerations at this layer include IP whitelisting and firewall configurations.

Understanding API Documentation

APIs (Application Programmable Interface) are a set of clearly defined methods of communication between two pieces of software. The most important step of software integration is understanding the API user guide, which is often embedded within the API documentation.

The API documentation usually defines the following:

API Specification
Schemas (Data format)
Security & Connection layer (application layer (API key, OAuth), network layer (IP address & port))

API Specification

The API specification describes the functionality and expected behavior of an API. It standardizes the data exchange between web services, examples of these specifications include GraphQL, REST, SOAP etc. Let's use REST APIs as a case study for expatiating on API.

REST APIs

REST stands for Representational State Transfer. In this specification, clients and servers exchange data using HTTP.

HTTP

The hypertext transfer protocol is the foundation of the World Wide Web, and it is used to load webpages using hypertext links. The HTTP is an application layer protocol designed to transfer information between networked devices and runs on top of other layers of the network protocol stack.

A simple flow over HTTP occurs by a client (user) making a request to a server which then responds with a message.

HTTP protocol requests are made up of a URL, method, headers and body.

HTTP Methods/Verbs

Methods can sometimes be referred to as verbs which describes the action to be taken by that HTTP request. These include:

POST - Create
GET - Read
PUT/PATCH - Update
DELETE - Delete

These are also known as CRUD operations: create, read, update, delete.

Headers

Headers represent additional information being requested (meta-information) along to the server. E.g Content-type= application/json, authentication token.

Body

A Body contains the data a client sends to the server as a request, in order to retrieve information in the form of a response.

Schemas

This is how the data in your request and response is defined. Examples include JSON, XML, with the most common one being JSON — JavaScript Object Notation.

{

"type": "object",

"properties": {

"id": { "type": "integer" },

"name": { "type": "string" },

"email": { "type": "string", "format": "email" }

"required": ["id", "name", "email"]

}

Security – API User Management

This is an important element of API management: managing the users of API resources, in terms of what and how much access each user has to the server.

Two most common types are: API Keys and OAuth (client ID & secret key).

OAuth

OAuth is an open protocol for authorization that allows users to share their private resources to a third party. There are two versions: OAuth 1.0 (deprecated) and OAuth 2.0 (active) with 2.1 in draft.

OAuth provides a client with a username and password (aka client ID & secret key) to access server resources in a process called authentication and authorization. OAuth is provisioned with different environments namely sandbox and production.

API Keys

An API key is simply a string of alphanumeric code used to identify & authorize an application user. It is often passed in the request header and can be extracted or intercepted by bad actors.

To protect your users against this loophole, it is advisable to adopt a second layer called IP whitelisting on the network layer of this connection.

IP Whitelisting

Creating an IP whitelist for your API key ensures the key is only valid when the API is being accessed from a registered and pre-approved IP address.

IP connection whitelisting occurs at two layers of the OSI model i.e the network and application layer. On the network layer, it is important to whitelist the client side IP and port addresses on the server side. Whilst on the application layer, the API user will store & send each request with the API key in the header.

As an Integration PM, you are expected to understand all of the above and then analyze each endpoint, their functionalities, and how it will all fit your business requirements. Once the alignment between business requirement and API feature is completed, then an Integration document should be created and must include certain key elements which are defined under the concepts of integration.

Concepts in API Integration

Key concepts in API integration include:

User Flow i.e UML, Unified Modeling Language
Frontend development
Backend development
Reconciliation and settlement
Commission configuration

To break these concepts down, let's define a use case.

Use Case: Online Order Management System with Payment Integration

An example could be an online order management system that requires payment acceptance. In this use case, there is a requirement for the integration of a payment service provider. The requirement here is clear and the order management system needs a flow whereby its users can complete a purchase online and receive value.

User Flow

Conceptually this can be defined using UML's, to visualize the flow between the user, actions taken, the request and the response involved. I often use flowcharts to communicate user flows to my development team.

Frontend Development

In this use case, the first step is to conceptually define how this payment system will flow into your order management user experience. The next step will be to build the UI which essentially defines the scope of the integration. The UX designer must have built a low or high level flow to incorporate both pieces of software to enable a user seamlessly complete a purchase. A front end engineer will bring the design to life by hardcoding and applying each step to its backend endpoints.

Backend Development

There is a bit of system design pattern consideration here. Assumption, where the order management system leverages a micro service architecture, the decision to be made will be to develop a payment micro service with a server-less DB. This Server less DB will store all attributes of the payment service provider and connect the order check out directly to the third party service provider (Payment service). This backend integration process is not a one size fits all but is highly adaptable. There are many ways to go about the backend integration depending on the system architecture, dependencies, patterns and other such factors.

Reconciliation & Settlement

This essentially describes the checks and balances between two systems. In our use case, payment occurs on the order management system however it is processed by the payment service provider. There are going to be successful, unsuccessful and pending transactions between these two systems and this process has to be defined clearly to enable both parties to understand how to handle these cases efficiently. An example of settlement processes include T + 1 which means settlement happens after a day on all accounts.

Commission Configuration

This is not a consideration for all integrations however for our use case it is popular. Automating this process also ties back into the reconciliation and settlement process. Developing the commissions flow into the reconciliation process is key and very important in the case of charge backs, refunds etc.

Integrated Platform as a Service (IPaaS)

A few companies have innovated around integrations for some use cases (ERP Integration, data loading and flow building) thereby birthing the concept of an Integrated platform as a service (IPaaS). The IPaaS enables low code integrations between two pieces of software i.e backend services or server side applications.

Key Elements of the IPaaS

Connectors

These are pre-built application instances whereby users can enter a username and password (client ID and secret key) and complete an integration.

Fields and Mapping

This enables each endpoint to be mapped one to one i.e orders will be mapped to inventory between two systems.

Universal Connectors

These are HTTP, RestAPI, GraphQL, webhooks etc.

API Gateways

It acts as an entry point for all client requests to your systems APIs. But also protects your resources against compromise.

White Labeling

The IPaaS serves as an integration layer which can be used to expose an instance of your software in a multi-tenant type of system.

Conclusion

Technology and innovation are like two peas in a pod. It's best to always keep your ears peeled and constantly rethink the art of software development.

Technology development is 60% art and 40% engineering. The art is in imagining and defining how the solution will work, the engineering is in bringing that solution to life.