As we mentioned at the very beginning of our blog series, all integration problems essentially fall into two large categories: the problems that have to do with the technical mechanics of integration and the problems that concern the correct application of business rules.
There are, however, certain types of problems that originate in both. In this article, which is closing the chapter on integration problems, we are going to review them and suggest a few ways how to deal with them.
Maintaining Consistency
If we receive a large number of errors, we need to find a way to sort them out depending on the type. The following errors can occur during an integration.
Authentication problems
These integration errors occur when the credentials provided to perform an integration are invalid. This can happen to a previously valid account as well when a token has expired, a password changed, or a system user who is trying to access it doesn’t exist in the system anymore. In order to solve these problems, the user will need to provide new credentials, or the permissions associated with a credential need to be expanded.
Authorization problems
These integration errors occur when the credentials provided to perform an integration lack the permissions to do the required operations. When systems support the concept of user specific permissions or user roles, API access usually happens in the context of a user or based on a list of granted/not granted permissions. This means that actions done through the API will be allowed or disallowed based on the permissions that the API user has or that are on a permission whitelist. In general, based on the principle of least access, this list of permissions should be as small as possible. However, if the granted permissions are insufficient, then authorization problems like this will occur. The resolution to this problem is to expand these permissions.
Business validation error
This type of integration errors happens when a system rejects a request because such a request would violate the enforced business rules. These are often the result of incorrectly formulated business rules. Another reason can be that the system A has different validation requirements than the system B. Let’s take as an example a fictional address. If you enter such an address into your CRM system, there is a high probability that it would accept this address because it doesn’t have the means to check its validity. It would look quite differently, though, if you were to enter the same address in, say, your navigation app.
One way to solve this problem is to ensure that all systems storing data have the same validity requirements. Another way is to only try to write data to a system once a record passes the validity requirements.
System availability problems
This type of integration errors occurs when an integration won’t complete because a system goes down. In order to solve this problem, the system that is down must be, obviously, brought back online. Some other possible causes of this include DNS, SSL and Disk space problems. One needs to handle these accordingly.
Transient problems
In this case, some sort of temporary issue appears that is self-corrected. You can compare it with accessing a website when you send your request and it times out. If you try sending the same request a minute later, you are most likely to succeed. In general, it is possible to deal with this type of problems by setting up some sort of an auto-retry for such cases.
System threshold problems
Most systems have a finite capacity, either in terms of a number of allowable API calls or computer resources. If the integration job exceeds those resources, then integrations will stop working.
Developer bugs
As the title says, such integration errors occur because the developer failed to take into account some edge cases, or in other words, an unusual but a valid set of inputs.
Conflict Resolution
Sometimes you would need to configure two systems to mirror each other. This means that if you enter certain data, it will be automatically copied into another system. In such a scenario, it is possible that users make conflicting edits in these two systems.
To avoid integration errors in such a scenario, there are multiple ways to handle this. One way is to set the system A to be the source of truth. The system B will be then, therefore, the mirror. In other words, if there is any discrepancy between the systems, the system A is considered accurate. It is possible to set this for either the entire record or you can set this for each field on the record.
Another way to handle this is to allow both to be the source of truth. However, if conflicting changes then happen, you would need to find a technique to resolve the conflict. This can become even more complicated by the fact that two systems may record semantically equivalent data in different forms. Unfortunately, there is no generic solution to this potential problem.
Hierarchy Preservation
Often there is linking between objects in a system. For example, in a CRM, a contact would likely has a direct correlation to a company. This means that in order to create a customer that is linked to a company, the company must be created before or at the same time as the customer. It is important to consider this when designing any integration solution. In general, one can solve it by understanding that you can describe the relationships between objects as a tree. And in such a tree, one must sync all “parents” before “children”.
At this point, we have covered the main problems you would typically face when trying to integrate and sync data between different systems. The next chapter will be all about different types of integration. In the first article of this chapter, we will have a look at the request-reply and async integrations, their pros and cons. Stay tuned by following us on Twitter and LinkedIn!
