The topic of data and application integration has such a wide coverage that there seemingly should be no questions left. And yet every now and then, we come across certain questions both from the IT people and the business users. So, we decided to answer the most frequently asked questions in our new blog series on data integration best practices.
This series will focus both on the technical side of the data integration as well as answering the questions that drive the business decision regarding the selection of a suitable tool and an appropriate software vendor. Despite the latter point, this series is equally suitable for those who prefer to handle all integration work on their own and for those who decide to go for a third-party solution for that.
Before we can engage with the data integration best practices, though, it is important to understand what issues both IT specialists and business users face when trying to integrate two or more systems, be it business applications, databases or even entire platforms. Hence, this is where we are going to start from.
Technical Mechanics vs Business Rules
Fundamentally, all integration problems can be broken down into two main categories: the problems that concern the pure technical mechanics of integration and the problems that are related to the correct application of business rules. Technical problems are generally lower level in nature and answer the question of “How?”, while business problems are normally higher level in nature and answer the question of “What?”.
For example, the desire to have customer information copied from the CRM into the ERP would be a business problem. The technical problem is, though, to identify and pick which APIs to interact with to solve the business problem.
It is important to keep in mind that the descriptions of the requirements associated with the problem on the business level may not necessarily line up with the technical details. For instance, a business expert may see and describe requirements involving contacts where contacts are associated with addresses. At the same time, from a technical standpoint, contacts and addresses may exist as different objects within the system.
In plain terms, an IT person will see contacts and addresses as two absolutely separate items. However, a business person would associate a given contact with a given address, seeing them as one piece of information.
This doesn’t mean, though, that one standpoint is better than the other. Quite on the contrary; the individuals who can answer technical problems are in general not the individuals who also should answer business problems, and vice versa.
Technical Mechanics Problem: Moving Data In and Out of Systems
In the first two articles of our series on data integration best practices, we are going to tackle the technical specifics of moving data in and out of business applications, databases, or platforms.
On a very high level of abstraction, one can sum up the process of data integration the following way. In order for data to move between systems at some point, the data will go from being inside the system to, obviously, not inside the system (or vice versa).
These operations can be categorized into read operations that don’t change data and write operations that do change data. There are multiple mechanics that can make such operations occur. Polling by a timestamp would be one example, bulk extra with delta deletion – another. The selection of an implementation mechanic is, in general, independent of the business requirements that drive the particular integration need.
Ultimately, the selected mechanic must have the following properties:
- It must be sufficiently performant
- There must be a commitment from the software vendor that this mechanism will continue to exist after any future updates to the software
- It must be allowed in the sense that your IT security team is not against permissions granted to that mechanism
- It must be capable of reading and modifying the data as required by the applicable business rules
- Last but not least, it must not break the actual business application / database / platform
In the next article we will be going through several mechanisms for the detection of data changes. In addition to that, we will cover different technical sub-decisions that need to be made in this respect.
