Data integration best practices – What systems move data

Obviously, data cannot move itself. A processor somewhere must pick up and move data somewhere else. So, while in the previous blog article of our series Data Integration Best Practices we talked about HOW we can move data, this time we are going to see WHAT moves data between systems and the design choices that we need to consider for …

Request-Reply vs Asynchronous Integration

Moving on to the next chapter of our blog series, namely about different types of integration, in this article we’re going to explore the main two types of integrations: asynchronous integration and synchronous integration. Whether you need a fast response of the systems involved or the results to be delivered further in the future; both, request-replay or asynchronous integration, offer …

Maintaining consistency handling integration errors

As we mentioned at the very beginning of our blog series, all integration problems essentially fall into two large categories: the problems that have to do with the technical mechanics of integration and the problems that concern the correct application of business rules. There are, however, certain types of problems that originate in both. In this article, which is closing …

Data Integration Best Practices - Data transformation

So far, we have addressed only the problems that concern the pure technical mechanics of integration. Now it’s time to review the challenges that are related to the correct application of business rules. These are, as we stated in the first article of our blog series, typically of higher level in nature and answer the question of “What?”. And the …

Three mechanisms to detect data changes

In the previous article of our series Data Integration Best Practices, we defined that all problems related to data integration can be essentially broken down into two large categories: The ones that have something to do with technical mechanics and the ones that are connected to business rules. In the second article of this series, we will review one of …

Docker and NodeJS as PID 1 on Kubernetes, Mesosphere and Co

At elastic.io, we are using the ideology “one process per Docker container“. Surely, we apply this ideology for running integration components as well. So, each of our integration components is actually one process inside of one Docker container, and each of these Docker containers is running on Mesosphere and Kubernetes. Recently, though we had been having some unexplainable issues with …