Webhooks vs API, oh well. When there is a talk about what method to use in order to seamlessly sync datasets between applications, platforms and databases (in other words, to perform application integration), APIs are often named as today’s best-practices method for handling this. Unlike ETL-based integrations, which are not scalable, quite expensive and a bad fit for dynamic changes in business needs, APIs allow for considerably more visibility into data consumption as well as more flexibility per se.
Yet using Application Programming Interface to sync data between various services can sometimes become rather a wasteful and inefficient way of using resources. Here is when Webhooks come to the fore.
In this blog article, we would like to touch upon the difference between how APIs and Webhooks “work” when used to sync datasets between apps, why API-based integration can sometimes become quite messy, and when exactly Webhooks are actually a better fit.
When to prefer APIs for data sync, and why
As you know, in order to obtain data updates through application programming interface, you need to “call” it on a regular basis, most commonly through an HTTP GET request (a side note: to retrieve only meta-information, you’d use HTTP HEAD method). With this in mind, it is safe to say that APIs are a great fit for application integration when you know you will have a constant change of data – no matter when an app requests for new datasets, there is always something in store for it.
The huge advantage of polling APIs is that you are the captain of your own ship. If you want more data, you just increase the paging size. If you know that right now you have a traffic jam somewhere on your server, then you, logically, reduce the paging size. This is practically an implicit flow control, which is very important when it comes to high data volumes and retaining the performance capability.
If you have rapidly changing data, for example, from outdoor sensors in an IoT integration scenario or shipment tracking systems in an eCommerce-related scenario, then you can be sure that data updates will be waiting for your call whenever you make it. In other words, you won’t find yourself empty-handed when polling API in such scenarios.
Did you know that with our connectors, you can poll virtually any API?
Really. Any.
Check out these generic API connectors
So, when would you want to use Webhooks then? Well, there is no guarantee that every call you make to API will have something ready for you on the other end. What if your datasets are not updated constantly and regularly? What if out of 8 API calls to a server, for example, to an inventory management system, only one gets an actual response? Just imagine this: Zapier is said to have made about 30 million poll requests over a substantial amount of time, out of which only 460,000 returned some results, which is only 1,5 percent.
In addition to that, as protection measures, most good APIs pose a rate limit on the number of calls per certain period of time, say, per hour. You certainly don’t want to waste these already limited resources by making empty calls.
This is when you can turn to additionally setting up a Webhook instead of simply polling APIs.
When Webhooks are a better fit to sync data between applications
The main difference between how Webhooks vs API generally work is that, while the latter place calls without knowing whether they get any dataset update as a response or not, Webhooks receive calls through HTTP POSTs from external systems only when those have some dataset updates.
Then, a Webhook can “activate” the whole integration workflow to act on these updates. The main point here is, however, that this activation happens for a reason (i.e. because there is a real update to process), and not just because it’s time to make yet another check for an update.
Here is an example of a Webhook configuration on elastic.io:
{
"recievedBody": "recievedBody",
"_query": {},
"_headers": {
"content-type": "application/json",
"accept": "*/*",
"accept-encoding": "gzip, deflate"
},
"_method": "POST",
"_url": "/hook/5d691738cb5a286adc1e68e2"
}
Another good thing about Webhooks is that they provide other applications and services with (near) real-time information. This is possible with APIs too, but it involves then an extra configuration step and is extremely resource-consuming. Webhooks make real-time data updates really simple, particularly when there is no need to communicate updated information back to the sender.
A very common example of that would be updating a CRM systems with an ‘unsubscribed’ status for a particular mailing list; a more “exotic” example is notifying a CI server to start a build every time any user pushes commits in a specific repository (a side note: this is what you can do with a webhook from Atlassian).
However, one of the dangers of using a Webhook is that you may never learn about any changes if the other system that sends updates to your Webhook goes offline at the wrong time for some reason.
In addition to that, you’ll have considerably less control over the flow of data because you’ll have to accept as much data as you get. Of course, there are ways to tackle this aspect effectively, for example storing datasets in a message queue. Yet this is by far not so efficient as being able to e.g. scale the paging size freely according to your immediate needs.
Webhooks vs API in a typical application integration scenario
To sum it up, webhooks are kind of mini-APIs that provide data communication one-way. In contrast, APIs enable a bi-directional data communication between different applications, but this is by far not the only aspect in which APIs surpass webhooks.
A crucial element of many data synchronization jobs is data modification. We are not talking here about actual manipulations of data formats and such – well, not in this article anyways –, but the very basic CRUD (Create, Read, Update, Delete) operations.
Apart from displaying information, APIs can also make changes to it. The list of methods is short but worthwhile to quickly go through anyways:
Create
HTTP POST: it will logically create data but beware – the POST method is NOT, as they call it, idempotent, meaning that if you make multiple POST requests, it might result in creation of multiple (identical) data records unless you have a certain data validation process in place.
Read
HTTP GET: retrieves information from an API server (already touched upon above)
HTTP HEAD: retrieves only metadata from a server. This is particularly useful when you’re dealing with very, very large data sets
HTTP OPTIONS – retrieves data about other methods and operations that the API server supports at the given URL. While this method has little to do with actual data sync jobs, it might be useful for API testing purposes.
Update
HTTP PUT: updates or replaces data record(-s), but can also be used for resource creation as it IS idempotent, meaning that it will only create a data record if it’s actually completely new. If a record already exists, it will replace it.
HTTP PATCH: updates only part(-s) of specific data. While PUT is not capable of performing partial modifications (it rather replaces a record completely even if there is the slightest change to it), the PATCH method is.
Delete:
HTTP DELETE: no clarification is required.
________
Webhooks are nowhere near as versatile as APIs are when it comes to data modification, which is why they alone have significant limitations within the scope of integration between applications and systems. But it doesn’t make them any less valuable.
Conclusion: Webhooks and APIs make a perfect team
So, is there a scenario where in the tug-of-war webhooks vs API, the former will win you over for data sync between apps, platforms or databases?
Oh yes. Probably the most obvious one would be when you need real-time dataset updates but you don’t want to waste your resources by constantly polling APIs.
Another very common scenario is when your integration flow includes some external application or system that has a bad API or no API at all. For example, a logistics company that sends shipment status updates only via a message (Impossible? We wish. Alas, a real example from our practice).
In this latter case, using a webhook is probably the only way to sync datasets between applications automatically. The message will be sent directly to the webhook that is configured specifically to “dissect” it and map its “contents” to another app, e.g. a fulfilment management system.
In a nutshell, a webhook can be actually regarded as a perfect, and sometimes even essential complement to API-based application integration.
Did we make you curious by any chance? Well then it’s a good thing that here, you can try Webhook together with REST API straight away.
Check our Webhook connector now
Webhooks vs API P.S.
If you’re thinking about how to design a great Webhook or a great API for other developers to use them to effectively sync data between applications, or if you’re just wondering how to recognise such a Webhook or API, here are the first places to check out:
6 Characteristics That Make APIs Fit for Application Integration – our own quick guide about APIs and what they should provide to enable an easy integration with them.
Webhooks do’s and dont’s: what we learned after integrating +100 APIs – a nice article focused on Webhook features that are important for developers who want to use them to sync data between applications and Co.
What Are Webhooks? And Why Should You Get Hooked? – if you want to “backpedal” a bit and get some basics first, this article does an excellent job at explaining in simple terms what Webhooks are and how they work – accompanied by some hands-on examples and a beautiful infographic.
Webhook integrations | How to use webhooks in your integration flows – webhooks are practically made for ensuring an event-driven architecture. In this article, we go over the main things to consider in webhook integrations scenarios and highlight a few curious examples.
Webhook security: Four risk scenarios and how to secure webhooks – it’s impossible to talk about using webhooks without considering how to ensure proper webhook security. Tampering with payloads, replay attacks, and more are all very real risk scenarios, and in this post we go over the respective mitigation strategies.
