Moving Data over the Web: AJAX vs. WebSockets vs. Webhooks

My colleague once referred to FME Server as “web glue”, and he’s not wrong to call it that. In fact, I spend a fair bit of my time experimenting with methods for moving data over the web.

In this post I’m going to focus on three of the most popular methods: AJAX, HTML5 WebSockets, and Webhooks. While I’ll primarily examine how to connect to data over the web, it’s worth noting that it’s also relatively easy to transform data and automate web processes at the same time, especially where spatial data or a map is involved.

Moving Data in a Client-Server Architecture

When the client is JavaScript running on an end-user’s computer you end up with two basic options: AJAX or WebSockets.

AJAX (Asynchronous JavaScript and XML)

AJAX allows your client-side JavaScript application to make a request to access a server-side resource. One nice thing about AJAX is that it won’t hold up your app while waiting for the response (i.e. it’s non-blocking). Instead, when the response is ready, a ‘callback’ function will be invoked to process the request. This was pretty awesome a decade ago and is still very relevant today. Here is an example of a simple coordinate-pair reprojection service that uses AJAX to send requests to FME Server’s Data Streaming service.

A decade ago you had to write your own AJAX request functions, but now there are great libraries like jQuery that provide cross-browser compatible AJAX request functions. jQuery is used on a lot of websites nowadays and I would call it the defacto standard for common JavaScript operations. In fact, it is a key component of the new FME Server JavaScript API that wraps up the most common FME Server REST API methods.

AJAX is great if you aren’t in a hurry, but if you’re moving a high volume of data then the overhead of creating an HTTP connection every time is going to be a bottleneck. You need a persistent connection instead. In addition, AJAX always has to poll the server for data rather than receive it via push from the server. If you want speed and efficiency you need WebSockets.

HTML5 WebSockets

WebSockets allow your client-side JavaScript to open and persist a connection to a server. With WebSockets, data is exchanged as messages, which can happen very quickly due to the persistent connection.  This makes it possible to develop some really cool web apps like this live spatial dashboard showing ship, bus and plane locations.

So, if you are dealing with real-time data you might need to upgrade to WebSockets. I say ‘upgrade’, because you can initialize a HTTP request on port 80 and request that it be upgraded to WebSockets in order to switch protocols. The benefit of using the standard web server port 80 is that many firewalls and organizations allow open access to this port.

Another powerful aspect of WebSockets is a capability called full duplex, which is a fancy way of saying that both the Client and Server can send each other messages. Contrast that with AJAX where the server has no method for pushing messages to the client. This two-way message sending allows web applications like Facebook and Gmail to provide the feel of a desktop application. Closer to home, it enables the creation of web applications like this Real-Time Map Editor that updates each users web map immediately with any additions, updates or deletions that are made.

Moving Data in a Server-Server Architecture

Strictly speaking, there are a number of ways of pushing data from one server to another server, including: Java Messaging Service (JMS), Amazon Simple Notification Service (SNS) and even HTML5 WebSockets. In many situations these are great solutions for robust, enterprise integration. However, the overhead of standing up a specialized server or paying for a service and also implementing the API is a barrier to system integration. This is especially the case with many of the newer cloud-based services coming online.

So how can we offer an easier method for pushing data from one server/service to another? The answer is remarkably simple and brilliant: Webhooks.

Webhooks

A Webhook is really just a URL that accepts a HTTP POST (or GET, PUT, DELETE). One server/service will provide another with a callback URL that it should push data to when it has some data to send. Brilliant! Any old web server can be used really. In fact, there are services available online that provide the web server, script editor and URLs for you.

One of our favorite web automation services, Zapier, supports Webhooks, which includes sending to another server/service’s callback URL and receiving data sent to their own callback URL. As it turns out, FME Server also support Webhooks. In the recent Developing for Data Transformation webinar, I utilized both Zapier and FME Server Webhooks to process a new file dropped into Google Drive, extract some spatial information with FME and then create a set of required actions in Google Tasks. (just thought of an idea: data validation that creates a task list of problems to fix – ™, all rights reserved).

In our recent webinar, we found that 59% of folks were familiar with AJAX, 44% with WebSockets, but only 7% with Webhooks. How about you? Each have their pros and cons and scenarios in which they’re best suited. Which methods are you employing? What’s been your experience?

Thinking about using FME Server to build an awesome web app or integration? It’s worth watching the aforementioned webinar, and please drop me a comment on what you’re up to or if you need assistance.