The Power of Events: Subscription and Notification Services

The technology of data movement is getting a lot more fun and exciting. In the past the data connection latency between data producer and data consumer was measured in days at best and months or years at worst. In today’s fast-paced interconnected world this doesn’t cut it. To support the latest applications the latency is expected to be real-time; measured in seconds or less. Data consumers increasingly expect to be able to subscribe and then receive data notifications as things happen.

Delivering this performance requires systems with new event-driven architectures, as old batch-oriented systems aren’t able to meet the need. Another exciting aspect of this real-time connection is that the line between data consumer and data producer is blurring, with many systems’ users being both. This is a natural evolution to the ease with which data flows in these new systems.

The Benefits of Event-Driven Architecture Systems are Huge

Here’s just one example. Many police departments provide their officers with a morning report showing all the crimes that have happened within the past day or week. With the event-driven architecture, police departments can now relay reports to officers in the field as crimes happen – a massive improvement making it possible for officers in the field to be more operationally aware of what is going on around them. These new systems also make it easy for citizens to report crimes as they occur.

Other examples include:

• Citizen Reporting System for Local Government - Graffiti, potholes, crimes, traffic accidents
• Severe Weather Reporting - Floods, hurricanes, snowstorms, lightning, tornadoes
• Public Utilities – Damaged facilities, service outages, emergency client response
• Emergency Response – injuries, crimes in progress, fires, traffic accidents, Amber Alert, natural or man-made disasters
• Traffic – congestion reporting, traffic accidents, road closures, construction

The Challenges Switches from Supporting Formats to Protocols

Whereas in the old batch days of moving data the challenge was all about data formats, it’s now expanded to also be about protocols. These new subscription-based systems keep subscribers up to date by sending notifications through any number of protocols. Similarly, the more protocols that the data reporter is able to leverage the better.

For these new systems to flourish they must be able to plug into as many other systems as possible. From Twitter, to sensors, to cloud-based data stores such as Google Fusion Tables and DropBox, to other notification systems such as Pusher and Amazon SNS – the more the merrier. Sometimes the most interesting protocols are not those that are new or exciting, but older established protocols such as UDP or email. Above all else, a state-of-the-art notification system must support mobile devices.

Our Experiments with Subscriptions and Notifications

Our own experiments have found this new model to be both flexible and powerful. To test this we built a very simple event notification system using FME 2012 Server SP2. This system which is completely driven by workspaces (i.e. no backend code) enables events to be logged through a web page, via email, and via mobile devices. As part of this experiment we built two mobile apps: one for the subscriber and one for the reporter – not in the Apple Store, yet…

The biggest surprise was how useful email was as both a reporting and notification mechanism. Need to report data? No problem; simply take a picture and mail it to the system. Want to be notified of updates? We can do that too with email. Email it turns out is a very easy entry point into the system. Want to an email based data validation system? Trivial.

We will be showing some of the things that this new architecture can do for you and your organization (shipping with FME 2012 Server SP2) on our upcoming World Tour. You can also bet that I won’t be able to resist sharing some of the exciting details on Twitter as well (@donatsafe). Here’s an overview video and a video that shows how to try it for yourself.

Will You Accept the Challenge?

The future of data delivery is changing. The subscription/notification model and its event-driven architecture make data available as never before. With this new capability organizations are able to ensure that their users are making decisions with the most accurate and up to-date data.

What uses do you see for this exciting new architecture? How can your organization benefit from real-time data movement?

Related posts:

1. Bringing Real-Time Data to the Web – A Live Example using Node.js, Pusher, and FME Server
2. Head in the Cloud with Amazon Web Services
3. Bringing Real-Time Spatial Data to the Web Using HTML5 Web Sockets
4. The National Information Exchange Model: An XML-based Framework for Data Sharing
5. Alternatives to SimpleGeo: Remaining Options for Spatial Database Services