I believe that in 2012 we are going to see a major shift when it comes to data access. Organizations will realize that the old models of data access are no longer sufficient in this hypercompetitive, always connected world. In 2012 organizations will need real-time data access to the latest information. One of the things that has me excited is the growing number of sensors and the data from this global sensor network. In this post I’m going to discuss the need for real-time data, with a focus on sensors and other real-time data sources.
Batch vs. Real-Time
For years data movement has been largely a “batch” or “on-demand” activity. Organizations would define times when data was to be moved, or users would manually copy the latest data when it was conveient for them. Many organizations have tasks that run every night or every week; and many data providers have tasks that run every quarter for a quarterly update. This works well in an environment in which:
- The data itself is collected by field workers who are disconnected.
- The data compilation has a manual component to it.
- The data is derived from other data delivered by a 3rd party.
The common thread with all three of these environments is that the data is dependent upon a “human” component, or as we refer to jokingly at Safe, the data has a “carbon” footprint. But what about the increasing amount of real-time data that doesn’t have a “human/carbon” component to it?
For real-time data, the old model of running a batch job, or having a user trigger a job to move the data isn’t effective at best, and useless at worst. For “real-time data” you need “real-time data movement”.
The World of Sensors
One of the biggest sources of real-time data is that which is delivered by the growing network of sensors. The number of these that are available already is mind boggling and truly exciting. Indeed, anyone now with a smartphone is carrying a number of sensors with them at all times, and many mobile applications want to use these to provide a better experience.
As everything, everyone, and every event has a location, work to standardize sensor interfaces is currently being done by the OGC (SOS, SAS, SES, SPS, WNS), the W3C, and ISO.
Here are some sites that we have found and worked with.
Pachube
This site is a must visit for anyone interested in sensors and the creative things that people are doing with sensors. Their recent post on building an open air quality sensor network (see discussion of Arduino below) is a great read to understand the power of this technology coupled with the “open” hardware platform. Pachube can be thought of as a portal to almost 100,000 different sensors with many of them being public!
Getting a free account on this site is very easy, and consuming data is also very easy with every feed being made available as JSON (default), XML (of course), and CSV (poor cousin of the three). Part of the feeds is location giving the site the ability to support moving sensors.
Using FME 2012 it is very easy to consume data from Pachube. I have posted a video showing how easy it is to consume this Pachube sensor data with FME 2012 (desktop and server). There is also a separate video about our support for real-time data movement at the end of this post.
Sensors for Your Personal Life
Illustrating further how sensors are coming of age, we found two new products for personal sensors. The first kit (which I’ve ordered) is from the folks at greengoose and costs just $49. It comes with a base station and 4 sensors. I like these folks a lot as they know how to have fun. A “coming soon sensor” is a “toilet lid” sensor which lets you know when the toilet lid is left up. (Saving you the time of constantly checking or perhaps reminding you to put it down!)
The second called Twine is coming soon and will have a whole host of sensors – we have donated to this cause to help them come to market. Once both of these arrive we’ll hook them up to Pachube as well as consume directly with FME Server 2012. I will keep you posted on the experience here once they arrive.
Sensors in Your Pocket
Look for more and more applications that make use of and collect the sensor data. The future of personal sensors has the potential to revolutionize things like health monitoring. On that topic, here’s a link to sensors that you swallow to enable physicians to collect diagnostic information. A link to what future iPhones may have is here.
Want to develop an application to consume iPhone sensors? Read more here.
Build Your Own Sensor – It’s Easy!
While working and consuming sensors on Pachube I discovered that many of the sensors were built by hobbyists using the Arduino platform, which is an open source electronics platform! The beauty here again is that it is very inexpensive and easy to get started. A starter kit is available for only $40! To think that I found Lego fun growing up!
Safe Software’s Role in this New World
It is clear that the times they are a changin’ when it comes to the world of data sources. The volume of data that we all need to deal with is growing by leaps and bounds. Sensors add another exciting source of data that are sure to change the world in which we live.
The role of Safe Software and FME in this new world is the same as it was in the old world. We focus on making it easy for you to move the data to where you need it. To be effective in the world of real-time data a new architecture was needed and that is precisely what we have delivered with FME Server 2012.
[Matt Ball recently interviewed Dale about Sensors and FME 2012. Here’s his summary of that conversation: Safe Software’s Expanded Role as a Conduit Between Sensors and Systems.]
What does your organization use sensors for? Do you need to move any real-time data?
Don Murray
Don is the co-founder and President of Safe Software. Safe Software was founded originally doing work for the BC Government on a project sharing spatial data with the forestry industry. During that project Don and other co-founder, Dale Lutz, realized the need for a data integration platform like FME. When Don’s not raving about how much he loves XML, you can find Don working with the team at Safe to take the FME product to the next level. You will also find him on the road talking with customers and partners to learn more about what new FME features they’d like to see.
