Of all the IoT (Internet of Things) devices available, a good many of them are designed for measuring fitness and tracking activity. Even if you don’t wear an activity tracker, look around your colleagues and you’ll very likely see someone wearing such a device.
Combine that with FME’s capabilities – to read IoT data, process it, and push it to an output format or notification – and what do we have? (Oh, I think you know where this is going!)
Yes, we have an article that covers FME and fitness trackers. In this case it’s the Fitbit® tracker; an activity tracking device that we added support for in 2018.1. I’m going to cover:
- Some high-level ideas on what you might do with this capability
- How to connect your Fitbit tracker to FME using a web service
- How to fetch your Fitbit data, and some suggestions on what to do with it in FME
- Which other supported devices connect with an FME web service
- How to create an FME web service, for unsupported devices
Even though I’m concentrating on a fitness tracker, there are many other web-based devices and sensors to which this content applies. Plus I think a combination of such devices with FME has potential way beyond personal use. So, looking at fitness in particular, let’s start out thinking about how you might use fitness trackers at an enterprise level…
Enterprise-Level Activity Tracking: Physical Literacy and Other Examples
I was at a health-related conference recently where the topic of Physical Literacy arose. As I understand it, the goal of physical literacy is not to push people into a particular sport just in order to keep fit. Instead the concept is to encourage people to learn basic movement skills, so they feel more confident in taking up physical activities and sports on their own account.
For example, because people learn by doing, as a town council you might encourage activity by adding a walking trail or placing outdoor gym equipment. Or even just paint a hopscotch pattern on the sidewalk! It can be that simple.
But the key to success is to measure outcomes. Even if you add physical challenges to your town’s environment, how do you measure if residents’ physical activity has increased? Simply select a number of residents and keep track of their steps using an activity tracker. Even measuring just step count – and keeping records using FME of course – can prove that activity programs are working.
That’s just one example. I can think of multiple enterprise-level uses of fitness trackers, and I’m sure you can too:
- Hospital physiotherapists tracking patient activity to ensure they are doing exercises
- Mountain rescue teams tracking the amount of search area covered
- School or university athletic programs tracking the amount of training and sports activity
- A medical facility researching sleep patterns
It’s a wide range because most fitness trackers include not just the amount of activity, but GPS positions, sleep, nutrition, and others.
But, of course, we’re not just here to talk about tracking activity, but to show how to retrieve and process tracking data using FME…
Connecting to IoT Devices with FME
As I said, although we’re talking trackers, this applies to other IoT devices, such as security cameras, cars, and home heating devices. So you can apply this to just about any device, but for now I’m using a Fitbit tracker.
There are two parts to setting up an IoT connection with FME: setting up a web service (the developer part) and setting up a connection through that service (the user part). Here I’m both developer and user, though – as I’ll mention later – I don’t have to be.
Setting up a Fitbit Web Service
Regardless of device, the first step is to register a developer account with the device’s manufacturer. Here I browsed to dev.fitbit.com/apps, registered, and created a new “app”:
This created a new Fitbit “app” and provided me with a set of information I can use to connect to that app:
No this is not my real ID/secret! Sorry. But it’s very easy to create your own.
Then I started FME Workbench, selected Tools > FME Options from the menubar, and clicked on the Web Connections option. From there I clicked a button labelled Manage Services. It showed me a list of available web services; both those that are installed and those that can be downloaded from the FME Hub. Fitbit is a template service stored on the FME hub, so I chose it and then clicked the Download button:
The FitBit web service template (hmmm – I notice we use FitBit not Fitbit. I’ll mention that to our developers) was downloaded and installed, and I could now enter my secret parameters:
And that is pretty much it for the developer side of things. Now I can access user information through that app using FME…
Setting up a Fitbit Connection
To be able to connect to my Fitbit account I need the web service set up with developer information. However, as a user I might not want to set up a developer account, or feel comfortable doing that. Well that’s OK. If someone you know (like me) has created an FME web service (like above), they can export it and share it with you (like, here)!
So feel free to download the web service I created and use it. Just go to Manage Web Services under FME Options, choose Import from File, and select the above file. The password is “fme”.
Once you have your FME web service – whether mine or one you created – as a user all you need is a web connection that makes use of that service. You’ve probably created similar connections already – either for a web service or database – so this should be more familiar.
To create a web connection, I can either go to the FME Options panel again, or a I can create the connection directly where I intend to use it. For example, here is a HTTPCaller transformer:
The HTTPCaller allows me to use authentication and here I choose a Web Connection authentication and to create a new connection. This opens up a simple dialog:
…where all I have to do is choose FitBit and click Authenticate. Authenticating the connection takes me through the standard Fitbit dialogs to enter my username and password, and then prompts me to allow FME access to my account:
So now both developer and user parts of the process are complete, I am connected and ready to go…
Accessing IoT Devices with FME
So what will I do with my Fitbit connection? Well, the Fitbit Web API documentation tells me what sort of calls are available and what HTTP method to use:
…and it will even tell me what the request URL is:
This makes it easy for me because I can just copy that URL and paste it into my HTTPCaller transformer. I must also select the correct HTTP Method (here it’s a GET). Run the translation and FME is provided with (in this case) my Fitbit activity for yesterday:
Obviously there’s a lot to process there, but that’s JSON format and FME has a JSONFlattener transformer to take care of that, giving the following:
Basically it’s a set of attributes recording my activity, plus a list of exercise activities (just a single walk).
I do take exception to the “strolling very slowly” part! To me 5.073773 km in 3152000 milliseconds (52.5 minutes) is not slow and is not less than 2mph; but that’s Fitbit’s interpretation and I can hardly blame FME for that! I also wonder whether measuring my walk to the nearest 1/10th of a millimetre is necessary, because we all know what problems excess precision brings.
Still, I’ve got the information out of Fitbit and can now work with it…
Using IoT Data with FME
If you’ve used FME for any length of time, you’ll know that there are countless tasks you can use it for; so these are really just suggestions:
Write my data to an online visualisation tool like Tableau. That’s not to say the Fitbit dashboard isn’t good, but I imagine a dedicated visualisation tool would give me more control and variety over my statistics.
- Keep track of your daily distance activity and mark it off on a map; for example you could pick a route like New York to San Francisco, and update the route with your theoretical progress. That would be easy to do in FME using a ShortestPathFinder transformer to make the route and a Snipper to cut the route to your current total distance.
- Write your data to an online database, and have your friends and family do the same. That way you’re recording everyone’s activity and can compare results. Maybe generate a league table of the top scores (the HTMLReportGenerator transformer will help).
- Use a notification – either on FME Server or with a transformer like the Tweeter – to publish your accomplishments.
- Don’t like the default Fitbit badges? Make up your own! You can easily automate this to award yourself badges when certain goals are met; for example, you’ve walked as many kilometres as there are FME transformers.
- Scan nutritional information on a food package, interpret it (try the TesseractCaller), and push the information to the Fitbit food intake tracker.
- Convert Fitbit exercise logs to spatial data, record the data, and publish it through a web map.
In short, let your imagination run riot! If you have issues trying to implement a particular task, everyone on the FME knowledgebase Q+A is ready to help.
But I did mention a couple of specific things there – location, and collaboration – and I want to mention these in a little more depth…
Using Fitbit Location (and other) Data Types
When I first tried to get my location data, the process failed. It was because I didn’t have the permitted “scope”. Return to the Web Service definition for Fitbit in the FME Options dialog, and you’ll notice one parameter called “Authorization Parameters: URL”. It’s value is this:
https://www.fitbit.com/oauth2/authorize?scope=activity&prompt=consent&response_type=code
That’s the argument for authorizing FME access. The “scope” part defines what data you have access to. By default the scope is activity only; so I can find out the length of exercise, but not where it took place. For that I would need the location authorization. In fact the documentation for Fitbit tells me there are multiple scope options:
- activity
- heartrate
- location
- nutrition
- profile
- settings
- sleep
- weight
So to grant location permissions I changed that parameter in the FME dialog to:
https://www.fitbit.com/oauth2/authorize?scope=activity+location&prompt=consent&response_type=code
Once I apply that to the web service, I must also re-authenticate my connection. So I find my web connection, right-click, and choose Reauthenticate. That pops up the same dialog as before, but now I have another scope to approve:
Now in the HTTPCaller I can run the API command to fetch the details of my exercise:
https://api.fitbit.com/1/user/-/activities/17902687982.tcx?includePartialTCX=false
Notice the ID number is the same as I retrieved in the general activity query. The results of this query include the location of GPS points:
For some reason – I know not why – it now returns XML rather than JSON. Odd, but we have transformers that can just as easily handle XML. Here I used the XMLFragmenter on the “Position” tag to return separate features for each location, each with Latitude and Longitude. Then a VertexCreator, LineBuilder, and CoordinateSystemSetter:
This gives me a line feature with the route of last night’s walk. After 5km, I must have slept well, something the sleep statistics would tell me if I cared to add that to the scope too.
Using FME and Fitbit Collaboratively: Gamification and Security
We now know that I can easily use FME to fetch Fitbit statistics and make use of them. But fitness trackers like this are really about “gamification” – scoring points in competition with others – so collaborative methods are very useful. How is that achieved? Can I make a single workspace that reads everyone’s statistics?
That is a little more complex. Although I only need create only one Fitbit application, the application needs a token for each user account it wants to access. The token is stored inside each FME web connection, therefore each user account needs its own FME web connection.
So I can create the FME web service (as the developer) and populate it with my own Client ID and Client Secret. I can then export that as a file (the client info is suitably encrypted) and share that service with other people (as I did above). These other people (users) can use that web service to create a new web connection, authenticated with their own username and password, to read their data.
But that doesn’t mean my workspace can read your data. For example, I was hoping to be able to create a league table of FME Fitbit users, using a master workspace to query everyone’s account. But I don’t think I can do that. I can’t use my connection to query your data, even if we’ve used the same app. So if you used my app you don’t need to worry that I can read your data, and I don’t need to worry that you can read mine.
Therefore, if I want one workspace that calculates statistics from several accounts, I have to try a method that’s more exotic.
- Load Data to a Database: I set up a workspace that transfers data from Fitbit to a database. Each person runs it as they wish, using their own connection. I have a separate workspace to create statistics from the database record. It’s easy to set up, but relies on users to run the workspace at regular intervals (they could, of course, set it up on a schedule).
- Set up Fitbit Friends: All my Fitbit users are “friends” with my account. Then my workspace can fetch statistics from their accounts. Still simple, but the statistics I can obtain from friends are less detailed than with full account access.
- Upload all Web Connections: Users export their Fitbit web connections and pass them back to me. I import their connections and run my master workspace once per connection. I can think of two possible methods:
- On Server: Set the connection to a published parameter. Use the FME Server REST API to read a list of available connections (/security/categories/NamedConnections/resources is the command to use I’m told) and then run the workspace, each time sending a different connection to the published parameter.
- On Desktop: Set the connection to an attribute. Put a list of connection names in a text file. Read the text file. Each line becomes the attribute used as the connection name.
I’m by no means a web services security expert, but I don’t see another way to have a master workspace create statistics from multiple users. Let me know if I’m wrong.
Other IoT Devices and FME
Browse to the Manage Web Services dialog in FME Workbench and you’ll see other supported fitness devices.
Strava, for example, is a multi-device app that records physical activity. To me it seems focused more on running and cycling, rather than general exercise and movement like Fitbit. The Strava web service can be downloaded from the FME Hub and used to fetch data using the Strava API, again in much the same way as I did with Fitbit.
Besides that, other devices I notice in the list are not for activity tracking. The Nest web service is for accessing Nest IoT devices, such as the Nest Thermostat or Nest Cameras. There’s even a video on YouTube that shows how to use the Nest API with FME. The Netatmo web service is for accessing similar devices for air quality monitoring, including thermometers and smoke alarms.
Creating Your Own Web Service
If you want to connect to an IoT device that doesn’t have a web service defined, then it is possible to create your own. At the foot of the list of available services, click the + button and then select the authentication type:
Then (for a Token Service) it’s a case of filling in the token request fields such as URL. Mostly the developer information for the device will include that. Whether it’s light bulbs or BBQ, you should be able to figure out which values go where to create your new web service.
I will say that creating your own web service in FME is not always simple. It’s not the FME part that’s so tough; it’s because the authentication standards that exist provide only loose guidelines, and different web services interpret these guidelines in slightly different ways. It can be a bit of a battle (I spent a week or so trying to get a response from one API, and eventually gave up) but definitely possible. You can always ask for help on the knowledgebase if necessary.
Summary
When it comes to the Internet of Things, FME has you covered. If the device has an open API then you can use FME to receive recorded information, or even to send commands to the device (yes Tesla, I’m thinking of you)!
Throw in FME Server (or FME Cloud) and you get the power of notifications as well. This allows event-driven processing; so you can automatically contact your device in response to an event from elsewhere (the weather forecast is cold, turn on the heating), or make a notification elsewhere based on information from the device.
As always, I hope that this content fires your imagination to do great things with FME. We have just decided that there will be an FME Holiday Hackathon for 2018, so that would be a great opportunity to try out some IoT integrations. Last year’s winning entry was a workspace to build 3D gingerbread houses. Maybe this year someone will be able to prepare a Christmas turkey using FME workspaces to control the cooking process? Or track your post-holiday exercises? I can’t wait to see what you come up with.
Keep a watch on the blog and on the Knowledgebase for official announcements.
Mark Ireland
Mark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)
