NoSQL Databases: Exploring What’s Possible with FME
Earlier this month, my colleague Tiana outlined why some organizations are adopting NoSQL databases into their operations. In this post I’ll take a look at a couple of ways FME can be used with NoSQL databases to take advantage of their capacity, performance and agility.
Example 1: Create a Data Warehouse
Goal: Aggregate data from different relational databases into NoSQL and create a central, searchable data warehouse for your organization.
Different departments within an organization often use relational databases that are finely tuned to the unique requirements of their operations. The problem: these independently functioning systems can result in productivity-squelching data silos.
The solution: aggregate all this data and use a NoSQL database as a central repository. Their capacity, search performance and unstructured nature provide a perfect environment for large volumes of structured data from disparate relational databases to coexist. To achieve this the data must be transformed into JSON and this is where FME is a big help.
Create an FME workspace that reads data from the enterprise relational database in question (Oracle, MS SQL Server, PostgreSQL, MySQL, etc) and loads it into your preferred NoSQL database (MongoDB, Elasticsearch, Azure DocumentDB, Amazon DynamoDB, IBM Cloudant, or CouchDB). FME’s NoSQL writers will automatically transform the tabular data into JSON documents.
Add any of FME’s JSON transformers to the workspace to build more complex JSON documents. For example, make changes to contents, create JSON lists, and reorganize the way the JSON is structured to suit your application’s needs.
To ensure NoSQL databases are always up to date, automate your workflows so it runs at scheduled intervals or in response to triggers using FME Server or FME Cloud.
Tip: Create a searchable metadata catalogue by using FME to bring the metadata into a NoSQL database. Metadata can include information on the physical location of a file such as a URL or the owner’s email address.
Example 2: Create a Reservoir for IoT Data
Goal: Automate the collection of data from the Internet of Things into NoSQL databases.
Typically Internet of Things (IoT) data is collected in a time series from sensors on a continuous basis and is made available through web services as JSON. For the same reasons outlined in the first example (large-scale storage capacity, agile schemas) NoSQL databases make great reservoirs for IoT data. The challenge is loading this data as soon as it is collected.
Create an FME workspace that extracts the IoT JSON from a web service, transforms the JSON, and loads it into your NoSQL database. Deploy this workspace using FME Server or FME Cloud, and configure it to run whenever new data is available from a sensor.
I hope these examples have inspired you to explore ways to use FME to take advantage of NoSQL databases in your organization. We explain these examples in more detail with demos in our recent webinar, Tips for Adopting NoSQL Databases using FME.