The Safe Software Blog
Author:
Tiana Warner
Technical content marketing at Safe Software
Google
Get the Blog Newsletter

Delivered by FeedBurner

About Data    |   October 3, 2016   |   By Tiana Warner

When (and How) to Switch to NoSQL

A lot of popular websites wouldn’t be able to function without a NoSQL database on the backend. I mean, using a relational database for something like a social network is like putting a Clydesdale in the Kentucky Derby. Don’t get me wrong — Clydesdales are an excellent breed and probably the best at hauling carts — but my money’s on Seabiscuit.

NoSQL databases have been gaining popularity since 2009. Facebook, Twitter, Buzzfeed, Expedia, eBay, craigslist, and Google all use them. (If Google gave its blessing, it must be good, right?)

Relational databases are designed for static schemas and consistent, reliable transactions. They are not designed to handle booming data volumes at top speed. A NoSQL approach is often preferred for:

✓ Real-time data collection.
✓ Big data storage. NoSQL database structures include key-value pairs (simple and fast), documents (store info as XML/JSON), columns (good for queries), and graphs (good for networks), among others.
✓ Agile schemas. You don’t need to define a structure and stick to it. This can change on the fly.
✓ Horizontal scaling. It’s easy to scale out your architecture when you need to.
✓ Access via APIs. NoSQL uses object-oriented programming as opposed to SQL queries.
✓ Searchable catalogues of information — text, metadata, GIS, or otherwise.
✓ Developing a mobile app or an Internet of Things device.
✓ More: Check out this guy’s list of things people use NoSQL for.

This JSON snippet of a real-time bus location from the TransLink API would do well in a NoSQL setup.

This JSON snippet of a real-time bus location from the TransLink API would do well in a NoSQL setup.

Hundreds of NoSQL DBMS options are available and it’s worth researching which data structures and requirements are best for your scenario. A few choices, in order of current popularity:

  • MongoDB
  • Cassandra
  • Redis
  • Elasticsearch
  • Couchbase
  • Amazon DynamoDB
  • CouchDB
  • IBM Cloudant
  • Microsoft Azure DocumentDB

Problems with going NoSQL

If you need to integrate a NoSQL database into your existing systems, there are some challenges involved. For example:

  • Converting a relational database into document stores (or key-value pairs, columns, graphs, etc.)
  • Shifting from unstructured tabular formats, like CSV, to JSON
  • Loading existing JSON / XML into your NoSQL database
  • Adding spatial capabilities, e.g. geospatial indexes, geometries via GeoJSON
  • Connecting to the NoSQL system’s API
  • Pulling source data from both cloud and on-premises systems
  • Managing big data volumes
  • Using NoSQL data in all your regular applications
  • Transforming and restructuring NoSQL data
The above JSON from the TransLink API converted to point geometries and seen on a map.

Making it spatial: The above JSON from the TransLink API, converted to point geometries and seen on a map.

Using automation

The trick is to understand how NoSQL systems store data. Often, adopting NoSQL is a matter of converting your data to JSON and then loading it to the destination system.

Data to JSON to NoSQL

Many NoSQL DBMSs have some level of built-in import/export tools. However, their format support is limited, and writing XML/JSON means coding.

Coding aside, the only way to switch to a NoSQL system is via software solutions. By using an automated process, you’re also enabling loading/updating at regular intervals (or in response to a trigger). This means your data store can be updated in real time. For example, you could automatically capture data from web and mobile sources and move it into your NoSQL system without touching a thing.

Webinar

We have a webinar on how to integrate a NoSQL database with your existing systems (without writing any code!). We go through real demos and show you how to automatically merge a wide variety of data, extract data from NoSQL, and work with raw JSON.

Watch: Tips for Adopting NoSQL Databases using FME

Watch: How FME helps you switch to NoSQL

FME is great at processing JSON and has specialized writers for many database systems. Building a workflow to synchronize data with your NoSQL system is a drag-and-drop process, one that you can automate in real time.

Learn more about building your own NoSQL integration workflow, and try FME for free.