When (and How) to Switch to NoSQL
A lot of popular websites wouldn’t be able to function without a NoSQL database on the backend. I mean, using a relational database for something like a social network is like putting a Clydesdale in the Kentucky Derby. Don’t get me wrong — Clydesdales are an excellent breed and probably the best at hauling carts — but my money’s on Seabiscuit.
NoSQL databases have been gaining popularity since 2009. Facebook, Twitter, Buzzfeed, Expedia, eBay, craigslist, and Google all use them. (If Google gave its blessing, it must be good, right?)
Relational databases are designed for static schemas and consistent, reliable transactions. They are not designed to handle booming data volumes at top speed. A NoSQL approach is often preferred for:
✓ Real-time data collection.
✓ Big data storage. NoSQL database structures include key-value pairs (simple and fast), documents (store info as XML/JSON), columns (good for queries), and graphs (good for networks), among others.
✓ Agile schemas. You don’t need to define a structure and stick to it. This can change on the fly.
✓ Horizontal scaling. It’s easy to scale out your architecture when you need to.
✓ Access via APIs. NoSQL uses object-oriented programming as opposed to SQL queries.
✓ Searchable catalogues of information — text, metadata, GIS, or otherwise.
✓ Developing a mobile app or an Internet of Things device.
✓ More: Check out this guy’s list of things people use NoSQL for.
- Amazon DynamoDB
- IBM Cloudant
- Microsoft Azure DocumentDB
Problems with going NoSQL
If you need to integrate a NoSQL database into your existing systems, there are some challenges involved. For example:
- Converting a relational database into document stores (or key-value pairs, columns, graphs, etc.)
- Shifting from unstructured tabular formats, like CSV, to JSON
- Loading existing JSON / XML into your NoSQL database
- Adding spatial capabilities, e.g. geospatial indexes, geometries via GeoJSON
- Connecting to the NoSQL system’s API
- Pulling source data from both cloud and on-premises systems
- Managing big data volumes
- Using NoSQL data in all your regular applications
- Transforming and restructuring NoSQL data
The trick is to understand how NoSQL systems store data. Often, adopting NoSQL is a matter of converting your data to JSON and then loading it to the destination system.
Many NoSQL DBMSs have some level of built-in import/export tools. However, their format support is limited, and writing XML/JSON means coding.
Coding aside, the only way to switch to a NoSQL system is via software solutions. By using an automated process, you’re also enabling loading/updating at regular intervals (or in response to a trigger). This means your data store can be updated in real time. For example, you could automatically capture data from web and mobile sources and move it into your NoSQL system without touching a thing.
We have a webinar on how to integrate a NoSQL database with your existing systems (without writing any code!). We go through real demos and show you how to automatically merge a wide variety of data, extract data from NoSQL, and work with raw JSON.
Watch: How FME helps you switch to NoSQL
FME is great at processing JSON and has specialized writers for many database systems. Building a workflow to synchronize data with your NoSQL system is a drag-and-drop process, one that you can automate in real time.