It’s 2017 – Introducing the Next Generation of FME Cloud Instances
We are excited to introduce a new generation of FME Cloud instances. We rolled out a set of new features over the past weeks, which started with the introduction of NGINX and was completed with the release of the new regions today. Last time we did a significant architectural change to our instances was more than two years ago, when we started leveraging local SSDs on our servers.
1. More Regions
As of today, you can launch FME Cloud instances in two new regions: Montreal (ca-central-1) and US West (us-west-2). Those two regions complement our existing three regions in US East, Frankfurt, and Sydney. If you are missing a region, do not hesitate to contact us.
2. Upgraded Instance Types
As of today, all new instances launched will leverage the new instance types. The price and existing instances will remain unchanged. See the table below for the specific instances we have upgraded. The new instance types will provide a bump in performance with new custom processors, and enhanced networking across all instance types.
|Old Instance Type||New Instance Type|
|Starter||M3.medium (1 core, 3.75GB RAM)||T2.medium (2 cores, 4GB RAM)|
|Standard||M3.large (2 cores, 7.5GB RAM)||M4.large (2 cores, 8GB RAM)|
|Professional||M3.xlarge (4 cores, 15GB RAM)||M4.xlarge (4 cores, 16GB RAM)|
|Premium||M3.2xlarge (8 cores, 30GB RAM)||M4.2xlarge (8 cores, 32GB RAM)|
|Enterprise||C3.4xlarge (16 cores, 30GB RAM)||C4.4xlarge (16 cores, 30GB RAM)|
Background on Instance Types
For Standard, Professional, and Premium instances, we simply upgraded from the M3 to M4 generation (read more here), sticking with the same instance family. The new M4 instance types support enhanced networking and feature a new custom Intel Haswell processor. You should therefore see a bump in performance—especially if you are doing network-intensive tasks.
For the Enterprise instance, we upgraded from the C3 to the C4 generation, again keeping the same instance type. The C4 generation instances feature an Intel Xeon E5-2666 v3 processor, which offers a noticeable improvement over its predecessors.
The Starter instance has gone through the most drastic change, getting switched to an entirely new family. We shifted from the M3 instances to the T2 class. Based on our benchmarking tests, we think the vast majority of Starter users will benefit from this change. T2 instances introduced burstable CPU performance based on credits. The full concept of the variable performance is quite complex. AWS do the best job of explaining in detail, but I will try to summarize it:
A T2 instance CPU has a base performance lower than usual but is able to burst its performance for a limited amount of time when CPU power is required. The amount of time an instance can burst depends on credits, which are applied to an instance over time. For a T2.medium instance, you will receive 24 credits an hour, which allow a burst of 24 minutes on one core or 12 minutes on two cores.
You may wonder what the advantage of T2 vs. M is. In short, you get an extra CPU core for the same price. We looked at usage patterns of Starter instances and noticed that almost none utilize the CPU close to 100% all the time. This means the server will be able to gather enough credits to deliver maximum performance at the moment a job is submitted.
Increase of Root Disk size
For all instance types, the side effect of shifting to the new generation instances meant we lost the local SSD. We were leveraging this disk for SWAP space, so we had to re-architect things. We ran tests and discovered we had stability issues on instances without any SWAP so we knew we needed to keep it. Therefore, we decided to add a 4GB swapfile to our root disk. This change should increase the overall stability of the server in case of high memory workloads, and we will expose the SWAP usage metric shortly. Unfortunately, due to the increased disk usage, we were forced to increase the root disk size. To keep a round number and be ready for future Ubuntu and FME versions with increased storage usage, we decided to increase the Root Disk size for all new instances from now 10GB to 15GB.
3. Temporary Disks
All instances now come with a temporary disk. This disk (EBS (gp2)) differs from the other main data disk in that it is not backed up and does not persist after pausing an instance. Because of this, you can dramatically lower your storage and backup costs by writing data to this disk that is not important. However, it also means that all data on the disk will be lost when you pause an instance, so keep that in mind.
How is the temporary disk used?
Automated temp files FME creates use this disk
Before temporary disks, if FME was writing temporary files it would use the local SSD. Even though the local SSD was fast, the disk was small and could not be increased in size. This meant if an FME workflow required more temp space than available, it would fail without a workaround. With the new temporary disks, it is now possible to pick an appropriate size and performance for your workflow (for the most customers, the default of 10GB should work just fine).
You can write your own data to the disk
Sometimes you need to pull data down to the instance for use in a translation that you don’t need backing up. To support this, the temporary disk is mapped to the FME Server Temp Resources folder. If you upload a file to that folder, it will end up on the temporary disk. This means the behavior of files in that folder change in two ways:
- They will not be backed up.
- They will be deleted whenever an instance is paused.
All other data uploaded to FME Server (e.g. FME Server Data Resources) are still backed up and persisted — this change affects only files in the FME Server Temp Resources.
This behavior was heavily inspired by workflows where an FME Workspace requires to download large datasets from another Cloud resources (for example AWS S3) to process the data on the FME Cloud instance. Those datasets only need to be on the instance for the runtime of the process and do not require a backup.
See also: FME Cloud Pricing Change