For Safe’s 10th Anniversary in 2003, our artist created us a Tesseract-inspired commemorative poster proclaiming that “Data Is Power”. But recently I’ve been thinking that perhaps we’ve had it slightly wrong all this time, and that true power may come from slightly more than just data after all…
As we’ve stated before, we treat FME’s performance very seriously at Safe. “If a new release is not faster than a previous release, we won’t ship it”. So when we heard rumours of a customer experiencing a slowdown in FME 2012 vs FME 2011, we began the investigation. The customer was very cooperative and supplied log files and transformer profiles, but it was clear that we were going to have to roll up our sleeves to get to the bottom of the issue. So the big ask – can we have a copy of the workspaces, and, gasp, the source data?
So imagine my surprise when the customer in question came back and said that actually there would be no problem to give us the source data. Indeed, they didn’t see much value in it particularly…
Typically that last ask is a deal breaker for many of our customers, because the source data is very core to their business: highly proprietary and the competitive advantage that they have over their rivals. So imagine my surprise when the customer in question came back and said that actually there would be no problem to give us the source data. Indeed, they didn’t see much value in it particularly – that data was easily available to all who asked. Instead, it was the FME Workspace, which expresses their rules governing the transformation of the input to the output, that they were a bit more concerned about. That was where they felt the real value was. So clearly, for this customer, the data alone was not the power. Instead, data combined with the transformation is the source of their power. It is the results of this potent cocktail which fuel their decision making, and not just the inputs to the transformation.
The Examples Are Everywhere
As Kevin and I were preparing for this Thursday’s webinar on creating 3D mashups by integrating CAD, GIS, and BIM, we were struck at how we see the above pattern repeating itself over and over in the examples and case studies we’re reviewing. Whether its our friends at Shell Canada, merging a variety of innocuous inputs to create stunning 3D PDFs, or the folks at UMass at Amherst who’ve created a 3D GIS dataset for space planning and facilities management out of 2D CAD floorplans and a space management database, or our partners at Sweco taking legacy drawings and measurements and creating a 3D model of a nuclear powerplant to aid decommissioning planning – the pattern is one of taking relatively bland data and fusing it together with just the right transformation, to magically create something vastly more useful for understanding and, ultimately, decision making.
So nearly 10 years ago we said “Data Is Power”…maybe for our upcoming 20th we’ll have to remake the poster and declare that “Transformation Is Power”.
Swedish FME god and Thursday Webinar co-presenter Ulf Månsson says that when you have an FME hammer, every data transformation problem looks like a nail…
Today most software vendors are able to run their applications in the cloud thanks to the work by folks at companies like Amazon, Rackspace, and others. This gives users the option to deploy their applications in the cloud.
Safe is no exception as we no longer have any physical servers for our web presence. We now run all of our training classes via Amazon Web Services (AWS), and allow our customers to deploy FME Server on AWS (our trial program gives people the option to use an AMI, or install on their own cloud-based machine). There is however a big difference between simply running in the cloud and leveraging the cloud.
New Capabilities Offered by the Cloud
Many applications are really “spiky” in terms of their usage patterns; that is they tend to have periods of low (or consistent) use, followed by times of heavy use. This usage pattern is problematic for traditional IT deployments and is one example where the elasticity of the cloud can take solutions to the next level. A cleverly architected cloud application will scale its resources to match the current demand of the system. The trick is to exploit the AWS (or other) licensing model to do the scaling in a way that minimizes cost.
Highway Model vs Electricity Model
In the past organizations have had to use the “highway” model when it came to compute resources. That is organizations had to build infrastructure to handle the peak times. As with highways this approach tends to force a compromise as it is rarely feasible to build the infrastructure to handle the peak loads with no degradation in service. The result is that during quiet times there is excess capacity sitting idle while at peak times the system is overloaded resulting in slower response.
The cloud gives the ability to deploy massively scalable solutions where resources are added and dropped as needed. This is similar to how we purchase electricity. We pay only for what we use.
Electricity Model 2.0 and Amazon’s Licensing Models
The electricity model is the basis to Amazon’s three general types of pricing: on-demand, reserved instances, and spot market.
- On-Demand: This is AWS’s default and is great for experimenting or for uses in which the machines are not up all the time. Here you are paying a premium hourly rate for the flexibility of making no commitment to Amazon for their services. No Commitment = higher cost. While this works for any application that is always running you can significantly reduce costs by using a different licensing model. From an electricity model this maps to how small consumers pay for their electricity. I pay only for what I use but I am not committed to buying any.
- Reserved Instances: With reserved instances you are entering into a more committed relationship with Amazon. I won’t get into all of the details, but for an annual fee you get a reduced hourly cost to your instances. There are several levels which all essentially follow the rule of “The bigger the commitment the lower your total cost” – and using them you can easily reduce your costs by 40%. Again, this maps to how very large power consumers negotiate with the power companies. A large power consumer would guarantee to buy “x” amount of electricity for a lower unit cost.
- Spot Market: Here you bid on instances with the price fluctuating depending on current market demand. You first specify a maximum price you are willing to pay. Then you get access to that instance until the bid price rises above the maximum price that you are willing to pay. Used properly you can reduce your costs by using “excess cloud capacity” to improve the throughput of your system. However, there is no guarantee that you will get resources, an important consideration for mission critical applications. In the electrical market, the spot market too exists enabling companies to buy electricity at lower rates when there is excess. As with any spot market you need to be careful as you could also pay more if resources become tight.
So What’s the Right Model?
I would argue that there’s no right answer, but it’s up to each organization to assess its own needs and requirements. At Safe, we’re using reserved instances more and more as we identify how many AWS instances we need on a continual basis. We still use on-demand for experimenting and for demand peaks.
For our clients and anyone running on AWS, we recommend you seriously examine the pricing models. For anything that has a regular usage pattern you should consider reserved instances. We are now looking at how to enable our products to play the spot market to drive down instance costs further. One product that looks very promising is StarCluster from MIT. We will keep you posted on our experience with all of this and with FME Server deployments in the cloud.
Have you deployed anything in the cloud? If so, how did it go? Which models worked best for your situation?
For me, the most significant development at Esri’s 2012 Developer Summit was the pre-release of ArcGIS Runtime. ArcGIS Runtime represents a new direction for building lightweight, focused mapping applications with Esri technology. The resulting applications are responsive, trivial to deploy, cloud friendly, and optimized for their target platforms – be they mobile or desktop.
The first hints about Runtime came at the 2011 DevSummit, more was revealed at the 2011 UC (see especially the third slide for platform support), and this year’s Esri Federal GIS Conference brought a demo and technical overview.
I see Runtime as a natural extension of two larger trends: Esri’s focus on a web service architecture (e.g. the transition towards ArcGIS Server and ArcGIS Online), and the proliferation and success of small, focused applications on mobile devices.
What is ArcGIS Runtime?
ArcGIS Runtime is a framework for developing map-centric applications with functionality added via widgets (for editing, geoprocessing, introspection, etc.). The user experience and programming model is consistent across desktop and mobile devices — “Desktop is just another device” — but is optimized for each platform’s capabilities and programming model. This means great user and development experiences on any one platform, but no portability of Runtime applications between platforms — a wise trade in my view. Map data and geoprocessing functionality can be accessed over the web (via ArcGIS online or an ArcGIS Server) or locally in tile, map, and geoprocessing packages.
The programming model closely matches existing web service APIs and treats local and remote data identically — a trick that’s accomplished by spinning up a tiny embedded ArcGIS Server to manage local resources. This web-service-centric approach implies coarse-grained interfaces and is naturally parallelizable (think performance, responsiveness).
Highlights
I found the following aspects of Runtime most interesting:
- Performance. In the demos, application start-up, map pan/zooms, and graphic animation were all extremely fast. These are recent improvements — they talked about redoing the drawing code for a huge increase in performance — and were well received. Getting the best performance (especially for start-up time) requires taking control of when the embedded services are started and stopped, a minor additional effort over this otherwise automatic activity.
- Deployment. It’s as easy as copying a folder, and nothing is left on the target machine. The minimum configuration is 10 Mb, and you add only the additional packages you need.
- Embedded Server. As above, the benefits are a consistent interface to local and remote data and processing, and good alignment with ArcGIS Server and ArcGIS Online. As a developer, I also find the minimalistic embedded web server and common C++ core (leveraging process-level parallelism and shared memory) cool.
Licensing
The most commonly asked questions about Runtime were around licensing. Some of the responses were inconsistent (and all were subject to change), so take the following with a large grain of salt:
There is a free (i.e., no extra cost to deploy) version, and a non-free one, along with three optional extensions (Network Analyst, 3D Analyst, Spatial Analyst). Licenses are purchased in packs of 25 (or possibly 50) and follow an honor system. (You embed the license key into your app, and then pay Esri based on the number of times you distribute it.) The non-free license is expected to be similar to MapObjects and in the “low hundreds of dollars” per deployment. Maplex labelling is included. If the customer already has an ArcObjects license, nothing further is needed.
Coming Soon
Two major upcoming features in Runtime were announced: 3D viewing (right now you can do 2D viewing and 3D analysis), and improved support for offline or partially connected scenarios.
Future Thoughts
One of the questions I’m left with is, “What does it mean for a future ArcGIS Desktop to be based on Runtime?”. I missed the ending session where Scott Morehouse revealed this would be the case (but my colleagues didn’t; see also Dan Levine’s comments on the GISi blog). Perhaps it means we’ll see ArcGIS Desktop on Linux or Mac? Is it possible to build such a rich editing experience on top of the coarse-grained Runtime? All will be revealed in time.
We’re also thinking about how FME might leverage Runtime’s new capabilities. Could we use Runtime as another option for reading and writing File and SDE Geodatabases? For now, the answer is a tentative no (especially as there is no advantage over File Geodatabase API for the former). What about exposing the included Geoprocessing capabilities in FME Desktop? (Runtime is off-limits to FME Server which would have to use ArcGIS Server.) Again, the answer is a tentative no, as there isn’t a good alignment between that use case and how Geoprocessing works in Runtime. (You define geoprocessing packages with logic and data in ArcGIS Desktop, and then pass small amounts of data to/from the Geoprocessing Tasks in Runtime.)
Conclusion
ArcGIS Runtime is a significant leap forward for building and distributing focused, map-centric applications based on Esri technology. It’s well aligned with the overall Esri ecosystem (including the increasingly highlighted ArcGIS Online), and represents a deep commitment for the web services architecture – both locally and over the web. It will be very interesting to see how these technology bets pay off.
Are you planning on investigating ArcGIS Runtime? Can you see ways of us using the runtime in FME that I’ve missed? How do you think this successful mobile paradigm (many small focused applications) will affect desktop computing in future?
