The Safe Software Blog
Author:
Mark Ireland

Google
Get the Blog Newsletter

Delivered by FeedBurner

About FME    |   September 27, 2008   |   By Mark Ireland

FME Evangelism – Smallworld UC Special

Contents:
FME/Smallworld Background
FME and Smallworld SIAS
Behind the Scenes
Technical Details of a Smallworld Workspace
Streaming, Linking and Uploading
What’s New in the FME SpatialBiz Plug-In
Smallworld 4.1 Support
Training Offer

This FME Evangelist post is a special edition for the 2008 Smallworld Americas Users Conference.

As P.G.Wodehouse said in many of his novels, the problem with writing is that you never know how much past history you have to mention to bring your readers up to speed.

In this posting I’ll be mainly covering a specific scenario: the use of FME with the Smallworld Internet Application Server (SIAS). Since the idea is to demonstrate this technology to Smallworld users who might not be familiar with FME, I’m going to take the time to introduce FME, go over the user experience, and cover various user options, before diving down into the technical information that reveals how the process is all put together.

But even experienced users should find this scenario of interest, so while the first section is your cue to go walk the dog, make a cup of tea, or catch up on your emails, the second section is where you need to be back and alert and ready to see something cool,

Regards,

FME/Smallworld Background
For those who are really new to FME, welcome!
FME is an ETL tool for moving, translating and transforming spatial data.

FME comes in two flavours: desktop and server. Both make use of FME Workbench, a tool for graphically defining flows of spatial data. Perhaps the easiest way to explain is to show a screenshot:

Here we are reading a set of city park data (MapInfo Tab format) and a set of Hydrography data (ESRI Shape format). We want to output only hydrography features where they fall inside a park, so we Clip the hydrography using the park features as clip boundaries. Then for good measure we calculate the remaining length of each hydrography feature and write them out to a GML format dataset.

So the above is an example of extract, transform and load (ETL), and one in which we could use Smallworld as a source or destination almost as easily. As FME comes in two flavours, we could either execute the translation on our own computer (Desktop mode) or send it off to be run on a remote machine (Server mode).

Perhaps the last word should go to our marketing team:

FME is used by thousands of customers worldwide in a variety of industries including government, utilities, and petroleum. Its powerful data access technology also makes FME the choice of leading GIS, CAD, and database vendors for integration into their own solutions. Designed for true data interoperability, FME unleashes spatial data so people can use it where, when, and how they want to.”

FME and Smallworld SIAS
For those FME users who are not familiar with Smallworld, it is a GE Energy product that is described as a “Geospatial Asset Management” application. It’s much used in the telecommunications and utilities fields as it has excellent tools for managing complex networks of distribution infrastructure.

Smallworld Internet Application Server (SIAS) is an add-on that deploys Smallworld tools through a web browser. Not only does it provide the means to visualize both map and attribute data, it also allows access to Smallworld functionality such as data analysis and maintenance.

At the upcoming Smallworld user conference, Alfred Sawatzky of Red Planet Consulting, and Mark Stoakes from Safe Software, will present a workshop that demonstrates how we integrated SIAS with FME Server.

As mentioned, FME Server is a means by which to run spatial ETL processes. The reason to integrate it with SIAS is that this creates a very powerful combination that can be used for a self-service data download tool, a service to disseminate data (as a KML, WFS or WMS service for example), and/or a tool for publishing updates into an existing Smallworld database.

Below: General Architecture of an FME Server and Smallworld SIAS integration (click to enlarge):

The demonstration shows how commands were built into the SIAS interface to send translation requests to FME Server for data download, data streaming and data upload. This small customization is required because we want to handle these processes transparently, without the user having to go through the usual Smallworld-FME interface; see the section “Behind the Scenes” (below) for more info on this.

Here’s the basic process for using this combination of tools:

Start SIAS Server and Client.

Below: The SIAS client looks like this (click to enlarge):

Notice how there are a number of tool options. The one we want is “Create Bounding Box” (below):

So now I zoom into the centre of Cambridge, and use the bounding box tool to define my area of interest:

Now I select the download tool from the menubar. These options are not part of the standard SIAS interface, but were added for this demo using some Javascript and HTML customization:

This action opens up an interface to the FME Server which includes an order form (click to enlarge):

Click Run here and the FME process is kicked off and the data returned:

Now I just need to simply click on that link to retrieve a zip file that contains an extract from the Smallworld database in the format of my choosing.

Of course the procedure as described above uses all the default values. It extracted all the objects for the defined area and output them in AutoCAD format with a particular coordinate system (WGS84 lat/long).

But notice that the order form has settings for format type, objects to read and coordinate system. So the user can decide which objects to output, to which format of data he wants, and in whatever coordinate system he desires.

To see more of this procedure you can visit Alfred’s blog and view a number of demo movies:

Starting the SIAS Server
Starting the SIAS Client
Downloading All Data
Downloading Defined Areas
Uploading Data

Behind the Scenes
The above demonstration reveals great potential, but little detail on how this miracle is accomplished.

The secret is that FME Server has what is known as a Model Driven Architecture; it determines how to transform and translate data from a set of rules known as a model.

Where does the model come from? Well, from FME Workbench which a user uses to author a model (we would call it a “workspace”) like the park/hydrography processing in the previous example. This workspace defines how to read and write the data, and how to transform it in between. When it is uploaded to the server it can be used by anyone without the requirement for specialist knowledge of FME or the data itself, or even a copy of FME!

Below: This is the workspace we use in our example (click to enlarge):

Yes this workspace is a little more complicated than the park/hydrography example; we’ll go into more technical detail in the next section for those that want it.

So this workspace/model is uploaded to the FME Server and awaits execution. This is simply what SIAS does – call up a page with a run button where the workspace process can be carried out.

Another innovation – for those of you who have used FME with Smallworld – is that all this is carried out without the need to go through the Smallworld-FME User Interface!

Typically GE’s Smallworld reader and writer in FME can only communicate with the Smallworld database through this interface (below – click to enlarge), which has to be set up first by the user.

Obviously the end-user in a server installation is in no position to manipulate this interface so Alfred wrote a slick piece of magik code which creates a direct connection between the FME Server engine and the Smallworld database.

To be more precise, Alfred runs a TICS server which waits for commands via a command line. In the workspace a Startup Python script will – when the workspace is run – issue Magik commands through the TICS server to the Smallworld interface, without the user needing to take any action.

Technical Details of a Smallworld Workspace
Of course the procedure as described above uses all the default values. It extracted all the objects for the defined area and output them in AutoCAD format with a particular coordinate system (WGS84 lat/long).

But notice that the order form provides a whole bunch of different options, and these are where there are technical items worth noting in the FME Workspace.

Format Selection
Although different formats can be output by using a workspace with multiple writers, the more elegant solution is to use a Generic Writer. The generic writer has no defined format until the workspace is run. At that point the user chooses which format it will write to.

How does the user choose? Well that’s done with a published parameter. The generic writer has a setting called “Output Format” which you can choose to publish, in which case the user will get to choose from the whole list of available FME formats.

To restrict the list of formats on offer the workspace author should create a custom published parameter; one of type “Choice” which has as options all the required formats. Then the “Output Format” parameter needs to be set up to ‘share’ this custom published parameter.

When a workspace is run in FME Server, these published parameters are automatically detected and displayed in the “order form” in a manner compatible with their parameter type.

See fmepedia for more information on published parameters, and check out this FME Evangelist posting for an example of using the Generic Writer with a custom published parameter.

Coordinate System Definition
Coordinate System Definition is also acheived through a published parameter. Here the parameter does not need to be custom, we simply publish the destination dataset coordinate system setting (below):

Collection Name Selection
Collection Name is also a published parameter, but this is where things diverge a little from the FME standard. Normal practice would be to map this to the parameter called “Feature Types to Read”. This would let the user select which layers are read from the source dataset.

However, because of how FME and Smallworld communicate, the Feature Types to Read parameter cannot be used. What happens instead is that Alfred’s Python script passes the information directly to Smallworld as parameters in his TICS command.

Above: The collection published parameter hard codes all the available object types.

Bounding Box Clipping
Again, Alfred’s Python script passes these values directly to Smallworld. However, for non-Smallworld FME Server workspaces there are a couple of techniques to use: Reader Parameters or a Clipper.

When a reader supports setting a bounding box then these parameters can be published and used as the area to read. Typically these are only supported by formats with a spatial index, which is usually database formats.

When a reader does not support setting a bounding box, then parameters are usually used to define a new feature (with a Creator2DBoxReplacer combo) which acts as a clip boundary in a Clipper transformer.

The prior method is usually preferred, since it restricts the amount of data read, whereas the second method simply reads ALL the source data and then clips it to size.

Streaming, Linking and Uploading
In this example Alfred and Mark do a data download to KML. The user is prompted to save a KML file and can open it in a KML browser such as Google Earth. However, FME Server also lets us stream data or link to it, and even permits data uploads into a Smallworld datastore.

Data Streaming
By registering an FME workspace as a data streaming service, when the user runs it the output will be opened directly in the matching application. For example, a streaming workspace with a KML writer will cause the output to open directly in Google Earth, rather than needing to be saved first. Of course, this requires a format that has a matching application.

An excellent example of data streaming (using KML) is
this one on our community wiki, fmepedia.com
– here we stream earthquake data from a live
USGS feed directly into Google Earth.

KML Network Link
By registering an FME workspace as a KML network link, running that workspace will also cause its output to open directly in Google Earth. The difference is that it will be a link, not the actual data. The link is like an automated FME Server request, which is triggered at user defined intervals to fetch updated data from the FME Server. This is useful for situations where the data is constantly changing, for example vehicle tracking.

fmepedia also has this
great example of KML network links
– a set of vehicle tracking data is constantly
updated and streamed to Google Earth

Data Upload
Creating a workspace to upload to Smallworld is a little more challenging, especially when you want to create multiple geometries, but still feasible for most users. And in the same way that a download can be run using FME Server, so can an upload. The upload can even be started from within SIAS.

Below: On running the workspace, FME Server permits you to upload the source data:

Below: Run the translation and FME reports it has completed successfully:

Below: After having run the translation, SIAS shows that the data has been uploaded (click to enlarge):

The really nice thing about the upload is that Alfred’s script automatically creates a new database alternative (version in non-Smallworld terminology) which saves having to manually create one and set it to be writable, as you would have to do with a traditional desktop FME translation.

What the workspace doesn’t do, but could easily be upgraded to handle, is data validation; so that non-permitted geometries and attributes can be filtered out before attempting to upload the data.

What’s New in the FME SpatialBiz Plug-In
As noted, there is a standard Smallworld reader/writer within FME, created and maintained by GE. However, there is also an alternative reader/writer in the form of the SpatialBiz plug-in.

The SpatialBiz plug-in, by Spatial Business Systems of Denver, Colorado, opens up the Smallworld database to FME, without the need to go through an intermediate interface. As such it makes FME Desktop much easier to use (with Smallworld data) and makes FME Server compatible with Smallworld almost out-of-the-box. A secondary benefit is that you only need to purchase an FME Oracle Edition license, as opposed to the standard FME Smallworld Edition which the GE reader/writer requires.

Anyway, this plug-in has undergone recent updates which have produced many more key benefits.

  • It can now return the sys_id field values generated during a translation run and retain these for use in looking up foreign key values and creating Smallworld joins between related records.
  • It is now compatible with the FME keyword fme_db_operation. This keyword controls how data is inserted into databases by FME. Its support means that the SpatialBiz plugin can (among other things) carry out incremental updates on Smallworld databases.
  • It will soon be able to write non-defined attributes, ie attributes that are not part of the schema. This is being implemented for SpatialBiz 3.3, scheduled for release with FME 2009
  • It supports FME Text encoding for import / export of Smallworld data. Data represented in local character sets can be translated as with other FME supported formats.

Smallworld 4.1 Support
Q) Does FME support Smallworld 4.1?
A) Yes. However you do need to use an updated DLL file. If you look in your FME2008 installation folder you will see we ship DLLs for both 4.0 and 4.1:

Notice there are 3 DLLs. swordswaf is the one we use. By default it is a copy of the 4.0. So, if you wish to get support for 4.1 you simply need to overwrite it with a copy of the 4.1 DLL.

Interestingly, although official support started with FME2007, because the FME components of the reader/writer are unchanged, you may find (unofficially) it works with FME versions back as far as FME2004-ICE, provided you copy the DLL into the right location.

Hopefully in the future, the Smallworld-FME interface will be backwards compatible so that we can use the newer DLL by default and still support both versions.

Training Offer
The following obviously comes straight from our marketing team. Who else would call Surrey beautiful? (As Dylan Thomas might say, it’s an ugly, lovely town)…

You are cordially invited to join us in beautiful Surrey, BC, Canada this fall for a complementary half-day public training course on FME Smallworld. Take advantage of this special opportunity to not only learn how to use FME, but to also see firsthand how you can apply FME’s powerful spatial ETL technology to your Smallworld environment.

Simply register for one of these FME Training Classes and receive a free half-day day Smallworld training:

Option A
– November 13-14 – FME Basic 2-day Training
– November 15 – FME Smallworld ½ day Training (Free!)

Option B
– December 9-11 – FME In-Depth 3-day Training
– December 11 – FME Smallworld ½ day Training (Free!)

Both classes will take place at Safe’s corporate headquarters in Surrey, Canada.

For more information or to register, contact us at 604-501-9985 ext. 287 or training@safe.com.

Further Technical Information
Further technical information for FME and Smallworld can be found on our community wiki, fmepedia.com, which has a whole section devoted to Smallworld related FAQs.

There is also a general Smallworld page on the Safe Software web site, plus a page written by the Professional Services team on Reading or Writing Smallworld Data with FME.

Credits
Thanks to Michael Weisman and Mark Stoakes at Safe for your help and technical support. Special thanks go to Alfred Sawatzky and Karthikeyan Bollu Ganesh of Red Planet Consulting for his work on integrating FME and SIAS.

This Edition of the FME Evangelist…
…was written to the tune of….. well, I haven’t had much time to listen to music recently. So firing up iTunes and picking something at random I plumped for one of my favourite bands Deacon Blue.

The chosen track is Chocolate Girl from their first (and best) album, Raintown.

As a comment on YouTube says: “Perfect lyrics and melody. Saw a concert once and man were they good”. Other good tracks I see on YouTube are Twist and Shout, Your Swaying Arms, and Real Gone Kid.