Hi FME’ers,
With the FME World Tour well under way, and FME coin photos starting to appear all over the map, it’s time to cover a few of the new features available in FME2016.1.
This article is going to cover something I never thought would be possible: the ordering of operations in a workspace.
The Background
Take a look at this workspace (click to enlarge):
First I’ll explain what it does. Basically it reads new planning applications (i.e. polygons on which development will take place) from a CAD file and writes them to a corporate database. It also determines which address is associated with each application by doing a point-in-polygon test (PointOnAreaOverlayer transformer). If no address falls inside the planning application, then one is created and added to the address table.
Simple, right?
The Problems
There are two problems in this workspace that relate to the order of features.
Firstly there are two Creator transformers, each of which triggers reading of data. If I want to use the Areas First parameter in the PointOnAreaOverlayer then I need to make sure the areas are actually first, i.e. Creator2 executes before Creator1.
But how can I do that? Traditionally it would be near impossible to tell in which order features would be created and processed. You could set Creators to execute either before or after Readers, but you couldn’t set the order in which each Creator was executed.
Secondly, look closely at the output schema. Notice I am writing to a database where the planning application table uses addressid as a foreign key. What you can’t see is a constraint on the table that addressid must match a record in the address table. In short, when creating a new address I need to write the address first, before I write the planning application that references it.
But, again, how can I do that? There are multiple connections coming out of the CreateAddress custom transformer and no way to tell which order the connections will fire.
Well, in FME2016.1, both of those issues are taken care of.
Creator Order
Right-click on a Creator in 2016.1 and – assuming there are two or more Creators in the workspace – there is now an option to Set Creator Runtime Order…
This opens up a dialog that lets you adjust the order in which Creators are executed:
Notice that this doesn’t just apply to “Creator” transformers, it also applies to the different forms of that; such as SQLCreator, 2DGridCreator, and PythonCreator. Basically any transformer that has an output port but no input (yes, the Player transformer is included in that).
Combined with the Creator “Create at End” parameter, I could even set up a Creator-Reader-Creator sequence in my workspace.
Setting the order in which data is created will be a big benefit in many scenarios. As with the above workspace, it is going to be particularly important as we switch focus from Readers to Creators/FeatureReaders, because it effectively lets you set the order of those FeatureReaders.
In my workspace above I can ensure that Creator2 is the first to execute, meaning that FeatureReader2 reads its data first, and that I can confidently set the Areas First parameter in the PointOnAreaOverlayer.
NB: Reading data in order like this, doesn’t mean it will stay in that order forever. If I put a Bufferer, for example, in the polygon stream, it could hold up the polygons and let the points overtake them. In short: beware of group-based transformers.
Connection Order
The other issue was connection order. In 2016.1, where there are multiple connections issuing from an output port, right-click on one of them (or on the output port itself) and you again have the option to set their order:
Like the Creator transformers, the dialog lets you order the connections connected to that port:
In this screenshot I’m setting the order in which I trigger connections to SQLExecutor transformers – but it doesn’t always have to be for databases or to the same transformer type. This is functionality that you could use almost anywhere in a workspace.
In my workspace above I can force the address connection (from CreateAddress:Output to AddressTable) to be triggered before the planning application connection. This means that address features get written first and the foreign key constraint is satisfied.
NB: One requirement for my specific example is that my address records need to be ‘committed’ right away, not just ‘written’. To ensure that happens I would set the transaction interval to 1 on my database writer. That way each address is committed as soon as it is written.
You might also have realized (if not, don’t worry, it took me a week to notice) that I could have used just one Creator transformer with two output connections, using the connection order instead of Creator order; the result would be exactly the same.
Transformer Ports
The other reordering I can do is that of the ports on a transformer. Did you notice that nasty overlapping connection in my workspace?
I could avoid that by reorganizing my workspace so that the CreateAddress custom transformer is lower down the screen. But in 2016.1 I can simply switch the order of the output ports on the Tester transformer:
With that right-click I can move the port up or down, uncrossing my connections. The same behaviour is available on both input and output ports.
I could also change the order of the output ports on my FeatureReaders, assuming there were multiple layers of data being read.
The one warning I have is not to expect the processing to be any different. Reordering ports doesn’t affect the order in which features emerge. That’s entirely down to the underlying factory and you have no control over it. So I shouldn’t expect Failed features to emerge first, just because I’ve moved their output port to the top of the list.
Oh, one other thing; some transformers – the AttributeFilter springs to mind – already allow you to change the output port order inside the parameters dialog. In that case these new options don’t appear. You would set the port order with parameters, as you always did.
This functionality got some rave reviews on world tour events so far. As I always say, it’s the little changes that mean the most!
One More Thing…
Just a quick, final comment. FME2016 has a FeatureWriter transformer, so I could replace my writer with that transformer. The benefit there is that I could follow it up with SQLExecutor transformers to confirm that the join was carried out correctly (or any other test I wanted to do):
The reason I didn’t do this is that you can’t see what attributes exist on the output schema of a FeatureWriter, at least not from a screenshot of the canvas, and I wanted you to see the common addressid attribute. But in a ‘for-real’ project, I’d probably go with the FeatureWriter.
In Conclusion
With FME2016.1 your workspaces really can be built to order, or rather re-order.
As I said, I really never thought that we could create functionality to handle the order of features – and particularly connections – so it’s great to see it happen. One customer even said that whoever worked on this deserves a medal!
So three cheers for our developers, who are far from the simple “coding monkeys” that they modestly call themselves.
If you haven’t tried FME2016.1 yet, it is available as a “release candidate“, which means we have a build that we think is suitable for release, but we’re just running a few last tests on it to make sure. I understand it will be released officially next week.
Mark Ireland
Mark, aka iMark, is the FME Evangelist (est. 2004) and has a passion for FME Training. He likes being able to help people understand and use technology in new and interesting ways. One of his other passions is football (aka. Soccer). He likes both technology and soccer so much that he wrote an article about the two together! Who would’ve thought? (Answer: iMark)
