FME 2013-SP1: Conditional Processing in FME

FME 2013-SP1: Conditional Processing in FME: By Mark Ireland

Dear FME’ers,
FME2013-SP1 is released today and, to celebrate, here’s an article all about “conditions” in FME workspaces.

Conditions can be described as IF THIS, THEN THAT, sometimes with an ELSE tacked on. For example, IF you’ve attended an FME Desktop training course, THEN you’ll know there is a section on Conditions, ELSE this will be entirely new to you!

Filtering (or Branching) uses conditions to subdivide data as it flows through a workspace. As an analogy, it’s like IF you are a BUS, THEN follow this road, ELSE, IF you are a CAR, THEN follow this road.

In FME the GeometryFilter transformer is a good example. It has multiple output ports, each of which carries data off in a different direction.

But I’d almost call the GeometryFilter a “passive” filtering transformer – FME defines the conditions for dividing data, you don’t. Conditional Filtering is where the decision about which features are output to which port is decided by a user-defined test. The Tester transformer, among many others, is the prime example of this.

Attribute Mapping is another common use for conditions in FME. Data doesn’t get divided but instead an attribute is set to define the result. To use the same analogy, it’s like IF you are a BUS, THEN you get a red sticker, ELSE, IF you are a CAR, THEN you get an orange sticker.

Conditional Mapping is – like conditional filtering – where the user defines the tests and conditions for the mapping to occur. Again there are many transformers that can do this.

This has particular relevance because there are some excellent updates in FME 2013-SP1 – hence a key reason I’m writing this article! But I’ll also show which transformers are best suited for which scenarios, and that will be useful regardless of your FME version.

Let’s start with filtering data.

The Tester
As you might already know, the Tester is the #1 most-used FME transformer. Since the implementation of the newer TestFilter, its popularity has declined a bit, but it’s still more used than any transformer in any category. That shows, I think, how important conditional filtering is.

The Tester allows you to set up a conditional test to apply to each feature. The test can be a single clause like this:

…or multiple clauses using an AND or OR like this:

For these simple scenarios you either check whether the feature passes one test (IF THIS, OR THIS, OR THIS, OR THIS, etc) or whether the feature passes all tests (IF THIS, AND THIS, AND THIS, AND THIS, etc). Basically it’s either all ORs or all ANDs.

But notice the other option there, Composite Test. This allows you to do a mix of ANDs and ORs, like so:

In that case a feature passes with a pass for clauses 1 OR 2, AND clauses 3 OR 4. This is useful because it enables you to put the test clauses in one Tester, so there’s no need to use multiple Testers to handle multiple clauses.

Also, because you can do multiple clauses it means you can test different attributes simultaneously. As we’ll see, that’s not something all transformers can do. So you could test myAttribute1=x AND myAttribute2=y

Of course you can now test for other things than attribute values – published parameters being the obvious one – and there are many more operators than the simple “equals” test, but that’s not important here. What I’m trying to get across is that the Tester is for a SINGLE test, whether it’s made up of multiple clauses or not.

The TestFilter
The TestFilter carries out the same sort of tests as a Tester, but it can incorporate MULTIPLE tests.

Think of the TestFilter as being equivalent to a string of Testers. For developers you might think of it as similar to CASE or SWITCH commands (or IF, THEN, ELSE functions) in a programming language.

The dialog looks like this (in SP1 it has been very slightly rejigged):

Each time you double-click on the Test Conditions column you open a dialog – exactly the same as the Tester – to define a single test. Because it’s the same dialog you can set up the same multi-clause, multi-attribute tests again (only here you can create multiple tests too).

Also, because there are multiple tests going on, instead of a simple PASSED/FAILED output port, you get to name the output ports for features that pass a test:

So, when a feature passes the AorB test, it emerges from that port. If it fails that test then it goes onto the next test (CorD) to see if it passes that. If it fails ALL of the defined tests it is output through the port.

I’ll repeat that, because it’s important. A feature is tested against each test – starting with the top one – until it PASSES one of them. So the order in which the tests appear is important, and there are buttons in the dialog to move the tests up or down within that order. If it FAILS all of them, then it is (and in FME 2013-SP1 you can also rename the port to a name of your own choosing).

Anyway, in brief, if you have a workspace with multiple Testers, like so:

…you really should be using a TestFilter, like this:

Notice each Tester you need is replaced by an output port on the TestFilter. That’s much tidier.

The AttributeFilter and AttributeRangeFilter
The AttributeFilter is a simple division based on the value of a single attribute. You pick the attribute and define a set of values by which to filter it, for example:

There are no ANDs or ORs in here. It’s simply IF THIS, THEN THAT; ELSE IF THIS, THEN THAT; ELSE IF THIS… etc. Also, the only test is a simple string comparison; there are no other operators allowed.

So, although it’s actually a little easier to use than the Tester, its ease of use is countered by more limited functionality. You can also only do one test (though you could, just about, claim it is a multi-clause test).

The AttributeRangeFilter is a step up from this in that it allows ranges of values. Because of this, you’d probably only use it on numeric values:

Again it’s only a single test, on a single attribute, with no AND or OR possibilities. However, it is possible to do a few more operations (less than, less than or equal to, greater than, greater than or equal to, equals, between) than the plain AttributeFilter.

Filtering Summary
So there you have a description of each transformer for Conditional Filtering. There are many different scenarios in which you could apply them, which is most easily summed up using a table:

When you look at it like this, the TestFilter is pretty much all you need. It can do single or multiple tests, with single or multiple clauses, on single or multiple attributes. It can test strings or numeric values and it has 16 different operators to test with.

On the other hand, all that capability always comes with a little more complexity than you might need. So – for the sake of simplicity – use an AttributeFilter for testing known string values for a single attribute, and an AttributeRangeFilter for testing known numeric ranges for a single attribute.

A Tester is fine for a single test, especially when it’s just a single clause operating on a single attribute. You could just as easily use a TestFilter instead, though for an old-school FME’er like myself that just seems plain wrong!

For everything else use the TestFilter for your filtering needs.

Filtering or Mapping
Having said that….are you sure that you need to filter your data? Mapping might be better. Basically the IF THIS parts are exactly the same, the difference is in the THEN. A Filter transformer says “THEN direct the data through this port” whereas a Mapper transformer says “THEN set an attribute to this value”.

Take this example:

Here the user is, admittedly, only doing a simple string test on a single attribute, against a set of known values. The problem here is that he is then setting a value according to the filter and merging the data back again. He doesn’t need to use a Filter transformer, he needs to use a Mapper. A Mapper transformer is good here because it incorporates all those attribute setting capabilities.

The AttributeValueMapper and AttributeRangeMapper are equivalent to the AttributeFilter and AttributeRangeFilter, but instead of splitting data up, they set new values based on the condition. For example the above workspace could use the AttributeRangeMapper instead, similar to this:

That’s one transformer instead of eight. Much better.

But, like the AttributeFilter, these transformers only do single tests with no ANDs or ORs. To do multiple tests and/or use multiple attributes would need to use a set of TestFilter/AttributeCreator transformers (similar to above).

Or you would up until now!

Conditional Mapping
In FME 2013-SP1, we now have a fantastic conditional attribute setting. It makes the AttributeCreator transformer the attribute-mapping equivalent of the TestFilter, and works like this:

Here the user is creating an attribute called “NewAttribute”. However, the value it will be set to depends upon the following conditions:

This is a new dialog and we’ve even added IF, ELSE IF labels to really get the point across.

The user has now defined the conditions (each condition opens a Tester-like dialog). If the value of OldAttribute is less than or equal to 1, then NewAttribute gets the value A. If the value of OldAttribute is less than or equal to 3, then NewAttribute gets the value B, and so on.

If OldAttribute fails all of these tests, then there is no mapping (Do Nothing).

Note, in case you were wondering, I don’t need to use >1 AND <=3 for the second test. There is no clash because anything <=1 has already been mapped as value A (it is an ELSE, after all).

Now the AttributeCreator tells me there are 4 possible values (including the "Do Nothing") for this attribute:

To keep everything consistent the dialog in the TestFilter has also been updated a little in SP1 to include those “If” and “Else If” labels, and to let you rename the port – all the actually functionality is exactly the same as before though.

So, now you don’t need to filter data just to carry out multiple-test attribute mapping on it! And for attribute mapping the functionality matrix looks like this:

Like filtering, simple single-clause transformers are probably easier to use, but for the extra ability to do multiple test conditional mapping, the new AttributeCreator updates can’t be beat! We’ve even thrown a “duplicate” button onto the AttributeCreator, so you can duplicate an entire set of tests to act on a different attribute.

What’s More…
Additionally, it isn’t just the AttributeCreator that gets these new updates. All transformers that let you “Set to Attribute Value” will now let you make that a conditional test. The only exception are existing mapper transformers (as they already have conditional functionality).

For example, check out the “interval” parameter on the PointCloudThinner transformer:

And I just created a training example that used conditional mapping in the 2DGridAccumulator (of all things). The user is prompted to enter a grid size and units (via published parameters). The 2DGridAccumulator sets the grid size conditionally, depending on what units the user selected (i.e. if they entered the value in metres it is converted to feet, because those are the coordinate system units I am using).

I hope this was useful. Like I mentioned, a big chunk of this content came from the FME training course – and the rest will be going into an updated version of the training for SP1.

We run many online courses at Safe, and there are FME Certified trainers all around the world, so you’ve no excuse not to keep up to date with FME updates. And – don’t forget – you’ll also see them if you visit us on the FME World Tour.

Regards,