Unexpected Input – The Toll Booth Model

Hi all,
It’s very common for me to see new users confused by this message from FME:

“During translation, some features were read that did not match a reader feature type in the workspace.  This can happen if the reader dataset is changed, or a reader feature type removed or renamed.”

I can’t believe I haven’t covered this before, so this post makes up for lost time by explaining what the message means, how it comes about, and how to resolve it.

THIS IS NOT AN ERROR!
OK, first things first: this situation might result from an error by the user (or their data provider), but it is not an FME error message. The translation hasn’t failed and FME hasn’t crashed. It is simply a warning. A warning from a function called the Unexpected Input Remover.

So what is the Unexpected Input Remover? Well, I try to use analogies for many FME functions, and in this case I’m going to compare this function to a highway toll booth or border crossing.

A Toll Booth
Let’s take a look at how a toll booth might operate. Here we have a toll booth where the operator has thoughtfully provided separate lanes for cars and trucks.

So everything works smoothly provided only cars and trucks turn up. But what happens if a bus tried to get through? There is no lane for buses, so the bus would be turned away. Its arrival would be “unexpected”.

In FME
The same mechanism applies in FME, but using Feature Types (aka layers, tables, classes, etc) instead of vehicle types. Take this workspace as an example:

Consider it as a toll booth for spatial data. The objects represent the layers in the data. There are lanes for layers of data called “Rail”, “Rivers”, and “Roads”. Like the real-world toll booth, everything works smoothly provided only data in a layer called Rail, Rivers, or Roads turns up.

But say the underlying data changed, and a new layer called “Buildings” was added. There is no lane for Buildings to get through. Its arrival is unexpected, and so it is removed from the translation:

And that’s pretty much what the Unexpected Input Remover is all about. It takes data that is from an undefined layer, and drops it from the translation.

Of course there are various reasons why a layer (Feature Type) is undefined. Maybe the underlying data changed and a new layer was added without you being aware of it. Or maybe you decided to read a different dataset and it has a different set of layers. It might even be a simple typo in the layer name; for example “Rial” instead of “Rail” or “roads” instead of “Roads” (the Unexpected Input Remover is case sensitive).

So, the key question might be, what can you do about this situation?

Adding a Lane
OK, so if we really do want buses to be able to go through the toll booth, the obvious solution is to add a lane for them, like so:

To create this the operator would probably look at a typical bus, measure it up, and use that information to construct the new bus lane. Simple.

In FME
In FME the equivalent is called Importing Feature Types. It’s activated on the menubar using Readers > Import Feature Types.

Say we want to include Buildings (from the previous example). Like the toll booth, what we need to do is look up a typical Buildings feature, measure it up and use that information to construct the new workspace Feature Type.

So first we choose a dataset that includes a Buildings layer:

This can be the dataset we are trying to read, or any dataset of any format. All we’re doing is finding a feature to scan for its “measurements”. Then we tell FME which feature type we are interested in:

Once that is done, FME scans the Buildings Feature Type to find what its “schema” (aka Data Model) is. Then it creates a new lane in the workspace to handle data of that type. Simple.

The only thing to do now would be to repeat the process using Writers > Import Feature Types to create a Buildings definition for the translation output.

Merging Traffic
Now let’s assume that someone turns up at the toll booth on a motorbike. As an unexpected vehicle it would usually be turned away. The operator doesn’t want this, but on the other hand doesn’t want to construct a whole new lane just for bikes. So this is what he does:

You can see that the lane for cars has now been configured to “Others”. Buses and trucks will still go through their own lane, but now both cars and bikes can pass through that redefined lane.

In fact, what makes this solution particularly appealing, is that by naming the lane “Others” any other vehicle can pass through; whether it is a van, SUV, taxi, fire engine, or anything. That solves the problem for whatever type of vehicle turns up.

In FME
In FME this technique is known as a Merge Filter. As you might guess, it involves taking an existing Feature Type and changing it to allow any type of data to pass.

For example, the dataset we’ve been working on now has a new layer of data called “Schools”. Rather than add it separately, as a type of building I’ll let it pass through the Buildings feature type.

To do this I first click on the parameters button for Buildings:

This opens the properties dialog. There I can click the “Merge Feature Type” option and enter into the Merge Filter field a definition of the new layers I would like to be able to pass. For the sake of simplicity I will enter an asterisk (*) character to allow any unknown data type to pass:

So an asterisk has the same effect as an “Other” lane on a toll booth. It handles traffic that isn’t explicitly defined.

Now when the workspace is run, there is no unexpected input and never will be: we’ve set up the workspace to expect the unexpected. The feature type name changes from “Buildings” to “” to reflect this fact (in the same way that the toll booth changes its name from “Cars” to “Others”):

The only potential problem is that all features now get lumped into the same lane. But if you do wish to process them separately they still possess an fme_feature_type format attribute that records their original type, and so can be filtered by this: it’s not like a motorbike suddenly becomes a car just because it’s passed through the car lane in a toll booth!

Points to Consider
Of course there are a number of other points we can consider.

Deliberate Blocks
Many users see this message as an error; but you should consider that the situation leading to it might actually be deliberate.

For example, with the toll booth the lane for trucks might be purposely left out because trucks aren’t permitted to travel this route. In the same way, the “Rivers” feature type might be purposely left out (or deleted) because that data isn’t required in the translation. The same message appears – but it isn’t an error because it’s a deliberate act.

Merge Filter Variations
Just like any toll booth lane can be varied (for example changing “Cars” to “Cars+Bikes” to allow bikes but exclude SUVs) so a merge filter can be set up with more precision than just “any other data”. That way you can get quite creative about what types of data you permit to pass and where.

Also note that – because they have their own feature type – “Roads” and “Rail” will always pass through there and not in the lane. It’s the same thing as saying that trucks always pass through the “Trucks” lane and never through “Other Vehicles”.

So there you are; that’s the Unexpected Input Remover. I hope this post helps to make the message seem less scary, and provide options as to what you might do when the message does appear.

It should also encourage use of data inspection before a translation, where you can ensure your source data feature types are what you thought. That way you avoid having to re-run translations because the workspace wasn’t expecting the data supplied to it.