FME Version
Files
Please note that there is an updated version of this tutorial that is also much shorter, as well as three other data type specific beginner tutorials. Although this tutorial can be used with newer versions of FME, the terminology and screenshots are for FME 2022.0. This article will no longer be maintained.
Introduction
Congratulations on taking the next step in learning about FME Form (formerly FME Desktop). This tutorial will teach you the essentials to get your first workspace up and running. In this six-part article series, you will learn the following:
- Common FME terminology
- A brief tour of the FME Workbench interface
- How to read data into FME
- How to visualize data in FME
- How to manipulate data using FME
- How to write data out from FME
The screenshots in this series were taken using Windows, but the steps are the same using macOS or Linux.
A workspace is the name of the workflow that you design on the FME Workbench canvas, which is also the file that is saved from FME Workbench, .fmw. The workspaces throughout the tutorial series are for FME 2022.0 or newer. If you are following along with an older version of FME, please download the workspaces that correspond to the version of FME that you are using.
Workspace Downloads
Scenario
A water pump has failed and will require maintenance. For the pump to be replaced, the water utility company will have to shut off water for an entire service area. You have a list of customer addresses but are not sure which service area these customers belong to. Using a dataset of all the water service areas, you will overlay the addresses on the affected area to determine which customers belong to that area. Then after the customers have been determined, a Microsoft Excel spreadsheet will be created so someone can notify the customers.
Step-by-step Instructions
If learning FME for the first time, it is recommended to follow the simplified instructions in Tutorial: Getting Started with FME Form.
Part 1: Read Data
After opening FME Workbench, you will learn how to read in a CSV (Comma Separated Value) dataset containing a list of customer addresses for the City of Surrey, British Columbia. Then you will read an Esri Shapefile dataset containing polygon (vector) data of water service areas.
1. Before You Begin
Before you begin, ensure that you have FME Desktop installed on your computer and that it is licensed. You can download a free 30-day trial to get started.
Next, to follow along with these instructions, download the data from the Files section of this article, which is located at the top-right corner. Unzip the data to a location on your computer; we recommend your documents folder but anywhere works.
If you get stuck along the way, there are beginning and complete workspaces in the Files section that you can refer to. If you are using FME 2021 or older, please download the workspaces that correspond to your FME version, from the Workspace Downloads section.
2. Open FME Workbench
Open FME Workbench to start creating a workspace. To do this in Windows go to Start > FME Desktop 2022.0 > FME Workbench. In macOS, go to Applications > FME 2022.0 > FME Workbench.
We will be using FME Workbench for the entirety of the tutorial series. FME Workbench is where you will author all of your workspaces.
3. Start a New Workspace
After FME Workbench opens, click on the New button to create a new workspace. The empty workspace is referred to as the “canvas.” The canvas is where all of the objects in a workspace will be contained. If you have just installed FME, click on Close to exit Workbench Essentials, it can be accessed again via Help > Workbench Essentials.
4. Drag and Drop to Add Data
In your operating system’s file manager, browse to the downloaded GettingStarted/Data folder. In the Data folder, click on the LocalAddresses.csv and drag it into FME Workbench.
5. Introduction to the Add Reader Dialog
A reader is a way to “read” data into FME. Depending on the dataset you are planning on reading in, the parameters for the Add Reader dialog may change. Let’s go over the Add Reader dialog and set our parameters for the LocalAddresses.csv file we just added.
a. Format and Dataset
When dragging and dropping data into FME, the Format, and Dataset auto-populates. Depending on the format you will want to confirm that the Format is correct, as there may be different format types for the same format extension. For our example, the format should be CSV (Comma Separated Value).
FME can also read different dataset types, like folders, compressed files (.zip, .tar, etc.), and web-based. For more information see the Dataset Types Documentation.
b. Parameters
Each format has specific parameters that can be set. To check the parameters, click on the Parameter button, which will bring up the Format Parameters dialog. If the Parameters button has an exclamation point ( ! ) at the end of it; this means that there are mandatory parameters that need to be set. To learn more about your format’s specific parameters you can click on the Help button in the Parameters dialog.
If you are using FME 2020 or newer, there are no parameters to set with this dataset, as FME automatically takes the Latitude and Longitude values in the dataset, and assigns them to the x and y coordinate data type. If you have time, explore the parameters.
If you are using FME 2019 or older, open the format parameters dialog and switch the Attribute Definition to Manual. Then change the LONGITUDE and LATITUDE Type to x_coord, and y_coord, respectively.
c. Coordinate System
FME is coordinate system aware, if FME can detect the dataset’s coordinate system, the Coord. System section will say “Read from Source.” If it cannot, it will say “Unknown.” If you know the coordinate system, yet FME doesn’t detect it, you can enter it into the Coord. System box.
If you opened the parameters, click OK to return to the Add Reader dialog. Set the Coord. System to LL84 by typing it into the box. For more information about Coordinate Systems, see the documentation or the Coordinate Systems 101: The Basics blog post.
d. Workflow Options
Workflow Options dictate how the reader feature types appear on the canvas; you can think of a feature type as FME’s version of a layer in a CAD file or sheet in an Excel workbook. “Individual Feature Types” will have each layer represented as its own feature type, and “Single Merged Feature Types” will have all of the layers represented as one feature type, and will operate as a single “layer.” Single Merged is a good option if you have data that is all the same across different files or layers and would like to manipulate it all the same.
For this example, leave this set to Individual Feature Types and then click OK to finish adding the reader.
6. Feature Type
Once you click OK on the Add Reader dialog, a reader feature type will appear on the canvas. A feature type defines the schema of the data being read as well as the layers the dataset possesses. This will be the start of the workspace that everything is built upon.
For this example, CSV is the feature type name. If you want the reader feature type to reflect the name of the file, you can change this in the reader parameters when you add the data.
7. Add Another Reader
We need to add one more reader to the canvas before we can get started. To add a second reader, click on the Reader button on the top menu bar.
Enter Esri Shapefile as the Format, and then browse to the dataset using the ellipsis button. Find the water_service_area.shp file in the GettingStarted/Data folder. Clear the Coord. System parameter and then click OK. (The Coord. System will still have a value from when we added the CSV file).
Optionally, you can drag and drop the file from your file browser; just be sure to drag the water_service_area.shp file and not one of the required Shapefile sidecar files.
Part 2: View Data
With the data read in, we should view it using the Visual Preview window. Viewing the data will help determine what we will need to do in order to output our final Microsoft Excel spreadsheet with the affected customers.
1. View Source Data
The data we added in Part 1 can be viewed within FME Workbench. Since we have two datasets, we can view them at the same time. To do this, we need to run the workspace with feature caching enabled.
On the top toolbar, click the drop-down arrow next to the Run button. In the drop-down menu, ensure that Feature Caching is enabled. Now when we run the workspace, the data will be cached along the way, and we can view those caches. At any point, while building your workspace you can Run with Feature Caching to see how the data has been modified at any step. Note that Feature Caching is memory heavy so it should be turned off when running the workspace for production, or limited in use when working with rasters or large datasets.
Now click on the Run button, the workspace will run and the data will be cached for each of the reader feature types. The numbers at the end of reader feature types show how many features were read in, we refer to these numbers as Feature Counts. Feature Counts are a good indication of if your data is being modified the way you would like it, if the Feature Counts are different from what you’d expect, go back a step in your workspace and debug.
2. View the Data Cache
Now that the workspace has been run, a magnifying glass icon appears on the reader feature types; this indicates that the data has been cached and can now be viewed. First, let’s view each dataset separately.
Click on the magnifying glass icon to open up the CSV reader feature type in the Visual Preview window.
Alternatively, you can skip the running the workspace step and view the data directly through the reader feature type menu. To do this, click on the reader feature type, and then in the pop-up menu, click View Source Data. Note that this option is not available for all datasets, such as a dataset being read from the web.
3. Introduction to Visual Preview
There are three parts to viewing data using Visual Preview in FME. Each part allows the user to view their data in different ways. To use Visual Preview, ensure that the workspace has been run and then select an object to load. Note: If you clicked on the Magnifying Glass and Visual Preview opened but you don’t see any of the following views, click the icons on the left-hand side to toggle them on (d).
a. Table View
The Table view displays the dataset’s visible attributes. These attributes can be sorted or searched to explore the data. To toggle this view, click on the Table button on the left-hand side of the window (d). Note that attributes cannot be edited in Visual Preview, editing attributes will be covered in a later part.
b. Graphics View
If the dataset has a spatial component, graphic, geometry, or coordinate values (like latitude and longitude), Visual Preview will display the graphics or geometry in the Graphics View. If there are no graphics or geometry, this view will not be available. To toggle this view, click on the Graphics button on the left-hand side of the window (d). By default, background maps are turned off. For more information on background maps, see the documentation.
c. Feature Information View
The Feature Information View is where any additional information about a feature can be found, including format-specific attributes and coordinate systems. To toggle this view, click the Show/Hide Feature Information Button on the left-hand side of the window (d).
4. Explore CSV Data
With the LocalAddresses CSV open in the Visual Preview window, take a moment to explore the dataset. In the Table View, we have addresses and a latitude/longitude location. FME 2020 or newer, automatically turns the latitude and longitude into point data which can be seen in the Graphics View.
5. Explore water_service_area data
Now, view the water_service_area data in Visual Preview by clicking on the green magnifying glass on the reader feature type.
Looking at the Table View for water_service_area you can see that there is an area that has the STATUS of RED. We will use this information in the next article to filter the data based on the RED value.
6. View Both Datasets Together
When you click on the magnifying glass to view data in Visual Preview, only one dataset loads at a time. If you want to view all the data together, click and drag a box around all objects you wish to view.
Part 3: Add a Transformer
When we viewed the data in Visual Preview, we noted a RED status in the water_service_area dataset. We will use the Tester transformer to only select the service areas with the RED status.
1. Add a Transformer
When modifying data in FME, you will need to use a transformer. To add a transformer, click anywhere on the canvas and start typing the transformer name you wish to add. When you start typing, the Quick Add dialog will appear; from this dialog transformers, readers, and writers can be added by just typing their name.
Once you have found the transformer you are looking for in Quick Add, either click Enter on the keyboard or double-click it with the mouse to add it to the canvas.
To start, we will add a Tester transformer. This transformer will let us filter the data based on an attribute value, such as STATUS = RED.
2. Connect the Transformer
To use the transformer, it must be connected to the workflow. Click on the output arrow on the water_service_area reader feature type and drag to the input arrow on the Tester transformer. When they are correctly connected, a line will appear between them. Additionally, once they are connected attributes from earlier in the workflow can be accessed in the transformer.
3. Open the Transformer Parameters
The parameter button (cogwheel) on a transformer is color-coded to reflect the status of the settings.
A red cogwheel indicates that there is at least one parameter value that needs to be set before this transformer will work.
A yellow cogwheel indicates the default parameters have been automatically chosen but should be confirmed. The transformer will still work even if these parameters are not reviewed.
A cogwheel that matches the transformer color indicates that the parameters were checked or set and the transformer is ready to use.
To open the transformer parameters dialog, double-click on the transformer. Parameters for each transformer are different. To learn about parameters specific to the transformer, click on the Help button to open the documentation.
Double-click on the Tester transformer to open the parameters or single-click on the cogwheel.
4. Create a Test in the Tester
In the Tester parameters, a test clause (conditional statement) can be created to filter data. Click on the red box under Left Value to show the drop-down arrow. Click on the drop-down arrow and expand Attribute Value, then click on STATUS.
This drop-down menu shows all of the attribute values available for a specific parameter.
Next, click on the box under Right Value and type in RED. Automatically the Operator gets set to equals by default. If you are looking for a different operator, click on the box to expand the list. Click OK to accept the parameters and close the dialog. For more information on using the Tester, see the documentation. If you do not see any attributes, double-check that the transformer is connected to the CSV reader feature type, which was detailed in step 2.
Note: As of 2020.1 or newer, certain transformers are data-aware, enabling the user to select cached values from within the transformer instead of manually typing them. For this tutorial, we will be manually typing in the values. For more information, see this Expert Note.
5. Run and Inspect the Test
Since feature caching is enabled, the workspace can be run to the Tester, and the feature caches can be inspected in Visual Preview. Running your workspace often to check that the output is what you expected from a transformer is good practice.
Click on the Run button on the top menu bar. After the workspace has run, there should be a 1 next to the Passed output port on the Tester, and a 3 next to the Failed output port. Click on the green magnifying glass on the Passed output port to open it in Visual Preview.
In Visual Preview, you can confirm that there is one water_service_area with the Status of RED.
Part 4: Perform a Spatial Filter and Modify Attributes
Since we want to know which customers are affected by the failed water pump, we will need to filter the customers based on their spatial relationship to the water_service_area with a RED status. After we have determined which customers are affected, we will need to clean up the attributes before we can write the data out.
1. Add Another Transformer
Now that you know how to add a transformer, let’s add another one to determine which LocalAddress points fall into the boundary of the water_service_area that we selected with the Tester.
Add a SpatialFilter transformer to the canvas by typing SpatialFilter on the canvas. The SpatialFilter filters data based on spatial relationships with the input data. Once the transformer has been added, connect the CSV reader feature type to the Candidate input port on the SpatialFilter. Then connect the Passed output port on the Tester to the Filter input port on the SpatialFilter.
It’s ok if your connection lines are crossed, but if you want to uncross them, either move the CSV reader feature type below the water_service_area one by clicking and dragging or right-click on the Filter input port and select Move Down.
2. Open the SpatialFilter Parameters
Double-click on the SpatialFilter transformer to open the parameters. There are a lot of parameters here, but we can accept the default for most of them. The only parameter that needs to be changed in the SpatialFilter is to disable the Merge Attributes option. This option will merge all of the attributes from the points and the boundary, which we do not need. Click OK to close the SpatialFilter.
Basically, how the SpatialFilter works is every point feature that comes into the Candidate port will be tested to see if it falls within the boundary of the water_service_area, which is the Filter. If it does, it will go to the Passed output port. This is a good transformer to use if you want to select features that are within (or not within) a certain boundary. For more information on the SpatialFilter, see the documentation.
Note: Depending on your version of FME, the Spatial Predicates to Test default name may be different, but it will provide the same results.
3. Run the Workspace and Inspect the SpatialFilter
If you have time, run the workspace once more and inspect the Passed output port on the SpatialFilter by clicking the green magnifying glass.
You should have 6,722 points output and should be in roughly the same shape as when we viewed the water_service_area.
4. Clean Up Attributes with an AttributeManager
The final step before we can write out the data is to clean up the attributes. There are several different transformers you can use to manage attributes, but the best one is the AttributeManager transformer. The AttributeManager allows you to add, delete, rename, and re-order your attributes. For more information on the AttributeManager, see the documentation.
Add an AttributeManager transformer to the canvas and then connect it to the Passed output port of the SpatialFilter. Open the parameters. You can see all of the attributes that the LocalAddresses started with when we viewed the dataset. There is also an additional parameter called _predicate, which was created in the SpatialFilter. We don’t need _predicate, so we can remove it. Click on the line with _predicate, then click the negative ( - ) button at the bottom. Alternatively, you can remove the attributes that are generated by certain transformers by deleting the value within that transformer. For this example, you would clear _predicate from the SpatialFilter Output Attribute Name parameter.
After removing _predicate, you’ll notice that the Input Attribute is still there, but the Output Attribute is blank; this is so you can reinstate it if you accidentally remove the wrong attribute or change your mind. If you were to change your mind, you can click on the drop-down box under Action and change it from Removed to Do Nothing.
While we’re here, let’s also rename the LOCALITY attribute to CITY. Click on the LOCALITY box under Output Attribute, and then type CITY. You’ll notice that the Action has changed to Rename. Click OK to close the AttributeManager.
Part 5: Add a Writer
Now that we have filtered out which customers are within the affected water_service_area, as well as cleaned up the attributes, we are now ready to write the data out to a Microsoft Excel spreadsheet.
1. Add a Writer
Now that the points have been filtered to include only the ones that fall within the water_service_area boundary that has a RED status, and we’ve cleaned up the attributes, we can now write out the data. The utility company has requested the list of addresses be in a Microsoft Excel spreadsheet.
To add a writer to the canvas, click on the Writer button on the top menu bar or by going to Writers > Add Writer.
2. Introduction to the Add Writer Dialog
A writer is a way to “write” data out of FME. Depending on the dataset you are planning on writing out, the parameters for the Add writer dialog may change. Let’s go over the Add Writer dialog, which is similar to the Add Reader dialog, and set our parameters for the Microsoft Excel spreadsheet.
a. Formats and Dataset
In the Add Writer dialog, select the Format in which you want your output data to be. Next, choose a location to save the dataset, depending on the format you might also input a filename.
Set the Format to Microsoft Excel and then browse to a location to save the file by clicking on the ellipsis next to Dataset. After selecting a location, input AddressesToNotify.xlsx as the filename.
b. Parameters
Each format has specific parameters that can be set. To check the parameters, click on the Parameters button, which will bring up the Format Parameters dialog. If the Parameters button has an exclamation point ( ! ) at the end of it, this means that there are mandatory parameters that need to be set. To learn more about the format’s specific parameters you can click on the Help button in the Parameters window.
We can accept the default parameters for the Microsoft Excel writer.
c. Coordinate System
The writer will keep the coordinate system that was in the original dataset or set during the workspace. If you would like the output dataset to be in a different coordinate system, you can enter the desired coordinate system in the Coord. System box to reproject on the fly.
We will leave it blank for this tutorial.
d. Add Feature Type(s)
The Add Feature Type(s) section determines how the schema will be written out. To have the output attributes reflect what was done in the workspace, select Automatic. To manually select which attributes are written out, choose Manual. Finally, to use the same attributes as the original reader, select Copy From Reader. To learn more about adding feature types, click on Help > Add Writer in the Add Writer dialog. Note that the terminology in this section will change based on the format.
Select Automatic for the Sheet Definition; this will ensure that our writer automatically reflects the attribute changes that we did in the AttributeManager. Click OK to add the writer.
3. Feature Type Dialog
When you set the Definition to Automatic in the Add Writer dialog, the details about the Feature Type name will need to be defined. After clicking the OK button in the Add Writer dialog, a Feature Type dialog appears. In this dialog, you will need to set the Feature Type Name; this will be the name of the layer or worksheet.
Set the Sheet Name to Addresses and click OK. There are other parameters that could be set for the Microsoft Excel writer, but they can be left as default for this tutorial.
4. Connect the Writer Feature Type
Once the Addresses writer feature type has been added to the canvas, it needs to be connected to the workflow. Click on the output port of the AttributeManager transformer drag to connect to the writer feature type input port.
Part 6: Write Data
The final step before we can hand off the affected customers list is to run the workspace to create the actual file.
1. Run the Workspace
To run a workspace, click on the Run button. Before clicking, if you mouse over the button, a green highlight will appear around all of the objects on the canvas that will be executed when the Run button is clicked. Which objects will be run is useful to know if you are testing a workspace and have made changes recently.
Because we’ve been running this workspace throughout the tutorial, only the AttributeManager and Microsoft Excel writer will run because the data is cached up to the SpatialFilter. Click on the Run button to finish running the workspace and write the data out.
Note: It is a good idea to rerun the entire workspace if you have been using feature caching. This ensures that all the data is up to date and any changes that have been made to the original dataset are reflected. To do this, click on the drop-down arrow next to Run and select Rerun Entire Workspace.
2. Confirm Workspace was Successful
After the workspace has been run, there are two main ways to tell if it was successful. The first way is to see the Feature Count numbers flow through the workflow; there should be at least one feature moving from each object.
The second way is to look for the “Translation was SUCCESSFUL” message at the bottom of the FME Workbench in the Translation Log window. This message will also note how many warnings there were and how many features were written out. The features that are written out should directly reflect the feature count number going into the writer.
To access the Translation Log go to View > Windows > Translation Log. If you have saved your workspace, a saved version of the Translation Log (which has the same name as the workspace) can be accessed in the same folder as your workspace. This allows you to view the Translation Log in a text editor, like Notepad ++ if desired.
3. Inspect the Output
Once it has been confirmed that the “Translation was SUCCESSFUL” it is a good idea to confirm the output by either opening the output file in that format’s native software or using Visual Preview within FME.
To view the final output data from this tutorial series, click on the Addresses writer feature type to show the popup menu. On the popup menu, click on the View Written Data button.
This will open the data in the Visual Preview Window.
Alternativly, to view the data in its native application, click on the Open Containing Folder icon, then double-click on the file to open it. The native application must be installed, in this case, Microsoft Excel.
4. Save the Workspace
Before closing FME Workbench or starting a new workspace, be sure to save the workspace by clicking the floppy disk icon or File > Save.
Congratulations, you have created your first FME workspace. To learn more about FME, check out our Knowledge Base or free training. If you get stuck along the way, post a question to our Community or contact our Support.
Saving a Template
When saving an FME Workspace, the data is referenced by a direct path. If the data is renamed or moved, it will not be available in the workspace. To prevent this, or if you are sharing the workspace with someone else, the workspace can be saved as a template, which will include the data. To do this, go to File > Save as Template.
Then ensure that the data you are using in the workspace is listed. If you want to include more data, you can click the ellipsis next to Add More Files:
Troubleshooting
Occasionally, there may be reasons why a workspace fails, and you will need to troubleshoot.
a. Translation Log Window
In order to troubleshoot successfully, the Translation Log needs to be open. If it isn’t already open, go to Tools > Windows > Translation Log. The log will auto-populate with details about the last time the workspace was run. Looking at the top of the Translation Log, there are two key buttons, Errors, and Warnings. If your workspace failed, these buttons would have a number next to them. You can click on these buttons to filter the log and jump right to the error. Once you find the error, read the details and resolve the issue. Then re-run the workspace.
This workspace failed because the filename is incorrect. The user would have to add new data or fix the filename before the workspace will run successfully.
Sometimes there are also warnings. Typically these can be ignored as they are just warnings about certain FME specific settings. In this example, the warning is that Feature Caching is turned on, which may slow performance. Even though these warnings can be ignored, it is always a good idea to investigate them.
b. Rejected Feature Handling
Another reason a workspace might fail is that a single feature causes a transformer with a <Rejected> port to terminate.
If you want to ignore those features, this function can be turned off. To turn this function off, right-click on <Rejected>, then go to Workspace: Rejected Feature Handling > Continue. This will turn this feature off for the entire workspace. If you only want this feature disabled for this port, just connect a Logger transformer.
Data Attribution
The data used here originates from data made available by the City of Surrey, British Columbia. It contains information licensed under the Open Government License - Surrey.
Comments
0 comments
Please sign in to leave a comment.