Reprojection and Spatial Overlay with FME

Introduction

Data comes in all kinds of projections or sometimes it is even missing a projection. This is where the Reprojector transformer comes in. It transforms data from one coordinate system to another. When your datasets are in the same coordinate system they can be overlaid on top of another. In this exercise, we will take two datasets, each with a different coordinate system and reproject them into the same one (UTM83-10). Then write the output to Tableau.

Step-by-Step Instructions

1. Set up CSV reader to read the business license points

Click Reader > Add Reader then enter the following:

Open up the parameters, set Feature Type Name(s) to From File Name(s) to create a feature type in the workspace that gets its name from the file name instead of calling it CSV.

If you are using FME 2020+, verify the Latitude and Longitude attributes are automatically set to the x_coordinate and y_coordinate data type in the Attribute Definition. For 2019.2 and older, set Attribute Definition to Manual then set the Latitude to y_coordinate and the Longitude to x_coordinate. Click ok. Finally set the Coord. System to LL84. Click ok to add the reader to the workspace.

2. Add AutoCAD Reader to read the Land Parcels dataset

Add a Reader then start typing DWG into the format and select the following:

Under parameters, make sure Group Entities By is set to Layer Name, click ok and then again to add the reader. The Select Feature Types dialog will appear, select only the layer called ParcelLines.

3. Inspect the data

Once the reader feature types have been added to the canvas, run the translation and inspect the feature cache(s) to inspect the data in the Visual Preview window. 

When you open the Visual Preview, if the Background Map is disabled from the FME Options... window, your data will appear as two small dots on the screen. This is because the two datasets are in different coordinate systems. You can confirm this by right-clicking on each of the two datasets in the Visual Preview graphics view, clicking Zoom to Extent, then clicking on a feature. Then in the Feature Information window, under Coordinate System you can see which coordinate system the datasets are in. If you have the Background Map enabled, go to the FME Options... to disable it to view this phenomena, as using Background Maps will reproject the data to the Background Map coordinate system.

4. Connecting a Reprojector transformer

Let's fix this problem. Connect a Reprojector transformer to the business_licenses reader feature type and set the destination coordinate system to UTM83-10. Click ok and run the translation. Inspecting the data both datasets now appear in the same coordinate space.

Reprojector transformer to reproject from LL84 to UTM83-10

5. Add a GeometryFilter to clean up data

As you recall from previous exercises some of our data in business_licenses is missing latitude and longitude data. We need to filter out all of the data missing those values so we won't get errors later on in our translation. Add a GeometryFilter after the Reprojector. Under "Geometries Type to Filter:" Click Point. Click Ok to add the transformer.

6. Determine which land parcel each business falls inside

We now want to figure out which land parcel each business falls inside. To to this, we can use the PointOnAreaOverlayer. Add the PointOnAreaOverlayer transformer to the workspace. Connect the GeometryFilter Points Port to the Points input port and the AutoCAD reader to the Area input port. Connect inspectors to each output port.

7. Handling rejected features

Looks like all the land parcel data was sent out to the rejected port. Double click on the rejected port to open up the Data Inspector. Take a look at the Feature Information window and look for the fme_rejection_code attribute to see why the features were rejected.

The code says: INVALID_POLYGON_GEOMETRY_TYPE. It looks like we have lines from the ParcelLines dataset and not polygons.

In the Data Inspector, fme_rejection_code(string) = INVALID_POINT_GEOMETRY_VERTICES

8. Create polygons from lines

Back in the workspace, connect an AreaBuilder transformer to the AutoCAD reader. This transformer takes connected lines and turns them into solid polygons. Open up the parameters and under Snapping Pre-Processing Parameters, set Snapping Type to End Point Snapping and the Snapping Tolerance to 0.1 . This will snap only the end points of lines together to create the polygon.

Set Snapping Pre-Processing Parameters in the AreaBuilder transformer

9. Rerun and inspect the workspace

Rerun the workspace and inspect the output from the PointOnAreaOverlayer. It should look much better without any rejection codes.

10. Determine which businesses only overlap with the land parcels

Add a TestFilter to the Points output port on the PointOnAreaOverlayer. Open up the parameters and set the test condition to find where _overlaps >= 1. Click Ok to add the transformer.

Set the Test Conditions in the TestFilter to _overlaps >= 1

11. Create an ID attribute to assist with connections in Tableau

We want to be able to create a visualization within Tableau that links the two datasets together so that we can click on a parcel and see only the business license data for the businesses that fall within that area. We can do this by creating an ID attribute on the areas and adding that attribute to each of the points inside the area.

Add a UUIDGenerator transformer between the AreaBuilder and the PointOnAreaOverlayer. Open the parameters and set the New UUID Name to ParcelID. The PointOnAreaOverlayer will automatically add that ID attribute onto each point feature that exits the transformer.

12. Write both features out to .hyper format

Finally, we need to write everything out to Tableau. We will need two writers, one for each dataset.

Click Writers >Add Writer

Under Add Feature Types > Table Definition select: Automatic and click ok. Set the Table Name to BusinessLicenseData and set the Geometry to hyper_point, click ok. Connect it to the output of the TestFilter (@Value(_overlaps) >= 1). Open the writer feature type parameters and clean up the attributes to remove anything we don't need and adjust the column types as needed (we don't need _overlaps, Longitude, or Latitude anymore)

To create an output for the Parcels data, right click on the workspace and click "Add Feature Type". Then set the Table Name to LandParcels and set the Geometry to hyper_polygon, click ok. Connect it to the Areas output port on the PointOnAreaOverlayer transformer.

13. Run the workspace to create the Hyper files

Ensure that you have removed all of the remaining inspectors, and then run the workspace to create the Hyper files. Check in Windows Explorer to confirm the files have been created. Open them up in Tableau to finish.

Final output in Tableau. Create a Business License Status map or view which businesses belong in each parcel and the status of their business license.