How to Map Point Data without Latitude and Longitude: Leveraging Linear Referencing Principles

Introduction

In the previous tutorial, we introduced the AdobeGeospatialPDFReader and showed how to configure it to read and extract information from a PDF. By the end of the tutorial, you were able to generate an output feature containing all the relevant extracted data. Now, in this follow-up tutorial, we will delve deeper into the GIS side of this project and explore methods for mapping all the Average Daily Traffic (ADT) collection points, even without geographical coordinates. To accomplish this, we will leverage the fundamentals of the Linear Referencing System, which involves measuring positions and distances along linear features. So, without further ado, let's dive in!

Scenario

You have completed the initial phase of an FME Workspace, which involves reading a Traffic Speed Report in PDF format, extracting the desired data, and subsequently joining it to the nearest intersection point. However, the actual collection point may not be directly at the intersection. Instead, it often happens at an adjacent mid-block location, depending on the direction of travel being recorded. 

For instance, when gathering vehicle counts for the Northbound direction, the collection point will be positioned north of the intersection. Conversely, for the Southbound direction, the collection point will be situated south of the intersection. While the distance between the intersection and the collection point can vary, your GIS team has decided to maintain consistency by using an arbitrary distance of 50 feet. As a result, you have chosen to leverage your expertise in the Linear Referencing System to explore an alternative mapping method and further enhance the previous section of this workspace.

Step-by-step Instructions  

Review Linear Referencing Fundamentals 

Let's start by reviewing the fundamentals of Linear Referencing:

Linear referencing is a technique employed in geographic information systems (GIS) and transportation engineering to locate and analyze features along a linear element, such as a road, pipeline, or river. It entails measuring positions and distances along the linear feature, typically using a linear reference system (LRS). In linear referencing, distances are measured from a known starting point, often called a reference point or zero milepost. These distances are typically expressed in linear units, such as meters or miles. By associating attributes or events with specific positions or distances along the linear feature, linear referencing enables analysis and management of information relative to the feature.

Linear referencing finds applications in various domains, including mapping, route analysis, asset management, and transportation planning. For instance, it can help determine the location of road signs or utility poles along a road, measure the distance between two points on a pipeline, or analyze the distribution of accidents along a highway segment.

For more information on Linear Referencing, please refer to the blog article Linear Referencing 101: Managing Infrastructure Data

Now, let's explore how this applies to our ADT mapping project. We already have the reference point: our intersection point. The distance along the linear feature will be a fixed distance of 50 feet. However, we still don't know which specific street segment contains our collection point. So, our first task is to select all street segments that intersect the intersection point before identifying the segment of interest.

1. Add an Esri Shapefile Reader to Read in the Streets Shapefile

To start, open the workspace you used in the previous tutorial. Alternatively, you can also open the starting workspace (part2_startingworkspace) provided under Workspace Downloads 

Add an Esri Shapefile Reader to the canvas and place it in the same bookmark as the Street_Intersections shapefile.

For Dataset, browse to the demo folder and select Streets.shp. Make sure Workflow Options remains set to the default (Individual Feature Types), then click OK and Run. Confirm that 35,699 street segments were read in. 

2. Add a PointOnLineOverlayer for Spatial Join

Next, we will use the PointOnLineOverlayer to perform a spatial join between the intersections and the street segments shapefile. Double-click on the canvas and add a PointOnLineOverlayer transformer. Connect the FeatureJoiner Joined output port to the Point input port, and the Streets reader feature type to the Line input port. 

Open the parameters and fill in as follows: 

• Point Tolerance: 10
• Expand the Attribute Accumulation dropdown and check the Merge Attributes box
  • Accumulation Mode: Merge Incoming
  • Conflict Resolution: Use Original

Click OK, then Run To This. Double-check that the Line output port has 35,699 features and the Point output port has only 1 feature. 

Browse to the Table under Data Preview, drag the bottom scroll bar to the right end, and check if a new attribute named _overlaps is created. Double-click on the attribute name to sort values in descending order. You can see that this is a 4-way intersection, with four street segments intersecting at the intersection point. 

3. Add a Tester to Filter Out Irrelevant Street Segments 

Now, we will have to filter out all other street segments that do not overlap our intersection point. You can do this easily with the Tester transformer. Add a Tester to the canvas and connect the Line port from the PointOnLineOverlayer to it. Fill in the following for the parameters:

• Left Value: _overlaps
• Operator: >
• Right Value: 0 

Click OK, then click Run To This and confirm that only 4 features passed through the Tester’s output port. Before determining the main linear feature along which to move the intersection point, ensure that the base vertex (the first or starting vertex) of all street segments overlaps the intersection point. In other words, all street segments should originate from the same intersection point. To verify this, extract the intersection coordinates and the coordinates of all vertices that make up each street segment. Next, use a Tester in your FME Workspace to filter out segments whose base vertex does not overlap the intersection.

4. Add Two CoordinateExtractors to Extract Intersection Point and Street Segments Coordinates

First, add a CoordinateExtractor between the FeatureJoiner and the PointOnLineOverlayer. Click the connection line between these two transformers. When the connection line is highlighted in blue, type in Coordinate Extractor and press Enter. After that, add another CoordinateExtractor and connect it to the Tester’s Passed port. 

For the first CoordinateExtractor, fill in the following parameters: 

• Mode: Specify Coordinate
• Output Attribute Names:
  • X: _x
  • Y: _y

This will output only one coordinate pair because our input is a single intersection point. You can double-check this by selecting the feature in the preview table and checking the Feature Information window under Attributes. Note the coordinate values (_x and _y) so we can use them in the next step. 

For the CoordinateExtractor_2, fill in the following Parameters: 

• Mode: All Coordinates
• Coordinate List Name: _segmentvertex

This will output several pairs of list attributes nested in every street segment feature. These attributes indicate the coordinates of each vertex forming the segment. 

Now, if you pay attention to the last two list attributes from this screenshot (_segmentvertex{4}.x and _segmentvertex{4}.y), do they look similar to the _x and _y list attributes extracted from the intersection point above? That means this segment's last vertex overlaps the intersection point. In other words, the segment does not originate from the same intersection. However, we want the base vertex (_segmentvertex{0}) of every segment to have the same coordinates as our intersection point. Let’s see how many segments have the same issue and find a way to fix it.

5. Add a Tester to Filter Out Segments that don't Share the Same Base Vertex with the Intersection Point

Add another Tester to the canvas and connect it to the Output port from the CoordinateExtractor_2. Set the parameters as follows: 

To explain, we are testing if the base vertex (_segmentvertex{0}) of the street segments has the same coordinates as the intersection point. Click OK, then Run To This. You can see two Passed and two Failed features, meaning two of the four street segments do not have their base vertices overlapping the intersection point. 

6. Add an Orientor to Flip the Vertex Order 

Now, let’s use a transformer named Orientor to flip the vertex order of the two segments at the Failed port above, making their last vertices become the base vertices. Connect the Failed port from the Tester_2 to the newly added Orientor. Fill in the following parameters:

• Orientor Type: Reverse
• Flip Appearance: Yes

Click Ok then Run To This. At this stage, you may not observe any immediate changes in terms of attributes or geometry. To verify whether the vertex order has flipped, extract the coordinates again. Alternatively, you can also select one of the segments from the preview table, then take a look at the Geometry dropdown at the bottom of the Feature Information window to quickly check the first vertex coordinates. Are the _segmentvertex(4) coordinates the same as coordinates 0? That means we successfully flipped the vertex order of that street segment.

If the Orientor tool is |a bit prplexing, refer to the following illustration for a clearer understanding.

7. Add Another CoordinateExtractor to Extract the Flipped Vertex Coordinates

Add another CoordinateExtractor to the canvas and connect it with the Orientor. 

When prompted, use the following for its parameters: 

• Mode: All Coordinates
• Default Z Value: (leave blank)
• Coordinates List Name: _flippedvertex.

It is a good time to save the workspace again, so click Save, then click Run. Now, take a look at the Feature Information windows and check if the first pair of list attributes (_flippedvertex{0}) of either output feature is as follows:

Make sure these numbers match the intersection-point coordinates you noted above before moving on to the next step. If not, double-check the connection between Tester_2 and the Orientor, and confirm that Tester_2's Failed port is connected to the Orientor, not the Passed port. 

Save the workspace again and document every step with annotations and bookmarks. Your workspace should be similar to the following screenshot. 

8. Add a HorizontalAngleCalculator to Calculate the Offset Angles 

Up to this point, we have successfully identified all street segments where the base vertex is at the intersection point. This ensures consistent angle calculations and helps determine the correct offset angle. Now, let's proceed to calculate the offset angle to determine the direction in which the intersection point will be moved.

To do this, add a HorizontalAngleCalculator custom transformer to the canvas. Connect the Tester_2 Passed port and the CoordinateExtractor_3 Output port to the HorizontalAngleCalculator. The HorizontalAngleCalculator transformer is distinguished by its green color, indicating that it is a custom transformer. In essence, a custom transformer combines standard transformers into a single transformer and can be customized in a separate tab on the canvas.

For more information on Custom Transformers, see this documentation: About Custom Transformers.

When prompted, fill in the parameters as follows: 

• Calculation Angle from: Feature
• Point 1 X-Coord: _segmentvertex{0}.x
• Point 1 Y-Coord: _segmentvertex{0}.y
• Point 2 X-Coord: _flippedvertex{0}.x
• Point 2 Y-Coord: _flippedvertex{0}.y
• Angle Unit Measurement: Degree

Click OK, then Run To This, and confirm that the output has a total of 4 features.

Next, remove the irrelevant attributes using another AttributeKeeper. Add an AttributeKeeper to the canvas and connect it with the HorizontalAngleCalculator. For the Attributes to Keep parameter, click the ellipsis button and select the following: _angle, _azimuth, ASTREETDIR, BSTREETDIR, BSTREETNAME, CollectionPoint, FULLNAME, and TravelDirection. You can leave the List to keep parameters blank.

Click OK, then Run To This, and double-check that the output includes only the attributes listed above. 

9. Add a TestFilter to Filter Out Segments per Cardinal Direction

Now that we have calculated the angles for all the segments, it's time to define the range of angles for each cardinal direction: North, South, East, and West so we can tell which segment is going towards which direction. Specifically, segment angles between 45 and 135 degrees will be assigned as North, 135 to 225 degrees as West, 225 to 315 degrees as South, and the rest as East. Note that the angle direction is counterclockwise starting from the X-axis to match the Polar Angle output from the HorizontalAngleCalculator. 

We will also consider the CollectionPoint attribute in this process. To accomplish this, add another TestFilter transformer to the canvas and configure its parameters using the instructions provided below. 

First, double-click on the first row of the If clause, under the Test column, to open up the Test Conditions dialog box, and fill in the following: 

• Left Value: _angle
• Operator: In Range
• Right Value: Click the ellipsis button and specify the range as follows:
  • Lower Limit: Greater than value
  • Value: 135
  • Upper Limit: Less than value
  • Value: 225 
• Mode: Numeric
• Logic (second row): AND
• Left Value (second row): CollectionPoint
• Operator (second row): Begins With
• Right Value (second row): W
• Mode (second row): Case Insensitive
• Output port: West

Click OK, then double-click on the second row to configure the next Else If clause for the West direction: 

• _angle In Range [225,315] Numeric. (Note that the square brackets indicate the inclusion of the limit values. Greater than or equal to, and less than or equal to) 
• AND CollectionPoint Begins with S Case Insensitive. 
• Output Port: South

Next, configure the second Else If clause (third row) for the East direction: 

• _angle In Range [45,135] Numeric. (Note that the square brackets indicate an inclusion of the limit values. Greater than or equal to, and less than or equal to) 
• AND CollectionPoint Begins with N Case Insensitive. 
• Output Port: North

Similarly, the last Else If clause should be configured for the North direction:

• _angle In Range [315,360] Numeric. (Note that the square brackets indicate an inclusion of the limit values. Greater than or equal to, and less than or equal to) 
• AND CollectionPoint Begins with E Case Insensitive. 
• Output Port: East

Click OK and use the following screenshot to double-check all the parameters for TestFilter_2:

Save the workspace, then click Run. Confirm that the North port has one output feature and the Graphics window is showing only the North segment of N 1st Street. Does this segment correspond to the CollectionPoint and TravelDirection we extracted from the PDF? (N of intersection, Northbound). 

10. Add an Extra TestFilter to Differentiate Segments with Non-cardinal Direction

The TestFilter we previously set up is effective at filtering most street segments with cardinal directions. However, it may fail to accurately identify segments with non-cardinal directions. For instance, if we examine a different location from another PDF file, the previously configured TestFilter may become slightly confused and produce two segments through the South port. This happens when both azimuths meet the test condition, which can lead to an incorrect offset angle calculation. To address this issue, we will create an additional TestFilter to capture such instances and ensure accurate filtering.

First, add a stomer transformer named FeatureCounter to count the number of segments that pass through TestFilter_2. Connect all four ports (North, South, East, West) of the TestFilter_2 to the new FeatureCounter. Set the Attribute Name to something more intuitive (_segmentcount)

Once done, click OK to confirm the configuration.

Next, let's add another Tester transformer to filter out all the non-cardinal segments. Our objective is to verify if only one segment passed through the TestFilter_2. To achieve this, we will condition the Count Attribute created from the FeatureCounter above (_segmentcount). If the count is 1, indicating that our TestFilter_2 succeeded, the segments will be directed to the Passed port. If the count is greater than 1, indicating that multiple segments passed, they will all be output to the Failed port and then directed to an additional TestFilter. Now, let's configure the parameters for Tester_3 as follows:

• Left Value: _segmentcount
• Operator: =
• Right Value: 1

Lastly, add another TestFilter and connect it to the Failed Port of Tester_3. Open its Parameters window to configure the Test Conditions. 

Click on the first If Statement to open the Test Conditions dialog box. The first Test Clause will be configured as follows: 

• Comparison Mode: Case Insensitive
• Output Port: StreetA_NS

The purpose of this Test Clause is to compare the street direction attributes (ASTREETDIR and BSTREETDIR) from the Intersection shapefile with the CollectionPoint attribute extracted from the PDF. Specifically, we are looking for segments whose direction aligns with the collection point. For instance, in the given location, Street A has an East-West Direction (ASTREETDIR), but the CollectionPoint is located south of the intersection. As a result, Street A will be eliminated because of the direction mismatch. The additional clause (FULL NAME Contains ASTREETNAM) ensures that the filtered segment has a matching name with the one recorded in the intersection shapefile. Consequently, no feature will be written to the first output port in this case.

Click on the second row of the Port Definitions to configure the Else If statement, the second Test Clause will be slightly different from the first one, which will test the direction and name of Street B instead of Street A:

• Comparison Mode: Case Insensitive
• Output Port: StreetB_NS

The result from this test clause is indeed a match, and one feature will be written out to this port. Because the CollectionPoint is South of the intersection and Street B Direction is North-South, the Street B name from the Intersection feature and the segment name from the Street Segments feature are also a match. 

For the opposite direction, East-West, continue to populate the next two Else If statements as follows:

Second Else If statement:

• Comparison Mode: Case Insensitive
• Output Port: StreetA_EW

Last Else If statement:

• Comparison Mode: Case Insensitive
• Output Port: StreetB_EW

Use the following screenshot to double-check your parameters: 

In case you want to check if our extra filters are working correctly, use the SENTER RD S OF CAPITOL EXPY SN.PDF as input for the Adobe Geospatial PDF Reader, then run the entire workspace. Only one feature should output through TestFilter_3. Now we are ready for the next step: merging the selected segment with the intersection point. 

11. Add a FeatureMerger to Merge the Filtered Segment with the Intersection Point

Before offsetting the intersection point, we need to merge it with the specific segment output from the previous step. Let's add another FeatureMerger transformer to the canvas. Connect its Requestor port to the FeatureJoiner in bookmark 6 (Summarize ADT and Join attributes to Intersection point), which you can find by scrolling to the left of the canvas. For the Supplier port, connect all four cardinal-direction ports from TestFilter_3, as well as the Passed port from Tester_3. You don't need to worry about connecting multiple ports to the Supplier, since only one output feature will pass through all the filters we set up earlier.

Open up the parameters and fill in the following: 

• Supplier First: No
• Join On
  • Requestor: 1
  • Supplier: 1
  • Comparison Mode: Automatic

This configuration is similar to what we did in the previous demo, performing a table join even when there is no common key attribute. We want to attach all the attributes from the street segment to our intersection, especially the _azimuth attribute calculated above. 

Confirm that the intersection point is output at the Merged port. We are now ready to offset the intersection point.

12. Add an Offsetter to offset the intersection point

Connect the FeatureMerger_5’s Merged Port to the newly added Offsetter, then click on the cogwheel icon to configure its parameters as follows: 

• Mode: Polar Coordinate
• Polar Coordinate:
  • Distance: 50 (Since we are using the NAD83 California zone III projection, FME will automatically recognize feet as the distance unit. You can confirm the Coordinate System under the Geometry dropdown in the Feature Information window.)

• Polar Angle: _angle. (This _angle attribute is created by the HorizontalAngleCalculator, along with the _azimuth attribute. Note that its direction is counterclockwise, starting from the X-axis, which aligns with the Polar Angle used for the Offsetter.)

Below is an illustration of the entire offsetting procedure. 

Click OK, then click Run Entire Workspace and confirm that the Offsetter output is 50 feet away from the intersection point, towards the North direction. Next, let’s have a final attribute cleanup before outputting the result to Esri’s File Geodatabase.  

13. Add an AttributeManager to Clean up Attributes

After adding another AttributeManager to the canvas, add it to the Offseter and configure the parameters as follows: 

• FACILITYID: Remove
• FULLNAME: Remove
• ASTREETDIR: Remove
• ASTREETNAM: Rename to StreetOne
• BSTREETDIR: Remove
• BSTREETNAM: Rename to StreetTwo
• INTNUM: Rename to NearestIntersection
• LONGITUDE, LATITUDE, _azimuth: Remove. 
• Make sure to use the up and down arrows at the bottom to rearrange the order of the attributes like the screenshot below. 

Click OK, then save the workspace and click Run Entire Workspace. Ensure that the output feature has all the relevant attributes, including Date, IntersectionID, TravelDirection, ADTOne, ADTTwo, NearestIntersection, StreetOne, and StreetTwo. Our last step is to write out the result ADT point to a File Geodatabase using the Esri Geodatabase (File Geodb Open API).

14. Add an Esri Geodatabase (File Geodb Open API) Writer

Click Add Writer on the toolbar and set the format as Esri Geodatabase (File Geodb Open API). For Dataset, browse to the demo OutputGDB folder and key in AverageDailyTraffic to create a new file geodatabase. 

C:\SJDOT_ADT_Mapping\OutputGDB\AverageDailyTraffic.gdb

Leave all other parameters at their defaults, then click OK. When the Select Feature Type dialog box appears, select Street_Intersections, then click OK. After the Writer is added, click on the cogwheel icon to open the Feature Type parameters window, and fill in the following information:

• Feature Class or Table Name: ADT_points_mapped
• Geometry: geodb_point
• Feature Operation: Insert
• Table Handling: Create If Needed
• Click on the User Attributes tab and make sure Attribute Definition is set to Automatic.

Click OK, make sure the workspace is saved, then click Run Entire Workspace. Select the writer on the canvas, then click the View Written Data button and double-check that our ADT point was written out to a new feature class named ADT_points_mapped inside the AverageDailyTraffic file geodatabase. 

Don’t forget to document the entire workspace with annotations and bookmarks. Use the screenshot below as a reference. 

Congratulations on completing the second part of the PDF Reader and ADT mapping workspace! You have successfully mapped point data without having longitude and latitude coordinates. By utilizing the principles of Linear Referencing, such as a reference point (intersection), a linear feature (street segment), and a specified distance along the linear feature (50 feet), you were able to overcome complex GIS challenges using FME Form. You should now feel more confident in working with list attributes, using Custom Transformers, and configuring various Testers and TestFilters to suit your project requirements. Well done!

Data Attribution

The data used is made available by the City of San Jose’s Department of Transportation provides the data used.