FME Version
Files
Introduction
With FME Flow (formerly FME Server) Workspace Apps, it’s easy to create a front-facing interface for users to download point cloud data using the FME Flow Data Download Service.
This article emphasizes the capabilities of the GeometryReplacer transformer, allowing the user to download a selected area. The user parameter manager is also highlighted as a key tool in creating a flexible workflow. The parameters let the user choose the output point cloud format, output coordinate system, an area of interest to which the results will be clipped, whether or not to split the output, and whether or not to reduce the number of points in the output by using a thinning algorithm. Part 1 walks through the workspace creation, and part 2 shows you how to create a workspace app on FME Flow.
Step-By-Step Instructions
Part 1: Creating the FME Workbench Workspace
The first part of this article explains how to create the workspace for the FME Flow Workspace App. The workspace will transform the data and be the main platform for setting up user parameters.
1. Add Creator Transformer
Start FME Workbench and open a new blank workspace. Begin typing on the blank workspace and enter “Creator”. Select the Creator transformer to add it to the workspace. We can use the default parameters.
2. Add GeometryReplacer
Add a GeometryReplacer transformer to the canvas. Connect the Creator to the GeometryReplacer. The GeometryReplacer will be used later in the FME Flow Workspace App to allow the user to clip the Point Cloud data to their desired area.
3. Setup GeometryReplacer Parameters
Next, we need to create a new Geometry parameter that can be used as the Geometry Source in the GeometryReplacer. To do this, right-click on User Parameters in the Navigator window and click Manage User Parameters.
In the User Parameters window, click on the green “plus” symbol in the top left and select Geometry to create a new Geometry user parameter. Use the following parameters:
- Identifier: GEOM_COORDS
- Prompt: Select Area
- Disable Attribute Assignment: Yes
Click the ellipsis to the right of Geometry Types and uncheck Point.
Check the box next to “Specify initial bounds for map display” and set the values as such:
- Top: 49.2856
- Left: -123.146
- Bottom: 49.2625
- Right: -123.107
Click OK to save the parameters. Next, open the GeometryReplacer parameters and set the Geometry Encoding parameter to GeoJSON and change Remove Attribute to No. Click on the drop-down arrow next to Geometry Source then select the GEOM_COORDS user parameter we just created.
4. Add CoordinateSystemSetter
Add a CoordinateSystemSetter transformer and connect the GeometryReplacer to it. Open up the transformer and set the Coordinate System parameter to LL84.
5. Add Reprojector
Add a Reprojector transformer and connect the CoordinateSystemSetter to it. Open up the Reprojector and set the Destination Coordinate System to UTM83-10.
Your workspace should look like this now:
6. Add FeatureReader
Add a FeatureReader transformer to the canvas and connect the Reprojector to it. The FeatureReader will be used to read the Point Cloud file.
Open the FeatureReader and set the parameters as follows:
- Format: ASPRS Lidar Data Exchange Format (LAS)
- Dataset: https://safe-kc.s3.us-west-2.amazonaws.com/Large+Datasets/Pointclouds/CoV_4910E_54580N.laz
- Coord. System: UTM83-10
- Feature Types to Read: LAS
- Spatial Filter: Initiator is OGC-Intersects Result
Click the Parameters box and click on Schema Attributes to reveal Additional Attributes to Expose. Assign this the value of “fme_basename,char(50)”.
7. Add Clipper
Add a Clipper transformer to the canvas. The Clipper transformer will be used to clip the Point Cloud dataset to the area of interest selected by the user.
Connect the LAS output port from the FeatureReader to the Clippee port of the Clipper transformer. Connect the Reprojector to the Clipper port.
8. Add AttributeExposer
We want to add the AttributeExposer to expose fme_basename, which will later be used for the file naming convention. Connect the Inside port to the AttributeExposer.
Open the Attribute Exposer and add “fme_basename” to Attributes To Expose.
9. Add a Tester
The next step is to create a workflow that allows the user to select whether or not they want to merge the data into one point cloud. This requires the use of a Tester to check whether the user says Yes or No.
Add a Tester and connect the AttributeExposer to it.
10. Setup Tester Parameters
We need to create a new user parameter so that the user can select Yes or No for this Tester.
Open up the User Parameter Manager and add a new Choice parameter. Set the name to TILED and change the Prompt to “Return Tiled Output”. Uncheck the Required box.
Under the Choices area on the right of the User Parameter Manager, add two new Values: YES and NO. Set the Default Value to No.
Go back and open the Tester transformer. Now, we will create the Test Clause that will decide whether the dataset will be merged into one point cloud. Set the test to $(TILED) = NO.
11. Add PointCloudCombiner
Add a PointCloudCombiner transformer and connect the Tester Passed port to it. We can use the default parameters.
12. Add PointCloudVerticalFilter
Next, we need to add a PointCloudVerticalFilter, which will allow us to keep the point cloud files as separate tiles. The PointCloudVerticalFilter is an FME Hub transformer that will automatically be downloaded via FME Workbench. For more information on this transformer, see the FME Hub.
Connect the Tester Failed port to the PointCloudVerticalFilter.
Open up the PointCloudVerticalFilter and set the Adjustment Parameter (# of Vertical Tiles) to 10.
13. Add Tester
Next, we want to create a workflow that will test whether the user wants to split their point cloud. To do this, we first want to add another Tester transformer. Then, we connect the PointCloudCombiner and the PointCloudVerticalFilter Output ports to it.
14. Setup Tester Parameter
Similar to Step 9, we want to set up a new user parameter that will be used in Tester_2. This will test whether the user wants to split the point cloud.
Go to the User Parameter Manager and add a new Choice parameter. Name it SPLIT_PC and change the Prompt to “Split Point Cloud”. Next, un-check the Required box. Under the Choices section, add Values YES and NO. Use the Display values it provides. Set the Default Value to No.
Go back and open the Tester_2 and set the test to $(SPLIT_PC) = YES.
15. Add PointCloudSplitter
Add a PointCloudSplitter transformer and connect the Tester_2 Passed port to PointCloudSplitter. Open PointCloudSplitter, click the dropdown next to Split By (under General), then User Parameter and Create User Parameter.
If you create a User Parameter directly from the transformer dialog (i.e., use the dropdown menu beside the parameter), the User Parameter will be created with the correct type.
Use the following values for the new parameter:
- Identifier: SPLIT_BY_DISPLAY
- Prompt: Split By
- Published: Yes
- Required: No
- Default Value: First Return Only
Press OK. Back to the PointCloudSplitter transformer, change Output Attribute Name to "_point_source_id".
16. Add Tester
Next, we need to use another Tester transformer to check whether or not the user wants to thin the output point cloud. This is useful for reducing the number of points in the point cloud and producing a smaller file. Add another Tester transformer and connect the Tester_2 Failed port and the PointCloudSplitter Split port to it.
17. Setup Tester Parameter
Again, we need to set up a user parameter in the Manage User Parameter window. Create a new Choice parameter and use the following values:
- Identifier: THIN_PC
- Prompt: Thin Point Cloud
- Required: No
Press OK to close.
Open up Tester_3 and set the test to $(THIN_PC) = YES.
18. Add PointCloudThinner
Add a PointCloudThinner transformer and connect the Tester_3 Passed port to it.
We need to establish three user parameters to be used in the PointCloudThinner transformer. Open the PointCloudThinner transformer. To create a new user parameter for each parameter, click the drop-down arrow, hover over the User Parameter, and click Create User Parameter.
Create a User Parameter for Points to Keep and call it POINTS_TO_KEEP. Un-check the Required box.
Create a User Parameter for Interval and call it THIN_INTERVAL. Un-check the Required box and clear the Default Value.
Create a User Parameter for Maximum Number of Points and call it MAX_NUM_POINTS. Un-check the Required box and clear the Default Value.
Your PointCloudThinner Parameters should now look like this:
19. Add AttributeCreator
Add an AttributeCreator transformer. Connect the Tester_3 Failed port and the PointCloudThinner Thinned port to it.
Open up the AttributeCreator and set the parameters as follows:
- New Attribute: _point_source_id
- Attribute Value: @Value(_point_source_id)
20. Add Tester
The fourth and final Tester transformer checks whether the point cloud was split into multiple point clouds. If it was split using the PointCloudSplitter, then an ID (_point_source_id) was created from the AttributeCreator used in the last step. This ID will be added to the output file name.
Add a new Tester transformer and connect the AttributeCreator to it. Open the Tester and set the following values:
- Left Value: _point_source_id
- Operator: =
- Right Value: _
Expand the Advanced heading and set Preserve Feature Order to Across Output Ports.
21. Add AttributeCreator
Add an AttributeCreator transformer. Connect the Tester_4 Failed port to it.
Open up AttributeCreator_2 and set the parameters as follows:
- New Attribute: fme_basename
- Attribute Value: @Value(fme_basename)@Value(_point_source_id)
22. Add Writer
Finally, we want to add a Generic Writer to write our Point Cloud file out. Add a new Writer to the Workspace and set the parameters as follows:
- Format: Generic (Any Format)
- Dataset: <any directory of your choice>
Click on Parameters and set the Output Format to ASPRS Lidar Data Exchange Format (LAS). Press OK to close the Parameters window and press OK again to set Feature Type.
In the Feature Type dialog that pops up, set:
- Feature Type Name: fme_basename
- Geometry: fme_any
We don’t need the User Attributes or Format Attributes. Click on the User Attributes tab, set the Attribute Definition to Manual, and remove all the values.
Click on Format Attributes and un-check fme_basename so it is no longer exposed.
Connect the Tester_4 Passed port and the AttributeCreator_2 Output port to the Writer.
23. Add Point Cloud XYZ Support
Because we want to let the user have the ability to export the output Point Cloud to either LAS or Point Cloud XYZ format, we have to create another writer that will be hidden in the background that will handle the Point Cloud XYZ parameters. This is because the writer needs to know which components are to be written out and the order of those fields in the output file (specified in the Component Mapping parameter for the writer). The Generic writer will use the parameter values set in the point cloud XYZ writer in the workspace when writing out to that format.
Add a Writer to the workspace. Set the parameters as follows:
- Format: Point Cloud XYZ
- Dataset: Any directory of your choice
Click on Parameters and add the following values to the Point Cloud Components to Output section:
Component | Column Name |
---|---|
intensity | Intensity |
classification | Classification |
return | Return |
number_of_returns | Number of Returns |
flight_line_edge | Flight Line Edge |
gps_time | GPS Time |
angle | Angle |
scan_direction | Scan Direction |
user_data | User Data |
Press OK to close the window and then press OK again to add the writer. Accept the default name. Since we don’t need the Writer on the workspace, we can right-click it and press Delete. When the window comes up asking you if you want to remove these readers and writers, click No. This will not delete the Writer we just created; it will just hide it in the background.
24. Add User Parameter for Output Format
We need to create a parameter that allows the user to choose whether to output an LAS file or a Point Cloud XYZ file.
Open up the User Parameter Manager window (again by right-clicking on User Parameters and selecting Manage User Parameters). Add a new Choice User Parameter. Set the following parameters:
- Identifier: OUTPUT_FORMAT
- Prompt: Output_Format
Expand Choice Configuration and add two new Values as follows:
Value | Display |
---|---|
LIDAR | ASPRS Lidar Data Exchange Format (LAS) |
POINTCLOUDXYZ | Point Cloud XYZ |
Set the Default Value to ASPRS Lidar Data Exchange Format (LAS).
25. Add User Parameter for Output Coordinate System
We also want the user to be able to choose the coordinate system to which they output their data. Create a new Choice User Parameter. Use the following parameters:
- Identifier: COORD_SYS
- Prompt: Output Coordinate System
Expand Choice Configuration and use the following values:
Value | Display |
---|---|
BCPOLY-83 | Polyconic Projection, British Columbia |
CSRS.BC<solidus>Albers | BC Albers [EPSG:3153] |
LL84 | WGS84 Lat/Long |
UTM83-10 | NAD83 datum, UTM Zone 10 |
Set the Default Value to NAD83 datum, UTM Zone 10.
26. Update Output Writer Parameters
To make sure that the option to choose which format to output to appears in the Workspace App we will create, we need to go to the Navigator window and expand <your folder name> [GENERIC]. Expand Parameters, right-click Output Format, and select Edit Parameter Value.
For Output Format, click the drop-down arrow on the right, hover over User Parameter, and select OUTPUT_FORMAT. In the same window, expand Coordinate System and change Coordinate System to the COORD_SYS User Parameter. Press OK to close the window.
27. Organize User Parameters
The order of User Parameters in the User Parameters Manager window determines the order in which they appear in the FME Flow Workspace App we create. We want parameters used for the same tool to be grouped together, so we need to organize them and move things around. You can press the green plus sign to create a new group. Other User Parameters can be dragged and dropped into these groups, and the order of these groups can be changed. The User Parameters should be grouped and ordered as follows:
- Output Format:
- Output Coordinate System:
- Return Tiled Output:
-
Split Point Cloud [Group]
- Split Point Cloud:
- Split By:
-
Thin Point Cloud [Group]
- Use Point Cloud Thinning:
- Points to Keep
- Interval:
- Maximum Number of Points:
- Select Area:
-
Private Parameters [Group] (Published: No):
- Destination Generic (Any Format) Folder:
- Destination Point Cloud XYZ Folder:
28. Add Bookmarks (Optional)
Bookmarks are a good way to organize your workspace and help you and other users understand the workflow.
To add a bookmark, select an area of the workspace (for example, a group of transformers) that you want the bookmark to encompass. Click the bookmark tool or right-click and select Create Bookmark. A bookmark appears around the selected area. For more information on bookmarks, see the documentation.
Part 2: Creating an FME Flow Workspace App
The second part of this article explains how to create an FME Flow Workspace App using the workspace created in Part 1. This section is relatively quick but rewarding as you create a user-friendly interface for point cloud data distribution.
1. Publish your Workspace to FME Flow
Save your workspace, then click Publish in the toolbar at the top of FME Workbench. Establish a new connection or use an existing connection. Choose a Repository and set the Workspace Name to PointCloudDataDownload.fmw.
Press Next, click the box to check off Data Download, and then click Publish.
For more in depth instructions on how to publish to FME Flow please see below:
-
To publish a workspace from FME Workbench to FME Flow, follow the generic instructions below using the specific details listed in the article. If the article does not have a specific repository name listed, we recommend creating a Training repository. If a specific service is not listed in the article, please use the default Job Submitter.
-
Click Publish
-
Add a Web Connection
-
- Web Service: FME Flow
- Server URL: http://localhost
- Connection Name: Training FME Flow
- Authentication: Basic
- Username: your username
- Password: your password
Click OK. FME will try to authenticate the connection. Once authenticated, this connection will be available whenever you publish a workspace. The connection will only need to be created once.
-
- Web Service: FME Server
- Connection Name: Training FME Server
- Server URL: http://localhost
- Username: your username
- Password: your password
Click Authenticate. FME will try to authenticate the connection. Once authenticated, this connection will be available whenever you publish a workspace. The connection will only need to be created once.
Click Next to move to the Create Repository page of the dialog
- Create Repository
- Upload Files
- Register Services
- Data Download Service returns results as a downloadable zip file
- Data Streaming Service returns results as a data stream
- Job Submitter Service accepts and runs workspace job requests
- KML Network Link returns a KML Network Link that can be used in Google Earth
- Notification Service allows for event-driven messaging
- Review Translation Log
From the File menu, click Publish to FME Flow, or from the toolbar ribbon, click Publish.
Toolbar Ribbon:
File Menu:
In the Publish to FME Flow wizard, a FME Flow connection needs to be selected or created. To create an FME Flow connection, click the drop-down under FME Flow Connection and select Connect to FME Flow. For more information on creating or managing web connections, see the documentation.
The following instructions are for a basic connection, but if your FME Flow is set up for Microsoft Azure, SAML, or Windows Credentials, please see the Using Web Connections to Connect to FME Flow documentation.
An FME Flow Connection dialog will appear; here, we will set up the connection. The connection information and credentials can be obtained from your FME Flow Administrator. If you have not yet logged into FME Flow after installing, please do so prior to continuing, as you will be prompted to change the password from the default:
Username: admin
Password: adminRepositories are used to store groups of related items that have been published to FME Flow (similar to directories). Items such as workspaces, source data, custom formats, custom transformers, and templates can be published to a repository.
If you need to create a new repository, click on the New button next to Repository Name. Enter the repository name and an optional description. Click OK to close the Create New Repository dialog.
To select an existing repository, click the drop-down next to Repository Name and select the desired repository.
Some workspaces require additional files to run correctly on FME Flow, so these need to be uploaded along with the workspace. To upload files, enable Upload data files at the bottom of the dialog.
When files are uploaded through the publishing menu, they are stored within the repository. If you want to upload folder files (such as geodatabases) or to specify where the files stored, the Select Files button allows you to do that. For detailed instructions on how to use the Select Files button, see the documentation.
The final step before publishing a workspace is to register services. Services control how the data will be output after the workspace is run.
Services return results in different forms:
Typically, the Job Submitter is always enabled, but multiple services can be enabled. Once services have been enabled, click Publish.
To confirm whether or not the workspace was published successfully, check the Translation Log. The Translation log will show which repository you published to, which files were included, and a quick link to run the workspace.
-
2. Log in to FME Flow
Go to your web browser of choice and then log in to FME Flow. If you are running on a local machine, use http://localhost as the URL.
3. Upload the Point Cloud Data
Before the Workspace App can run the workspace, it needs the point cloud data we are using for this tutorial. On the left side of the FME Flow home page, click on Resources and go to Data folder. Create a new folder called PointCloud. Click the PointCloud folder we just created. Click the Upload button in the top right and select Files to upload the CoV_PointCloud.zip file which is available for download from the Files section on this article.
4. Build Workspace App
On the side menu, click on Flow Apps to expand the menu and click Create Workspace App.
In the next page that loads, set the parameters as follows:
- Name: PointCloudDataDownload
- Title: Point Cloud Data Download Service
- Repository: <Repository where you published your PointCloudDataDownload.fmw>
- Workspace: PointCloudDataDownload.fmw
- Service: Data Download
Once everything looks right, press OK to create your Workspace App. You should be redirected to a new page that includes the link to your newly created Workspace App.
4. The finished Workspace App
You’ve now created a workspace app using published user parameters, that can transfer point cloud data smoothly over the Workspace App interface. Below you can see what your workspace app should look like.
Data Attribution
The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.
Comments
0 comments
Please sign in to leave a comment.