Data Distribution for Raster Data

Introduction

This tutorial walks through the steps to create an FME Flow Data Download service to distribute raster data. The application will allow users to do the following:

• Download specific tiles of Vancouver aerial imagery using the Geometry Parameter
• Choose a  raster format output
• Choose between a greyscale and a color image
• Build a raster pyramid with the selected number of levels
• Create a thumbnail with a specified size

You will create this application using FME Workbench and FME Flow Workspace Apps.

Files

• RasterDataDownload.zip (144.5 MB)

Step-by-Step Instructions

Part 1: Load the Data

1. Start FME Workbench and Save the Workspace

When FME Workbench starts up, select New under Create a Workspace. Click Save and choose where you want to save your workspace. Call the workspace RasterDistribution.fmw. Download the RasterData folder from the Files section above, then move it to the same location as your workspace.

2. Add a Creator Transformer

We will use the Creator transformer to create a dummy feature since we don’t have a Reader. If we did not have this, the workspace would not run. 

Add a Creator to the canvas and open the parameters. Set the coordinate system to EPSG:4326. Leave the rest of the settings at their defaults.

3. Add a Published Parameter for the Geometry Parameter

In our FME Flow app, we want to let the user specify the area on the map for which they want raster data. 

Under Navigator, right-click User Parameters. Select Manage User Parameters to open a new window. Click on the green plus icon, and in the drop-down options, select Geometry. Fill in the parameter with the following:

• Parameter Identifier: GEOM
• Prompt: Create Clipping Area
• Published: Enabled
• Required: Enabled
• Geometry Encoding: GeoJSON
• Geometry Types: Polygon
• Top: 49.2548
• Left: -123.244
• Bottom: 49.3034
• Right: -123.071
• Default Value:     

{"type":"Polygon","coordinates":[[[-123.109703,49.280212],[-123.089447,49.27954],[-123.09494,49.268788],[-123.110046,49.273492],[-123.109703,49.280212]]]}

Click OK.

4. Add a GeometryReplacer

The GeometryReplacer will change the raster image to the user-selected area using GeoJSON coordinates.

Add the GeometryReplacer and connect the Creator to it. Open settings and set the Geometry Encoding to GeoJSON. The Geometry Source can be set to our new GEOM parameter by selecting the drop-down arrow and User Parameter. Click OK.

5. Match the Coordinate System of the User-Selected Area to the Raster Data

Insert the Reprojector transformer and connect the GeometryReplacer Output to it. 

In the parameters, enter the following:

• Source Coordinate System: LL84
• Destination Coordinate System: UTM83-10

Click OK.

6. Find the Data

Add a FeatureReader to the canvas and then connect the Reprojector Reprojected port to the Initiator input port. In the parameters, set the format as TIFF (Tagged Image File Format). 

We will temporarily set the Dataset to look for data locally to ensure everything runs smoothly. Click on the ellipsis and locate the file that contains all of the data. Select one of the images and click Open. Once this is added to your Dataset parameter, change the image file extension to \*.tif. This will allow the transformer to select any image in the folder. 

Under Feature Types to Read, click the ellipsis and check the TIFF option. Click OK.

For the Spatial Filter, set it as Bounding Boxes OGC-Intersect.

In the Output, select Specified and, after clicking on the ellipsis, select the TIFF option again. Click OK to close the Select Specified dialog. Finally, click on Parameters, then expand Schema Attributes. Click on the ellipsis next to Attributes to Expose and select fme_basename, click OK twice, and OK again to close the transformer dialog. 

7. Add an AttributeExposer

We need to expose the fme_basename attribute using the AttributeExposer. This will give us some metadata information on the rasters. Connect the TIFF port of the FeatureReader to the AttributeExposer. In the parameter dialog, type fme_basename as the Attributes to Expose.

8. Run the Workspace

Before we continue, we must ensure the workspace is working properly. Run the workspace with Feature Caching Enabled. On the FeatureReader, click on the TIFF green magnifying glass to view the data in Visual Preview. The result should look like this:

9. Upload the Data to FME Flow

Log in to your FME Flow account and under Files & Connections, select Resources.

Click the Data folder, then create a new folder called RasterData. Open this folder and select Upload > Files to upload all the image files. You can use Ctrl-A to select all of the images at once. 

After the upload completes, select Data at the top to return to the Data folder. Check the box next to the RasterData folder, then, under Actions, select Properties. Copy the System Path. 

For detailed instructions on how to create a connection to FME Flow and upload workspaces, please see the section below:

• To publish a workspace from FME Workbench to FME Flow, follow the generic instructions below using the specific details listed in the article. If the article does not have a specific repository name listed, we recommend creating a Training repository. If a specific service is not listed in the article, please use the default Job Submitter.

  1. Click Publish

  From the File menu, click Publish to FME Flow, or from the toolbar ribbon, click Publish.

  Toolbar Ribbon:
  
  
  File Menu:
  

  Deploy Menu (2025.1+):
  

  2. Add a Web Connection

  In the Publish to FME Flow wizard, a FME Flow connection needs to be selected or created. To create an FME Flow connection, click the drop-down under FME Flow Connection and select Connect to FME Flow.

  

  The following instructions are for a basic connection, but if your FME Flow is set up for Microsoft Azure, SAML, or Windows Credentials, please see the Using Web Connections to Connect to FME Flow documentation.

  An FME Flow Connection dialog will appear; here, we will set up the connection. The connection information and credentials can be obtained from your FME Flow Administrator. If you have not yet logged into FME Flow after installing, please do so prior to continuing, as you will be prompted to change the password from the default:
  Username: admin
  Password: admin

  • • Web Service: FME Flow

    • Server URL: http://localhost

    • Connection Name: Training FME Flow

    • Authentication: Basic

    • Username: your username

    • Password: your password

    Click OK. FME will try to authenticate the connection. Once authenticated, this connection will be available whenever you publish a workspace. The connection will only need to be created once.

  • • Web Service: FME Server

    • Connection Name: Training FME Server

    • Server URL: http://localhost

    • Username: your username

    • Password: your password

    Click Authenticate. FME will try to authenticate the connection. Once authenticated, this connection will be available whenever you publish a workspace. The connection will only need to be created once.

    Click Next to move to the Create Repository page of the dialog

  3. Create Repository

  Repositories are used to store groups of related items that have been published to FME Flow (similar to directories). Items such as workspaces, source data, custom formats, custom transformers, and templates can be published to a repository.

  If you need to create a new repository, click on the New button next to Repository Name. Enter the repository name and an optional description. Click OK to close the Create New Repository dialog.
  

  To select an existing repository, click the drop-down next to Repository Name and select the desired repository.
  

  4. Upload Files

  Some workspaces require additional files to run correctly on FME Flow, so these need to be uploaded along with the workspace. To upload files, enable Upload data files at the bottom of the dialog.

  When files are uploaded through the publishing menu, they are stored within the repository. If you want to upload folder files (such as geodatabases) or to specify where the files stored, the Select Files button allows you to do that. For detailed instructions on how to use the Select Files button, see the documentation.

  5. Register Services

  The final step before publishing a workspace is to register services. Services control how the data will be output after the workspace is run.

  

  Services return results in different forms:

  • Data Download Service returns results as a downloadable zip file

  • Data Streaming Service returns results as a data stream

  • Job Submitter Service accepts and runs workspace job requests

  • KML Network Link returns a KML Network Link that can be used in Google Earth

  • Notification Service allows for event-driven messaging

  Typically, the Job Submitter is always enabled, but multiple services can be enabled. Once services have been enabled, click Publish.

  6. Review Translation Log

  To confirm whether or not the workspace was published successfully, check the Translation Log. The Translation log will show which repository you published to, which files were included, and a quick link to run the workspace.

10. Change the Local Path to the System Path

Back in FME Workbench, open the FeatureReader Parameters and paste the System Path in the Dataset. Append to the path /* so it looks like the following:

Dataset: $(FME_SHAREDRESOURCE_DATA)/RasterData/*    

Your RasterData folder must be in the same folder as your workspace in order for this to work.

11. Run the Workspace

Let’s make sure that the workspace still works after this change. Rerun the workspace, and you should see the same results.

Part 2: Create Mosaic

12. Mosaic the Rasters

Add the RasterMosaicker and connect the FeatureReader TIFF output port to it. This will merge the raster images together. Leave the default settings as they are.

Part 3: Create Conditions

13. Create Greyscale Published Parameter

Right-click User Parameters in the Navigator and select Manage User Parameters. Click the green plus icon, then select Choice. Set the parameters as follows:

• Parameter Identifier: Greyscale
• Prompt: Output as Greyscale
• Published: Enabled
• Required: Enabled
• Drop Down: Selected

• Allow Choice Edit: No
• Default Value: Yes

14. Create Thumbnail Published Parameter

Create another Choice parameter using the following parameters:

• Parameter Identifier: Thumbnail
• Prompt: Create Thumbnail
• Published: Enabled
• Required: Enabled
• Drop Down: Selected 

• Allow Choice Edit: No
• Default Value: Yes

@Value(fme_basename)_thumb

@Value(fme_basename)_pyraLvl_@Value(_pyramid_level)

Data Attribution

The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.