FME Flow Resources - Uploading, Referencing and Configuration

Liz Sanderson
Liz Sanderson
  • Updated

FME Version

Introduction

Resource folders are convenient storage areas for uploading and storing data. Data stored in a resources folder can be used when publishing or running workspaces. In addition, existing directories (such as UNC shares) can be mounted and used when publishing from FME Workbench or browsed when submitting a job through the web interface.

Requirements

  • Download and extract required-data.zip

Step-by-step Instructions

Example 1: Upload Data via FME Flow

This example will demonstrate how to upload data to a Resources folder on FME Flow (formerly FME Server) through the web interface and then utilize the data when running a job from a published workspace.


1. Upload Data to FME Flow through the Web UI

Enter the URL in a web browser to access the FME Flow web UI. In the left menu bar, click Resources > Data. To create a new folder, select the New button (1), or to upload existing data, click the Upload button (2). 

UploadtoFlow.png


2. Create a New Workspace 

Open a new FME Workbench > File > Generate Workspace (Ctrl+G)

Configure Generate Workspace.

  • Reader Format (1): Esri Geodatabase (File Geodb Open API)
  • Reader Dataset (2): Click the ellipsis to find the CityData.gdb folder path
  • Reader Parameters (3): Click Parameters, then the ellipsis under Constraints > Tables, and select CityGrid/CITY_GRID
  • Writer Format (4): Esri Geodatabase (File Geodb Open API)
  • Writer Dataset (5): Same as Reader (or any preferred output directory)


3. Run the Workspace

It is important to ensure the workspace runs correctly on your computer before uploading it to FME Flow. Click the green arrow from the menu bar to check if Prompt for Parameters is checked. Then select Rerun Entire Workspace.
RunWorkspace.png

4. Upload Workspace to FME Flow
In the menu bar, click the upload button next to the FME Flow connection name to bring up the publish screen.
PublishButton.png

Select your FME Flow Connection (1), the repository you wish to publish to, and name your workspace example1-citydata.fmw (2).


Under the Register Services window, select both Data Download and Job Submitter.


5. Browse and Use Data Uploaded to FME Flow

Open the FME Flow Web Interface. On the homepage, under Last Published Workspaces, select the workspace you just published. This will display the Run Workspace page. Under Published Parameters, click the Browse Resources button.

Browse to the folder with your name that you created in the Data folder. Click the (+) button to choose the CityData.gdb folder and then click OK.

Back to the Run Workspace page, you will see the path to the geodatabase has changed to the Shared Resources path. Run the workspace.

Example 2: Upload Data via FME Workbench

This example will demonstrate how to upload data to a Resources folder when first publishing a workspace from FME Workbench using the ‘select files’ option. This data then becomes available for use at any time by other published workspaces.

1. Create a New Workspace 

Open a new FME Workbench > File > Generate Workspace (Ctrl+G)

Configure Generate Workspace.

  • Reader Format (1): MapInfo TAB (MITAB). If the selection is not available, search in More Formats
  • Reader Dataset (2): Click the ellipsis and insert Parks.tab data path
  • Writer Format (3): MapInfo TAB (MITAB). If the selection is not available, search in More Formats
  • Writer Dataset (4): Click the ellipsis and insert the data folder path

Once the workspace has been generated, open the writer feature type parameters and change the Table Name to Parks2 to prevent a naming conflict. 

Run the workspace to ensure that it is working correctly.

2. Upload Workspace along with Data to FME Flow

In the menu bar, click the upload button next to the FME Flow connection name to bring up the publish screen. Configure your FME Flow connection, choose a repository folder for your workspace, and name workspace example2-parks.fmw.

On the same publish screen, check Upload data files (1), click Select Files, and ensure the files selected are the correct data files (2), then click Select Location (3). In the final pop-up window, check Upload to a shared resource folder (4), then choose the folder with your name to deposit your data files (5).

Finally, register with the Job Submitter service and finish publishing.

3. Run the Workspace on FME Flow

Return to FME Flow and run example2-parks workspace.

Under Published Parameters, notice this time the resource folder is already specified for the Source MapInfo TAB file.

Now that the data has been uploaded to the Resources folder, you can reference it in any future Workspace that requires the Parks.tab dataset.

Example 3: Upload Data via the FME Flow System Share

The workspace created in example 2 can be used here. This example demonstrates how to upload data into the Resources folder without logging on to the FME Flow Web Interface by looking directly into the operating system’s file system.

Copy data into the Resources Folder Within the File Directory

In the local drive, select View > Show > Hidden Items.

The folder ProgramData has now been revealed. This is where the FME System Share is located and is home to the Resources folders.

Navigate to ProgramData > Safe Software > FME Flow > resources > data.

If you have already completed the previous examples, you will find <your name> folder located here.

If not, open the File Directory and navigate to the data folder you downloaded and extracted at the beginning of this article (required-data). Copy and Paste the data folder here and rename it to <your name>.

When you return to the FME Floe Web UI, this data will be available.

Example 4: Using a Default Resource Folder in a Workspace

The workspace created in example 2 can be used here. This example demonstrates reading data from a default FME Flow Shared Resource folder using a parameter so that it doesn’t need to be selected at runtime.

You can view the parameters available for default FME Flow Shared Resource folders in Navigator.


Add the Resource Location to a Reader Using an FME Flow Parameter 

Open example2-parks.fmwt, or create a new workspace following the steps in Example 2 above if you have not already done so.

Right-click the MITAB reader > Edit Parameters, or expand the MITAB Reader in the Navigator panel > Source Parameter.

For the Source MapInfo File(s), click the drop-down arrow and then User Parameter > FME Flow Parameters > FME_SHAREDRESOURCE_DATA. This parameter corresponds to the Data folder path under Resources in FME Flow.

Add \<your name>\Parks.tab to the end of the FME Flow parameter to complete the path to the Parks.tab file uploaded in Example 2.


If you try to run FME Workbench at this stage, you will encounter an error. This occurs because FME_SHAREDRESOURCE_DATA corresponds to a resource location on FME Flow. To run the workspace successfully in FME Workbench, you must first define the parameter within Workbench.

In the Navigator panel > FME Flow Parameters > [FME_SHAREDRESOURCE_DATA], point the FME Flow Data Directory to your local operating system's file system in example 3. In this case, it would be "C:\ProgramData\Safe Software\FMEFlow\resources\data".


The workspace is now ready to be saved and published to FME Flow.

You can also browse the Resources available on FME Flow by expanding the drop-down tab next to Dataset and choosing Select File From Web > Browse FME Flow…. This can also be done in the Add Reader Wizard when the reader is first added to the workspace, as shown below.
 

Example 5: Using a Custom Resource Folder Within a Workspace

This example demonstrates how to use a parameter that contains a resource folder you created within an FME Workspace.

If you see an error message such as "Failed to open...", you may need to change the user account that the FME Flow Engines service is using. For more information, please see the documentation here.


5a: Reading Data

1. Get the Parameter Name

In the FME Flow web UI side menu, select Resources. Click the check box next to the resource folder. In the example below, a custom network-based resource named My Data. From the Actions menu, select Properties.

The System Path displays the parameter name that can be used in an FME Form Workspace.

 2. Testing the Parameter for Reading Data

Open example2-parks.fmw, in the Navigator panel, right-click User Parameters > Manager User Parameters.

Modify the linked parameter created by FME Workbench when the MITAB reader was added. Uncheck Published to make the parameter private and change the Default Value to use the parameter name for your resource, followed by the file name.

Now, expand the Reader. Notice the source file location has changed since the linked parameter default value was updated.

The parameter may appear with a red cog, as shown below. This is because FME Form is not aware of parameter names for new resources created in FME Flow. You can disregard this—as long as the parameter name is correct, the workspace will still run successfully once published to FME Flow.
 

Publish the workspace to FME Flow with the Job Submitter service enabled. Since this Workspace has already been published, you may be able to republish it to save repeating the entire publishing process.

Return to the web interface and run the workspace example2-Parks.fmw. This time, you will notice that there is no option to point FME Flow to a Source file as this is no longer a parameter accessible to the User (because Published was unchecked).


5b: Writing Data

This example demonstrates writing data using a parameter to specify file location.

1. Add the Parameter to a Simple Workspace Writer

Open example2-Parks.fmw, add a writer to the workspace, and set the Format to Esri Shapefile. Temporarily set the Destination Esri Shapefile Folder to a local folder, for example, C:\Data. You will receive an error if you set the destination with the parameter during creation since FME Workbench is not aware of it.

Open the writer and enter the resource folder’s parameter, followed by parks, for the destination folder. This will create a new Parks subfolder within your resource folder for the shapefile and its sidecar files. 


2. Publish and Run the Workspace

Publish the Workspace to FME Flow, making sure that the Upload data files option is left unchecked. Return to the FME Flow Web Interface and run the newly published workspace.

Upon completion, return to your Resources folder (for example, Resource > My Data), and you should see a new folder containing the Park shapefile and sidecar files.


 

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.