Bulk Edits PostgreSQL: Update a Database Using an FME Server App

Introduction

FME Server can create FME Server Workspace Apps that allow users to run a workspace and enter parameters without the need to log in. These apps allow users to update datasets on the fly while out in the field using their mobile phones. In this article, we show you how to set up PostGIS and FME Server to respond to bulk feature database edits. If a small number of edits are being made, a single insert can also be set. This is described in the following article Single edits PostgreSQL: Update a Database Using an FME Server App

Requirements

• PostgreSQL database with write permissions. See Creating a PostgreSQL/PostGIS Training Database with FME for instructions on how to create a local database. 

Step-by-Step Instructions

Part 1: FME Desktop

Before we begin, we will need to open the two workspaces in FME Desktop to change the parameters to our own PostGIS database. If you do not have a PostGIS database, please follow the instructions in Creating a PostgreSQL/PostGIS Training Database with FME before continuing. 

1. Open FME Desktop

Note: If you completed the Single Edits PostgreSQL article, you can skip to Part 1 - Step 3 as you will already have the table loaded into your database.

First, we need to create a table in our database. Open CreateTableInPostGIS.fmwt and expand the PostGIS writer in the Navigator window. Double-click on Connection to open the connection parameters.

 
In the Edit ‘PostGIS Training [PostGIS]’ dialog, click the drop-down next to Connection and either add a new database connection or use the FMETraining connection created in Creating a PostgreSQL/PostGIS Training Database with FME. 

2. Run the workspace

Save the workspace and then run it. This will create a new table in your PostGIS database called Tree_Survey.

Go into your PostGIS database, or inspect the output in FME to confirm that the table was written correctly.

3. Open Second Workspace

Close the first workspace and then open BulkPostGISTreeSurvey.fmw. This workspace reads in an Excel spreadsheet containing an entire neighborhood’s worth of a tree survey. The workspace will also read in the database table that we just created. Then using both data sources, the workspace will determine whether or not the data already exists in the database. Then based on the result, it will INSERT, UPDATE, or DELETE the Excel data in the database.

Let’s go through what this workspace does:

a. Reads in the Excel spreadsheet containing all of the tree surveys for each neighborhood

b. Reads in the Tree_Survey database table created with CreateTableInPostGIS.fmwt. Then tests Tree_ID from the Excel spreadsheet that exists in the database. If Tree_ID doesn’t exist, it will move on to the INSERT section. If the Tree_ID exists, it will move on to the UPDATE or DELETE section.

c. Tests to see if the tree has been removed, based on the Removed attribute. If yes, then it triggers the DELETE workflow. If no, then it moves to UPDATE.

d. Sets the database operation with the FME specific attribute fme_db_operation. Additionally, it cleans up any unneeded attributes.

e. Writes out to the same PostGIS database table as the FeatureReader, and sets the database operation based on the fme_db_operation in the earlier AttributeManagers.

f. Cleans up attributes in preparation to write out to a log file.

g. Writes out the Database Operation, Date and which attributes were written to the database in a Microsoft Excel file. This file can be used to monitor changes to the database.

4. Update FeatureReader

In this second workspace, open the FeatureReader parameters and change the Connection to the PostGIS connection you created in the previous workspace. Additionally, click the ellipsis next to Feature Types to Read and select the Tree_Survey in your database. Click OK to close the Feature Reader.

When you update the FeatureReader ensure that the output port public.Tree_Survey is connected to the FeatureJoiner Right input port.

5. Update PostGIS Writer

Next, we will need to update the connection parameters for the PostGIS Writer. In the Navigator window, change the PostGIS connection to your connection the same way as the previous workspace.

6. Test the Workspace

Before we publish this workspace to FME Server, it is a good idea to test it locally. Ensure that Prompt for User Parameters is enabled and then run the workspace.

In the Translation Parameter Values dialog, select any ONE feature type from the Excel file. You can choose more than one, but it will cause the workspace to run slower.

You can ignore the Microsoft Excel File parameter, this will become relevant in FME Server.

Uncheck Save As User Parameter Default Values

Once the parameters are entered, click Run. This will trigger an INSERT in the database since this is new data. You can inspect the database to confirm this update. As well you can also view the Microsoft Excel TreeLog, which will include the parameters we just set, the database operation INSERT and today’s date and time.

If you want to test DELETE and UPDATE, rerun the entire workspace and select the SAME Survey to Read parameter as before.

7. Publish to FME Server

Once you have confirmed that this workspace runs successfully with your database connection, it is time to publish it to FME Server.

Before publishing to FME Server, you need to decide if you want your Excel log file to be available to the user running the Server App. If you do not, you can either leave the Excel file to be updated locally on your machine OR you can set it up to update in the FME Server Resources folder.

To update the file to save it to FME Server, expand the TreeDatabase-LogReport writer in the Navigator window. Then double-click on the Destination Microsoft Excel File parameter to open up the edit dialog. In the edit dialog, click on the drop-down next to Destination Microsoft Excel File, then choose User Parameter > FME Server Parameters > FME_SHAREDRESOURCE_DATA. Then after $(FME_SHAREDRESOURCE_DATA) type in \ServerApps\TreeDatabase-LogReport.xlsx

$(FME_SHAREDRESOURCE_DATA)\ServerApps\TreeDatabase-LogReport.xlsx

Click OK. Save the workspace, and then click on the Publish icon in the top toolbar.

In the Publish to FME Server dialog, create an FME Server connection if you do not already have one. Then ensure this workspace is saved to a Repository called ServerApps. Be sure to upload your PostGIS connection. Then register with the Job Submitter service if you want your log file private and/or the Data Download service if you want the log file to be public. The Data Download service will download the log Excel file every time the app is run.

Part 2: FME Server

With the workspace published, we can create an FME Server Workspace App. An FME Server Workspace App allows users to run an FME Server workspace without logging in. It also allows the user to run the workspace on their mobile phone. For more information on FME Server Apps, please see the article Getting Started with FME Server Apps.

1. Log into FME Server

Log into FME Server with an account that has privileges to create FME Server Apps. Once logged in, expand the Sever Apps section, then select Build Workspace App.

2. Create FME Server Workspace App

On the Create Workspace App page, give your app a name. Then select the Repository and Workspace.

Choose which Service you want to use. Remember, if you want your log file public, use the Data Download service; if you want it private, use the Job Submitter.

Next, check the Expiration date. By default, apps are set to expire in 10 years. It might be a good idea to set this expiration date to a shorter time. This expiration time can be changed at any time.

Finally, ensure that User Can Upload is enabled. This allows the user to upload a new Excel file to the app.

Now expand the Parameters section, and double-check the defaults for the app. If your parameters have values next to them, click on the Reset button. Setting all the values to blank will ensure that the user doesn’t miss a parameter when they enter their own values. Also, ensure that all of the desired parameters are shown in the app, which is indicated by the checkmark next to each parameter.

Finally, you can expand the Customize Appearance section and add color, text, or images to your final app page.

Once you are satisfied with your app, click OK. You will get a URL that links to your app.

3. Test App

Click on the URL that was provided at the end of the app setup.

Note: To share the FME Server App with anyone, the FME Server must be accessible externally, not internally using localhost.

If you navigated away from the page with the URL, you can access it again by going to Manage Workspace Apps on the side menu and then clicking on the URL icon next to your app name.

To test the Workspace App, enter the following parameters:

• Survey to Read: Kitsilano_Trees
• Source Microsoft Excel Files(s): Upload the Vancouver_Trees.xlsx dataset

You can also change the parameters to trigger a DELETE or UPDATE by rerunning the app and selecting the same Survey to Read.

Data Attribution

The data shown here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.