Data Loading Options With FME Server

Introduction

The data loading capabilities of FME Server allows your users to take data from almost any source and automatically load it directly into your databases. In the background, data loading itself is a two step process. First, access to the data is provided by the user to a workspace hosted on FME Server, then that workspace is run taking that data as its source and writing it out to its final output location. This process can be handled either through web services or the Notification Service.
 

Data Loading with Web Services

To allow for the data to be read into a workspace, the source dataset parameter on the reader must be published. Once the workspace is published to FME Server, your users can drag and drop files, browse and select files locally or on the server, or even provide a URL to a web service to make the data accessible to the reader inside the workspace.

Uploading Data Through a Web Form
This data loading method makes use of the Data Upload Service which allows users to push data up to the FME Server, making it available for use as a source datasets inside a workspace. Users can interact with the Data Upload Service through the FME Server Web User Interface or through a custom web page.

For detailed information about the Data Upload Service, see the following documentation: FME Server Data Upload Service. For some practical examples on uploading data through the Web Interface and creating your own web page for data upload, try out this tutorial: Data Upload with FME Server

Consuming Data from a Web Service
Instead of having your users upload the data to be loaded, a workspace on FME Server can be configured to read data directly from a URL. When the source dataset is published, all you need to do is provide the link as the source dataset and the reader will get the data directly from that location.

For more information about consuming data directly from a web service, take a look at the tutorial page here: Read Data from a URL.

Posting Data to an FME Server Web Service
Another option for loading data with FME is to POST it directly to FME Server and trigger a workspace to read that data as its source dataset. This means is that instead of telling FME where to find the source dataset, you can simply send the data itself directly to the workspace on FME Server.

Take a look at the following tutorial for more information about how this works and how to configure workspaces on FME Server to accept this type of input: Sending Source Data to FME Server with HTTP POST
 

Data Loading with the Notification Service

As of FME Server 2019+, this functionality can be completed using FME Server Automations. For more information, please see our tutorials on receiving email in Automations, directory modified trigger, and the manual Automations trigger.

The Notification Service can be used to push data to FME Server in the form of messages. These messages can be sent to FME Server in a variety of ways including via an HTTP POST, through an email sent to the Server, or when data inside a particular directory is updated.

Sending Data in a Notification Message
Using the Notification Service, you can send messages to FME Server that can contain any kind of information. For example, you could send a JSON message containing the location of a dataset that is ready for loading into a centralized database and use transformers such as the JSONFlattener and FeatureReader to extract the file path and then read in the data. Or, simply configure a sensor to send updates in the form of text directly to FME Server and kick off other FME processes in response to the information in the update.

For more information on sending and receiving notifications in FME Server, take a look at this article: Receiving, Inspecting, and Processing Notification Messages

Email Data into the Server
FME Server can be configured to receive emails and then pass the content of the email (including any attached files) into a workspace in JSON format. For information on how FME Server's email capabilities work, see the following article: Email in FME Server: A Tutorial Series

For more specific information on how to configure FME Server to receive emails and set up a workspace to process the corresponding notification message, try out the receiving email tutorial that starts here: Receiving Email, Part 1: Setting up a Topic and an Email Address

Watch a Directory and Load New Datasets
Another option for streamlining your data loading process is through the Directory Watch publisher (a new addition to the notification service for FME 2014). It can be set to actively monitor activity on a shared resource or other directory and send out messages when files are added, updated, or deleted.

To try this out for yourself, take a look at the demo here: Directory Watch Tutorial