Reading Utility Network Data from ArcGIS Portal Feature Service and Writing to ArcGIS Portal Feature Service

As of FME 2024.0, support for Esri ArcGIS Online (AGOL), ArcGIS Enterprise Portal, and ArcGIS Server Feature Service formats has transitioned to a unified approach using the new Esri ArcGIS Feature Service (Format). This format replaces the legacy reader/writer formats for each of the three services.

Starting in FME 2026.1, the legacy formats will be hidden in the Quick Add menu. Instead, equivalent functionality is now provided by the downloadable Esri ArcGIS Connector package, available on FME Hub. The package also includes new web services and key transformers such as the ArcGISOnlineConnector, ArcGISAttachmentConnector, and ArcGISBranchVersionManager. The package can be installed directly through the Quick Add menu in FME Workbench. Once installed, the new Esri Feature Service format will appear in the Gallery as a unified method for interacting with all three types of Esri ArcGIS Feature Services.

For details on this transition, including guidance on updating existing workspaces, please refer to the article: Working with Esri ArcGIS Feature Services in FME.

Introduction

In this tutorial, we will explore how to effectively read and write data to Esri ArcGIS Utility Network using FME, specifically by utilizing the ArcGIS Portal Feature Service format for both the reader and writer. If you are interested in a tutorial on reading from or writing to a File or ArcSDE Geodatabase, please refer to the Tutorial: Geodatabase Transformation article. This tutorial will specifically demonstrate reading Utility Network data from ArcGIS Portal Feature Service and writing to ArcGIS Portal Feature Service using two different methods: Writer and FeatureWriter.

For a deeper understanding of the Esri ArcGIS Portal Feature Service Reader and Writer, consult the following documentation:

• Esri ArcGIS Portal Feature Service Reader
• Esri ArcGIS Portal Feature Service Writer

Throughout the tutorial series, we will guide you through configuring the reader and writer, as well as retrieving and publishing UN data, demonstrating how you can leverage FME for working with Utility Network data.

There will be a tutorial on writing to File Geodatabase with all the proper domain values rather than coded domain values for those who want to make localized versions. 

Requirements

• FME Form Workbench installed and licensed
• Esri ArcGIS Pro or ArcGIS Enterprise installed and licensed
• Basic understanding of Utility Network data structure and dataset, FME transformers, and FME and geodatabase
• Utility Network-configured ArcGIS Portal Feature Service item

Sources

• ReadPFS_WritePFS.fmw contains the completed workspace for this tutorial. Portal URL, Credentials, Feature Service, Layers/Feature Types, and Branch Versioning for Reader, Writer, and FeatureWriter will need to be re-configured to make the workspace usable.

Step-by-Step Instructions

Part 1: Configuring Reader for Esri ArcGIS Portal Feature Service 

1. Add an Esri ArcGIS Portal Feature Service Reader

In a blank workspace, add a Reader.

In the reader parameters, set the following: 
A. Set the Format as: Esri ArcGIS Portal Feature Service
B. Specify your organization's ArcGIS Portal URL for the Dataset.
C. Click "Parameters...".

In the Esri ArcGIS Portal Feature Service Parameters dialog:

Click the ellipsis next to Feature Service.

2. Edit Feature Service Parameter

In the Select From dialog: 
A. Specify your credentials for User and Password

• • Filling out the Credentials should give you access to the files. 
    • If you still don't have access, then follow step B,
    • If you do have access, you can skip step B.

B. Set Verify HTTPS Certificates: No

• • Filling out the credentials should give you access to the files.
  • Note: Verify HTTPS Certificate may need to be toggled to Yes, depending on your organization's protocols.

C. Select the parent folder of the Utility Network Data, then select the desired Utility Network Dataset, 

• • For this tutorial, “admin” then “Water Distribution Utility Network” will be selected.
  • If you cannot see anything in the area that is highlighted by the green outline in the screenshot, click the refresh button as indicated by the blue arrow.

D. Click OK

Tip: You can use Presets to retrieve the same configuration of parameters. Recommended
parameters to be set before saving as a Preset for Esri ArcGIS Portal Feature
Service Reader are: Portal URL, Authentication Type, User, Password, Feature Service,
and optionally, Branch Versioning.

Back in the Esri ArcGIS Portal Feature Service Parameters dialog:

Click the ellipsis next to Layers 

3. Edit Layers Parameter

In the Select Layers dialog:
A. Click "Select all". 
B. Click OK.

4. Enable Branch Versioning Parameter

Back in the parameters, expand and enable the Branch Versioning. 
A. For Version Name, select the desired Version to read, for this tutorial, select default.
B. For Version Data to Read, select the desired data to read, for this tutorial, select Features.

• • Features = read features on the selected Branch Version.
  • Difference = compares the selected Branch Version against the DEFAULT version.
    • Resulting features have the ‘fme_db_operation’ format attribute with a value of "insert", "update", or "delete", corresponding to the type of difference.
  • Optional: If you want to resolve domain values for “Asset Group”, you can expand on Schema Attributes, click the ellipsis next to Additional Attributes to Expose.
    This opens the Select 'Additional Attributes to Expose' Items dialog

a. Select "arcgisonline_subtype_name". 

b. Click OK.

• • Optional: If you want to write data to a File Geodatabase, creating two versions is recommended: one with resolved domains for better readability of attribute values, and one with non-resolved (coded value) domains.
  • If you are writing to an Esri ArcGIS Portal Feature Service, we recommend not resolving domains so the coded values are used. The non-resolved (coded value) version makes it easier to reconcile and post updates compared to a resolved, human-readable version when working with the Portal Feature Service across different environments or branches.

C. Click OK twice to finish adding the reader.

You might see some activity in the logs, then you will see these feature types on the workspace.

Data Structure Comparison between FME Reader and Esri ArcGIS Portal Feature Service

Upon examining the data structure in FME Workbench and comparing it to the ArcGIS Portal Feature Service, we can observe a clear correspondence between the two. The "Water Utility Network" layer in the ArcGIS Portal Feature Service has been decomposed into nine distinct System tables, each representing a specific aspect of the water utility network, such as the network's structure and connectivity. Currently, it is impossible to consolidate decomposed components for the "Water Utility Network" layer and the associated system tables in FME due to the data being very different. The system tables are for more advanced operations and may not be necessary for all workflows to interact with them.

Part 2: Configuring Writer for Esri ArcGIS Portal Feature Service

1. Add Writer to the Workspace Canvas
Continuing in the same workspace as Part 1, add an Esri ArcGIS Portal Feature Service writer to the canvas. 

In the parameters, set the following: 
A. Set the Format as Esri ArcGIS Portal Feature Service
B. Specify your organization’s Esri Portal URL for the Dataset.
C. Set the Layer Definition as: Import from Dataset… 
D. Click “Parameters…”.

In the Esri ArcGIS Portal Feature Service Parameters dialog:

Click the ellipsis next to Feature Service.

2. Edit Selected Feature Service Parameter

In the Select From dialog:
A. Specify your credentials for User and Password.

• • Filling out the Credentials should give you access to the files. 
    • If you still don't have access, then follow step B,
    • If you do have access, you can skip step B.

B. Set Verify HTTPS Certificates: No

• • Note: Verify HTTPS Certificate may need to be toggled to Yes, depending on your organization's protocols.

C. Select the parent folder of the Utility Network Data, then select the desired Utility Network Dataset

• • For this tutorial, select “admin” then “Water Distribution Utility Network”. 
  • If you cannot see anything in the area highlighted by the green outline in the screenshot, click the refresh button, which is highlighted with a blue arrow.

D. Click OK

3. Enable Branch Versioning

Back in the Esri ArcGIS Portal Feature Service Parameters dialog:  
A. Expand and enable the Branch Versioning.
B. For Version Name, select the desired version to write to:

• • • sde.DEFAULT (equivalent to sde.DEFAULT in ArcMap)
    • Or other Branch Versions available. These available Branch Versions will have the Owner's Username as the prefix. Prefix 'ADMIN.' will be present if an admin has created the Branch Version.
    • Or create a new Branch Version by entering the desired name, must be unique name. This will be the name present when viewing the branch version on Manage Versions in ArcGIS Pro and the default access to this Branch Version will be private.

Example: If the user used for the credentials for the Portal Feature Service is 'BOB' and the entered name for Branch Version Name is 'QA version' on FME Workbench, the name for the Branch Version on FME will be 'BOB.QA version'.  

• • Optional: Specify desired Reconcile and/or Post options (Reconcile must be enabled to select Post).
    • Branch Versions must be reconciled before being merged (posted) to the DEFAULT Version.
      • • Detects the existence of conflicts and logs warnings. 
    • For Post options:
      • Yes - Post the Branch Version to the DEFAULT Version.
      • If No Conflicts - Post the branch version to the DEFAULT Version if the reconcile operation did not detect conflicts.
      • No - Do not post to the DEFAULT Version.

Tip: Make sure to set the value for POST to "No" if you do not want to post to the DEFAULT version. 

C. Expand  Advanced and set “Yes” for Use Global IDs and Preserve Global ID 

• Optional: modifying the Feature per Request 
• This limit applies to Insert, Update, Delete, and truncation. 
• This parameter is useful for avoiding server timeouts or HTTP 500 responses that may result from long write operations.

D. Click OK.

Tip: You can use Presets to retrieve the same configuration of parameters. Recommended
parameters to be set before saving as a Preset for Esri ArcGIS Portal Feature
Service Writer are: Portal URL, Authentication Type, User, Password, Feature Service, Use Global IDs, Preserve Global ID, and optionally, Branch Versioning.

4. Import Writer Feature Types and Select Layers

Finally, in the Add Writer dialog: 
Click  OK

This will open the Import Writer Feature Types window
Click “Parameters…”.

In the Esri ArcGIS Portal Feature Service Parameters dialog:
Click the ellipsis next to Layers

In the Select Layers dialog:
A. Select the desired layers to write to

• • For this tutorial, we will be using all the layers, except tables and Utility Network System Tables, this includes: 
    • Service Territory
    • Structure Boundary
    • Structure Junction
    • Structure Line
    • Water Assembly
    • Water Device
    • Water Junction
    • Water Line
    • Water Subnet Line

B. Click OK three times to finish adding the writer

5. Connect the Readers to the Writer's Feature Types
Connect the corresponding layer from the reader or transformer to the writer. Disable all the layers that are not connected to the writer, as they are not needed for the current workflow.

If clicking on the arrow of the reader/transformer to the writer is too tedious, refer to this documentation and add connections using the "Feature Type Connections" view window.

6. Save and Run the Workspace
Now, save and run the workspace. View the output data in Esri ArcGIS Pro to see if it was properly written to the portal feature service.

Part 3: Configuring FeatureWriter for Esri ArcGIS Portal Feature Service

1. Add FeatureWriter to the Workspace Canvas
Continuing in the same workspace as Part 1 and Part 2, add a FeatureWriter transformer to the canvas.

In the FeatureWriter parameters, set the following:
A. Set the Format as Esri ArcGIS Portal Feature Service
B. Specify your organization's ArcGIS Portal URL for the Dataset.
C. Click "Parameters...".

In the Esri ArcGIS Portal Feature Service Parameters dialog:
Configure the parameters similar to what you have done in Part 2, Step 2, and Step 3. For branch versioning, specify a name different from what you have specified for Part 2, Step 3. 

Your parameters should look similar to this: 

Tip: If you have saved a preset after Part 2: Step 3, you can load in all the parameter configurations!

2. Defining Schema for Feature Types to be written

Back in the FeatureWriter dialog: 
A. Select 'Layer1' and remove the selected feature type definition by clicking on the minus (-) sign.

B. Add feature type by clicking on the plus (+) sign and selecting "Import from Dataset...". 

This will open the Import Schema Definitions window
A. Set the Format as Esri ArcGIS Portal Feature Service
B. Specify your organization's ArcGIS Portal URL for the Dataset.
C. Click "Parameters...".

In the Esri ArcGIS Portal Feature Service Parameters dialog:
The configuration for this section is the same as that for Parts 1, Step 2, and 3. For this tutorial,

A. Click the ellipsis next to Feature Service.

In the Select From dialog: 
A. Specify your credentials for User and Password
B. Set Verify HTTPS Certificates: No
C. Select the parent folder of the Utility Network Data, then select the desired Utility Network Dataset, 
D. Click OK

Back in the Esri ArcGIS Portal Feature Service Parameters dialog:
A. Click the ellipsis next to Layers

In the Select Layers dialog:
A. Select the desired layers to write to

• • For this tutorial, we will be using all the layers, except tables and Utility Network System Tables, this includes: 
    • Service Territory
    • Structure Boundary
    • Structure Junction
    • Structure Line
    • Water Assembly
    • Water Device
    • Water Junction
    • Water Line
    • Water Subnet Line

B. Click OK three times

In the Select schema definitions to add dialog:
A. Click "Select all".
B. Click OK

Your FeatureWriter window should look similar to this, where the Feature Types are now populated: 

Finally, in the FeatureWriter dialog:
A. Click OK

3. Connect the Readers to the FeatureWriter Feature Types
Connect the corresponding layer from the reader to the FeatureWriter feature types. 

Your final workspace should look similar to this:

4. Save and Run the Workspace
Now, save and run the workspace. View the output data in Esri ArcGIS Pro to see if it was properly written to the portal feature service.

Troubleshooting

Cannot write out 19700101000000 for dates for ESRI ArcGIS Portal Feature Service Writer

Solution: AttributeFilter and DateTimeConverter; 19700101000001/Local Timezone
Currently, if you specify the value of 19700101000000 for date fields for attributes, when you check the date on ArcGIS Pro the value is 12/30/1899. There are two methods to resolve this problem. 

1. If Hours, Minutes, and Seconds are not important, use the AttributeFilter transformer to select features with the value “19700101000000” for your date fields. Then, use the DateTimeConverter transformer and select “%B %e, %Y” for the Output Format.
2. If specifying Hours, Minutes, and Seconds is important, you can use the value "19700101000001" or coordinate with the local timezone to represent 01/01/1970 (i.e., 19700101060000 represents 1/1/1970 at local midnight for the Central timezone).

Other FME and Esri ArcGIS Errors 

If you encounter FME and Esri ArcGIS, this documentation is a comprehensive resource containing various solutions for troubleshooting.

Data Attribution

Water Distribution Utility Network Foundation data is copyright 2024 Esri. Licensed under Apache-2.0. Source: ArcGIS Solutions.