Reading Utility Network Data from ArcGIS Portal Feature Service and Writing to ArcGIS Portal Feature Service

Introduction

In this tutorial, we will explore how to effectively read and write data to Esri ArcGIS Utility Network using FME, specifically by utilizing the ArcGIS Portal Feature Service format for both the reader and writer. If you are interested in a tutorial on reading from or writing to a File or ArcSDE Geodatabase, please refer to the Tutorial: Geodatabase Transformation article. This tutorial will specifically demonstrate reading Utility Network data from ArcGIS Portal Feature Service and writing to ArcGIS Portal Feature Service using two different methods: Writer and FeatureWriter.

For a deeper understanding of the Esri ArcGIS Portal Feature Service Reader and Writer, consult the following documentation:

• Esri ArcGIS Portal Feature Service Reader
• Esri ArcGIS Portal Feature Service Writer

Throughout the tutorial series, we will guide you through configuring the reader and writer, as well as retrieving and publishing UN data, demonstrating how you can leverage FME for working with Utility Network data.

There will be a tutorial on writing to File Geodatabase with all the proper domain values rather than coded domain values for those who want to make localized versions. 

Requirements

• FME Form Workbench installed and licensed
• Esri ArcGIS Pro or ArcGIS Enterprise installed and licensed
• Basic understanding of Utility Network data structure and dataset, FME transformers, and FME and geodatabase
• Utility Network-configured ArcGIS Portal Feature Service item

Sources

• ReadPFS_WritePFS.fmw contains the completed workspace for this tutorial. Portal URL, Credentials, Feature Service, Layers/Feature Types, and Branch Versioning for Reader, Writer, and FeatureWriter will need to be re-configured to make the workspace usable.

Step-by-Step Instructions

Part 1: Configuring Reader for Esri ArcGIS Portal Feature Service 

1. Add an Esri ArcGIS Portal Feature Service Reader
In a blank workspace, add a Reader.

In the reader parameters, set the following: 
A. Set the Format as: Esri ArcGIS Portal Feature Service
B. Specify your organization's ArcGIS Portal URL for the Dataset.
C. Click on "Parameters...".

In the Esri ArcGIS Portal Feature Service Parameters dialog:
A. Click on the ellipsis (...) next to Feature Service.

2. Edit Feature Service Parameter

In the Select From dialog: 
A. Specify your credentials for User and Password

• • Filling out the Credentials should give you access to the files. 
    • If you still don't have access, then follow step B,
    • If you do have access, you can skip step B.

B. Set Verify HTTPS Certificates: No

• • Filling out the credentials should give you access to the files.
  • Note: Verify HTTPS Certificate may need to be toggled to Yes, depending on your organization's protocols.

C. Select the parent folder of the Utility Network Data, then select the desired Utility Network Dataset, 

• • For this tutorial, “admin” then “Water Distribution Utility Network” will be selected.
  • If you cannot see anything in the area that is highlighted by the green outline in the screenshot, click on the refresh button as indicated by the blue arrow.

D. Click on "OK".

Tip: You can use Presets to retrieve the same configuration of parameters. Recommended
parameters to be set before saving as a Preset for Esri ArcGIS Portal Feature
Service Reader are: Portal URL, Authentication Type, User, Password, Feature Service,
and optionally, Branch Versioning.

Back in the Esri ArcGIS Portal Feature Service Parameters dialog: 
A. Click on the ellipsis next to Layers 

3. Edit Layers Parameter

In the Select Layers dialog:
A. Click on "Select all". 
B. Click on "OK".

4. Enable Branch Versioning Parameter

Back in the parameters, expand and enable the Branch Versioning. 
A. For Version Name, select the desired Version to read, for this tutorial, select default.
B. For Version Data to Read, select the desired data to read, for this tutorial, select Features.

• • Features = reads features on the selected Branch Version.
  • Difference = compares the selected Branch Version against the DEFAULT version.
    • Resulting features have the ‘fme_db_operation’ format attribute with a value of "insert", "update", or "delete", corresponding to the type of difference.
  • <Optional> If you want to resolve domain values for “Asset Group”, you can expand on Schema Attributes, click on the ellipsis next to Additional Attributes to Expose.
    This opens Select 'Additional Attributes to Expose' Items dialog

a. Select "arcgisonline_subtype_name". 

b. Click on "OK".

• • <Optional> If you want to write data to a File Geodatabase, creating two versions is recommended: one with resolved domains for better readability of attribute values, and one with non-resolved (coded value) domains.
  • If you are writing to an Esri ArcGIS Portal Feature Service, we recommend not resolving domains so the coded values are used. The non-resolved (coded value) version makes it easier to reconcile and post updates compared to a resolved, human-readable version when working with the Portal Feature Service across different environments or branches.

C. Click on "OK" and “OK” again to finish adding the reader. 

• • You might see some activity in the logs, then you will see these feature types on the workspace.

Data Structure Comparison between FME Reader and Esri ArcGIS Portal Feature Service

Upon examining the data structure in FME Workbench and comparing it to the ArcGIS Portal Feature Service, we can observe a clear correspondence between the two. The "Water Utility Network" layer in the ArcGIS Portal Feature Service has been decomposed into 9 distinct System tables, each representing a specific aspect of the water utility network, such as the network's structure and connectivity. Currently, it is not possible to consolidate decomposed components for the "Water Utility Network" layer, the associated system tables, in FME due to the data being very different from one another. The system tables are for more advanced operations, and may not be necessary for all workflows to interact with them.

Part 2: Configuring Writer for Esri ArcGIS Portal Feature Service

1. Add Writer to the Workspace Canvas
Continuing in the same workspace as Part 1, add an Esri ArcGIS Portal Feature Service writer to the canvas. 

In the parameters, set the following: 
A. Set the Format as: Esri ArcGIS Portal Feature Service
B. Specify your organization’s Esri Portal URL for the Dataset.
C. Set the Layer Definition as: Import from Dataset… 
D. Click on “Parameters…”.

In the Esri ArcGIS Portal Feature Service Parameters dialog:
A. Click on the ellipsis next to Feature Service.

2. Edit Selected Feature Service Parameter

In the Select From dialog: 
A. Specify your credentials for User and Password.

• • Filling out the Credentials should give you access to the files. 
    • If you still don't have access, then follow step B,
    • If you do have access, you can skip step B.

B. Set Verify HTTPS Certificates: No

• • Note: Verify HTTPS Certificate may need to be toggled to Yes, depending on your organization's protocols.

C. Select the parent folder of the Utility Network Data, then select the desired Utility Network Dataset

• • For this tutorial, select “admin” then “Water Distribution Utility Network”. 
  • If you cannot see anything in the area that is highlighted by green outline in the screenshot, click on the refresh button as highlighted with blue arrow.

D. Click on “OK”.

3. Enable Branch Versioning

Back in the Esri ArcGIS Portal Feature Service Parameters dialog:  
A. Expand and enable the Branch Versioning.
B. For Version Name, select the desired version to write to:

• • • sde.DEFAULT (equivalent to sde.DEFAULT in ArcMap)
    • Or other Branch Versions available. These available Branch Versions will have the Owner's Username as the prefix. Prefix 'ADMIN.' will be present if an admin has created the Branch Version.
    • Or create a new Branch Version by entering the desired name, must be unique name. This will be the name present when viewing the branch version on Manage Versions in ArcGIS Pro and the default access to this Branch Version will be private.

Example: If the user used for the credentials for the Portal Feature Service is 'BOB' and the entered name for Branch Version Name is 'QA version' on FME Workbench, the name for the Branch Version on FME will be 'BOB.QA version'.  

• • <Optional> Specify desired Reconcile and/or Post options (Reconcile must be enabled to select Post).
    • Branch Versions must be reconciled before being merged (posted) to the DEFAULT Version.
      • • Detects the existence of conflicts and logs warnings. 
    • For Post options:
      • Yes - Post the Branch Version to the DEFAULT Version.
      • If No Conflicts - Post the branch version to the DEFAULT Version if the reconcile operation did not detect conflicts.
      • No - Do not post to the DEFAULT Version.

Tip: Make sure to set the value for POST to "No" if you do not want to post to the DEFAULT version. 

C. Expand  Advanced and set “Yes” for Use Global IDs and Preserve Global ID 

• <Optional>: modifying the Feature per Request 
• This limit applies to Insert, Update, Delete, and truncation. 
• This parameter is useful for avoiding server timeouts or HTTP 500 responses that may result from long write operations.

D. Click on “OK”.

Tip: You can use Presets to retrieve the same configuration of parameters. Recommended
parameters to be set before saving as a Preset for Esri ArcGIS Portal Feature
Service Writer are: Portal URL, Authentication Type, User, Password, Feature Service, Use Global IDs, Preserve Global ID, and optionally, Branch Versioning.

4. Import Writer Feature Types and Select Layers

Finally, in the Add Writer dialog: 
A. Click on “OK”.

This will open Import Writer Feature Types window
A. Click on “Parameters…”.

In the Esri ArcGIS Portal Feature Service Parameters dialog:
A. Click on the ellipsis next to Layers

In the Select Layers dialog:
A. Select the desired layers to write to

• • For this tutorial, we will be using all the layers, except tables and Utility Network System Tables, this includes: 
    • Service Territory
    • Structure Boundary
    • Structure Junction
    • Structure Line
    • Water Assembly
    • Water Device
    • Water Junction
    • Water Line
    • Water Subnet Line

B. Click on “OK” three times to finish adding the writer

5. Connect the Readers to the Writer Feature Types
Connect the corresponding layer from the reader or transformer to the writer. Make sure to disable all the layers that are not connected to the writer, as they are not needed for the current workflow.

If clicking on the arrow of the reader/transformer to writer is too tedious, refer to this documentation using the "Feature Type Connections" view window to add connections.

6. Save and Run the Workspace
Now, save and run the workspace. View the output data in Esri ArcGIS Pro to see if it was properly written to the portal feature service.

Part 3: Configuring FeatureWriter for Esri ArcGIS Portal Feature Service

1. Add FeatureWriter to the Workspace Canvas
Continuing in the same workspace as Part 1 and Part 2, add FeatureWriter transformer to the canvas.

In the FeatureWriter parameters, set the following:
A. Set the Format as: Esri ArcGIS Portal Feature Service
B. Specify your organization's ArcGIS Portal URL for the Dataset.
C. Click on "Parameters...".

In the Esri ArcGIS Portal Feature Service Parameters dialog:
Configure the the parameters similar to what you have done in Part 2 Step 2 and Step 3. For Branch Versioning, specify a different name than what you have specified for Part 2 Step 3. 

Your parameters should look similar to this: 

Tip: If you have saved a preset after Part 2: Step 3, you can load in all the parameter configurations!

2. Defining Schema for Feature Types to be written

Back in the FeatureWriter dialog: 
A. Select 'Layer1' and remove the selected feature type definition by clicking on the minus (-) sign.

B. Add feature type by clicking on the plus (+) sign then selecting "Import from Dataset...". 

This will open Import Schema Definitions window
A. Set the Format as: Esri ArcGIS Portal Feature Service
B. Specify your organization's ArcGIS Portal URL for the Dataset.
C. Click on "Parameters...".

In the Esri ArcGIS Portal Feature Service Parameters dialog:
The Configuration for this section is the same as Part 1: Step 2 and Step 3. For this tutorial,

A. Click on the ellipsis (...) next to Feature Service.

In the Select From dialog: 
A. Specify your credentials for User and Password
B. Set Verify HTTPS Certificates: No
C. Select the parent folder of the Utility Network Data, then select the desired Utility Network Dataset, 
D. Click on "OK".

Back in the Esri ArcGIS Portal Feature Service Parameters dialog:
A. Click on the ellipsis next to Layers

In the Select Layers dialog:
A. Select the desired layers to write to

• • For this tutorial, we will be using all the layers, except tables and Utility Network System Tables, this includes: 
    • Service Territory
    • Structure Boundary
    • Structure Junction
    • Structure Line
    • Water Assembly
    • Water Device
    • Water Junction
    • Water Line
    • Water Subnet Line

B. Click on “OK” three times.

In the Select schema definitions to add dialog:
A. Click on "Select all".
B. Click on "OK".

Your FeatureWriter window should look similar to this, where the Feature Types are now populated: 

Finally, in the FeatureWriter dialog:
A. Click on "OK". 

3. Connect the Readers to the FeatureWriter Feature Types
Connect the corresponding layer from the reader to the FeatureWriter feature types. 

Your final workspace should look similar to this:

4. Save and Run the Workspace
Now, save and run the workspace. View the output data in Esri ArcGIS Pro to see if it was properly written to the portal feature service.

Troubleshooting

Cannot write out 19700101000000 for dates for ESRI ArcGIS Portal Feature Service Writer

Solution: AttributeFilter and DateTimeConverter; 19700101000001/Local Timezone
Currently, if you specify the value of 19700101000000 for date fields for attributes, when you check the date on ArcGIS Pro the value is 12/30/1899. There are two methods to resolve this problem. 

1. If Hours, Minutes, and Seconds are not important, use AttributeFilter transformer to select features with the value “19700101000000” for your date fields and using the DateTimeConverter transformer and select “%B %e, %Y” for the Output Format.
2. If specifying Hours, Minutes, and Seconds is important, you can use the value "19700101000001" or coordinate with the local timezone to represent 01/01/1970 (i.e., 19700101060000 represents 1/1/1970 at local midnight for the Central timezone).

Other FME and Esri ArcGIS Errors 

If you encounter FME and Esri ArcGIS, this documentation is a comprehensive resource containing various solutions for troubleshooting.

Data Attribution

Water Distribution Utility Network Foundation data is copyright 2024 Esri. Licensed under Apache-2.0. Source: ArcGIS Solutions.