Reading Utility Network Data from ArcGIS Portal Feature Service and Writing to File Geodatabase

Introduction

In this tutorial, we will explore how you can effectively write Utility Network data using FME, specifically by utilizing the File Geodatabase format for the writer to create a localized version of the UN data with domain values instead of coded value domain. While you can use also use FeatureWriter to write to a File Geodatabase, this tutorial will showcase the writer method. This tutorial will specifically demonstrate writing to a File Geodatabase, but the process of writing to an ArcSDE Geodatabase is very similar.

This tutorial assumes that you do not have an existing geodatabase to write to. We highly recommend using the most up-to-date File Geodatabase of the Utility Network data as the Template File Geodatabase to create a localized version. This approach allows better version control and ensures redundancy, which is crucial to maintaining the integrity of the Utility Network data. If you have an up-to-date File Geodatabase version of Utility Network data for Template File Geodatabase, some of the instructions and steps provided in this tutorial may not be applicable, as they are specific to creating a new geodatabase.

For an in-depth tutorial on writing to a File geodatabase or ArcSDE Geodatabase, please refer to this article: Tutorial: Geodatabase Transformation.
For a deeper understanding of the Esri File Geodatabase (File Geodb) Writer and the Esri ArcSDE Geodatabase (ArcSDE Geodb) Writer, consult their respective documentation.

Resolving Domain Values:
Although you can directly write Esri ArcGIS Portal Feature Service items or ArcSDE Geodatabase into a File Geodatabase, a Utility Network usually contains coded values for attributes associated with domains. If you write the data directly, your attributes will have the coded domain values instead of the actual domain values. To write the data with all the proper domain values, you must use the following transformers in your FME workspace:

1. AttributeRemover: This transformer is used to remove the original attribute column that contains the coded domain values.
2. AttributeRenamer: This transformer is used to resolve the domain values from the reader. It takes the coded domain values and replaces them with the corresponding actual domain values.
3. BulkAttributeRenamer: This transformer is used to rename the resolved attribute column (which contains the actual domain values) to the original attribute name. This step ensures that the attribute names in the output File Geodatabase match the original attribute names in the Utility Network.

The Esri ArcGIS Portal Feature Service Reader has some limitations in how it handles null/empty/missing attribute values, coded domains, and ranged domains

• Null/Empty/Missing Values and Coded Domains: If an attribute in the source Feature Service has a null, empty, or missing value, or if it uses a coded domain and the value does not correspond to any defined domain value, the reader will resolve that attribute as a missing value in the output.
• Ranged Domains: The reader does not resolve ranged domain values at all. Regardless of the actual value in the source data, if an attribute uses a ranged domain, the reader will output a missing value for that resolved attribute.

Requirements

• FME Form Workbench installed and licensed
• Esri ArcGIS Pro or ArcGIS Enterprise installed and licensed
• Basic understanding of Utility Network data structure and dataset, FME transformers, and FME and geodatabase
• Utility Network-configured ArcGIS Portal Feature Service item

Sources

• ReadPFS_WriteFGDB.fmw contains the completed workspace for this tutorial. Portal URL, Credentials, Feature Service, Layers/Feature Types, and Branch Versioning for the Reader and Dataset path location for Writer will need to be re-configured to make the workspace usable.

Step-by-Step Instructions

Part 1: Configuring Portal Feature Service Reader for File Geodb Writing

1. Add an Esri ArcGIS Portal Feature Service Reader
In a blank workspace, add a Reader.

In the reader parameters, set the following:
A. Set the Format as: Esri ArcGIS Portal Feature Service 
B. Specify your organization's ArcGIS Portal URL for the Dataset.
C. Click on "Parameters...".

In the Esri ArcGIS Portal Feature Service Parameters dialog: 
A. Click on the ellipsis next to Feature Service.

Tip: You can use Presets to retrieve the same configuration of parameters. Recommended parameters to be set before saving as a Preset for Esri ArcGIS Portal Feature Service Reader are: Portal URL, Authentication Type, User, Password, Feature Service, and optionally, Branch Versioning.

2. Edit Feature Service Parameter

In the Select From dialog: 
A. Specify your credentials for User and Password

• • Filling out the Credentials should give you access to the files. 
    • If you still don't have access, then follow step B,
    • If you do have access, you can skip step B.

B. Set Verify HTTPS Certificates: No

• • Filling out the credentials should give you access to the files 
  • Note: Verify HTTPS Certificate may need to be toggled to Yes, depending on your organization's protocols.

C. Select the desired dataset for Feature Service.
D. Select (presumably) “Admin” (next to the folder icon), then select the desired Utility Network Dataset. 

• • For this tutorial, “Water Distribution Utility Network” will be selected
  • If you cannot see anything in the area that is highlighted by the green outline in the screenshot, click on the refresh button as indicated by the blue arrow.

E. Click on “OK”.

Back in the Esri ArcGIS Portal Feature Service Parameters dialog: 
A. Click on the ellipsis next to Layers.

3. Edit Layers Parameter

In the Select Layers dialog:
A. Select the following layers:

• • Structure Boundary
  • Structure Junction
  • Structure Line
  • Water Assembly
  • Water Device
  • Water Junction
  • Water Line
  • Water Subnet Line

B. Click on "OK".

4. Enable Resolve Domains Parameter

Back in the parameters, enable the Resolve Domains. 
A. Enable Resolve Domains.

• • Important Note: Attributes will not be resolved if the original value is null or does not match any code in the domain. In such cases, FME will return an empty string for the attribute value.
  • Currently, the only way to obtain domain values for the “asset group” attribute without adding a reader for the lookup table is to expose the "arcgisonline_subtype_name" attribute in the reader
  • <Optional> enabling Branch Versioning.

B. Expand Schema Attributes, click on the ellipsis next to  Additional Attributes to Expose.

5. Select Additional Attributes to Expose 

In the Select ‘Additional Attributes to Expose’ Items dialog:
A. Select "arcgisonline_subtype_name".
B. Click on "OK".

Back in the Esri ArcGIS Portal Feature Service Parameters dialog 
A. Click on "OK" and “OK” again to finish adding the reader. 

Part 2: Domain Resolution Attribute Management 

General Attribute Management involves removing the original attribute column names of the resolved columns and renaming the resolved columns to the original attribute column name using the following transformers:

• AttributeRemover is used to remove all the original attribute columns.
• AttributeRenamer is used to rename “arcgisonline_subtype_name” to “ASSETGROUP”.
• BulkAttributeRenamer is used to rename all the resolved columns by removing “_resolved” suffix.

Example of Attribute Management for Water Device:

1. Add an AttributeRemover for Each Layer
Continuing in the same workspace as Part 1, add an AttributeRemover to the workspace canvas for each layer from the reader and connect an AttributeRemover to each layer (reader feature type).

In the AttributeRemover Parameters dialog:
A. Select the following attributes for each layer. 

2. Add an AttributeRenamer for Each Layer

In the AttributeRenamer Parameters dialog:
A. Add AttributeRenamer Transformer to your workspace canvas. 

a. Set “arcgisonline_subtype_name” for Input Attribute.
b. Set “ASSETGROUP” for Output Attribute.

3. Add a BulkAttributeRenamer for Each Layer

In the AttributeRenamer Parameters dialog:
A. Configure the BulkAttributeRenamer transformer for each layer to rename all the resolved columns by removing ‘_resolved’.

Repeat Part 2, Steps 1 to 3, for all the layers that you want to write to a File Geodatabase.

Part 3: Configuring Writer for File Geodatabase

1. Add an Esri Geodatabase Writer
Continuing in the same workspace as Part 2, add a writer to the canvas. 

In the parameters, set the following: 
A. Set the format as "Esri Geodatabase (File Geodb)".
B. Specify the file path and name you want to save to by clicking on the ellipsis (next to Dataset).
C. Set the Feature Class or Table Definition as Copy from Reader...
D. Click on "Parameters...".

When creating a localized File Geodatabase version of the Utility Network data, always reference the Feature Class Definition to the schema definition of the Utility Network data in ArcGIS Portal Feature Service.

In the Esri File Geodatabase (File Geodb) Parameters dialog:
A. Enable Overwrite Existing Geodatabase.
B. Expand Advanced and set "yes" for Preserve GlobalID.
C. Click on "OK" and “OK” again, to select feature type layers to write to.

2. Select Features Type Layers to Write to

In the Select Feature Type dialog:
A. Select the desired layers to write to, for this tutorial select “Select All”.

• • For this tutorial, feature type layers from the reader includes 
    • Service Territory
    • Structure Boundary
    • Structure Junction
    • Structure Line
    • Water Assembly
    • Water Device
    • Water Junction
    • Water Line
    • Water Subnet Line

B. Click on "OK".

Part 4: Creating Feature Classes within a Feature Dataset and Resolving Data Type Formatting Issues

To create feature classes within a feature dataset using FME, you need to modify the output feature class name by prefixing it with the desired feature dataset name followed by a '/' symbol.

• • In this case, we want to create the feature classes generated on the workspace canvas inside a feature dataset called "UtilityNetwork".
  • For a deeper understanding of creating feature datasets, consult the following documentation: Feature Datasets

1. Creating Feature Classes Within a Feature Dataset
Continuing in the same workspace as Part 3, the Feature Type parameters dialog should appear after clicking “OK“, in the previous Part. 

In the Feature Type dialog:
A. Add 'UtilityNetwork/' in front of each Feature Class or Table Name.

• • Example: If the original feature class name is "WaterDevice", modify it to "UtilityNetwork/WaterDevice".

Repeat for all the feature class writers that you want in the "UtilityNetwork" feature dataset.

2. Connect the Bulk Attribute to the Writer Feature Types
Connect the corresponding layer from the transformer to the writer. Make sure to disable all the layers that are not connected to the writer, as they are not needed for the current workflow.

If clicking on the arrow of the reader/transformer to writer is too strenuous, refer to this documentation using the "Feature Type Connections" view window to add connections.

3. Save and Run the Workspace
You will most likely encounter Translation Failed in the Translation Log due to the difference in the data type format for some attribute because we set Copy from Reader… for Feature Class or Table Definition, skip Step 4 if you have already have an up-to-date File Geodatabase version of the Utility Network data to use for Template File Geodatabase or if Translation did not fail.

Translation failed due to differences in the data type format for some attributes. We know that the data type format is affecting the translation because when you scroll above the red error text, you can see the reason why the translation failed.

• • In our case, the translation failed because "The invalid value 'Unknown' was supplied to the field 'adddetails' for the feature type 'WaterLine'".

4. Handling Data Type Formatting Issues

Follow the subsequent steps to troubleshoot and rectify this issue:
A. Specify the Max Features to Read and Max Features to Read per Feature Type option to 100.

• • This will limit the number of features read from each feature type, allowing for faster testing and debugging.
  • To do this, locate the Reader [portal] in the navigator, expand it, then expand Parameters, then expand Features to Read, and double-click on any of the options below Features to Read.

In the Edit '[Name for Reader]' dialog.
A. Specify 100 for Max Features to Read and Max Features to Read Per Feature Type.

B. On the FME workspace canvas, locate the feature type causing the issue and double-click on it to open its properties.
C. Navigate to the User Attributes tab in the feature type properties. 
D. Set Attribute Definition from Manual to Automatic 

• • Note: If File Geodatabase already exists, leave Attribute Definition as Manual 

• • Alternatively, you can manually find the attribute causing the error in the list of attributes.

a. Change the data type of the problematic attribute to a more flexible data type such as "blob" or "string".

• • This will allow FME to handle a wider range of values for that attribute.

Repeat steps 5B to 5D for any other error messages you encounter, adjusting the data type appropriately for each attribute causing the translation failure.

Once you have made the necessary data type changes, run the workspace again.

4. Save and Run the Workspace
The translation should now be successful, and the data should be processed without any data type-related errors. View the output data in Esri ArcGIS Pro to see if it was properly written to the portal feature service.

Troubleshooting 

Unable to resolve Asset Group directly from Reader

Solution: Exposing and Renaming Attribute:
Currently, the only way to obtain domain values for the asset group attribute without adding a reader for the lookup table is to expose the "arcgisonline_subtype_name" attribute in the reader. To handle this, you can use the AttributeRenamer transformer to rename the "arcgisonline_subtype_name" column as "assetgroup". This approach allows you to populate the asset group attribute with the correct domain values without the need for an additional lookup table reader.

To resolve for AssetGroup, refer to Part 1: Configuring Portal Feature Service Reader for File Geodb Writing Step 5 and Part 2: Domain Resolution Attribute Management Step 1 and 2.

Populating Null Values in Resolved Domain Columns

Solution: Custom Transformer for Populating Null Values:
When working with resolved domain columns (columns ending with '_resolved'), you may encounter situations where the resolved domain column has null values, even though the original column has a valid value. To address this issue, you can create a custom transformer that automatically populates the null values in the resolved domain columns with the corresponding values from the original columns.

The custom transformer iterates over all the attributes of each feature and identifies the columns ending with '_resolved'. For each '_resolved' column, it checks if the corresponding original column exists. If the '_resolved' column has a null or empty value; the transformer retrieves the value from the original column and populates the '_resolved' column with that value.

Other FME and Esri ArcGIS Errors 

If you encounter FME and Esri ArcGIS, this documentation is a comprehensive resource containing various solutions for troubleshooting.

Data Attribution

Water Distribution Utility Network Foundation data is copyright 2024 Esri. Licensed under Apache-2.0. Source: ArcGIS Solutions.