Synchronizing ArcGIS Utility Network Data

Introduction 

Maintaining the accuracy, consistency, and accessibility of ArcGIS Utility Network (UN) data across desktop, web-based, and mobile GIS applications is crucial for effective utility management. Synchronization in this context refers to the process of ensuring that all edits made by multiple users are accurately integrated into the enterprise geodatabase, maintaining data integrity and consistency across all platforms accessing the UN data.

In the context of geodatabase versioning, ArcGIS UN datasets typically utilize branch versioning, which leverages a services-based architecture through ArcGIS Portal Feature Services. This architecture allows editors to work within their own isolated named versions and make edits via web feature layers (feature services).

The following diagram illustrates a general branch versioning framework:

Refer to this Esri article for more information on Branch Versioning and this Esri article for an Overview of Versioning

Overview of Utility Network Data Synchronization

A basic synchronization workflow involving complex datasets, such as UN datasets, can be achieved using FME to manage data transformations within the branch versioning and traditional versioning framework. 

While the 'Default' or parent version of the UN dataset is stored in an enterprise geodatabase and accessed via ArcGIS Portal Feature Service, editors typically create named versions derived from the 'Default' version for data modification. Editors connect to the UN using web feature layers (feature services) and work within their own isolated named versions. After completing their edits, they can reconcile and post their changes back to the 'Default' version. 

Synchronization of UN data in a branched versioning environment occurs through the reconcile and post operations within the enterprise geodatabase. This process ensures that all edits are made in the named versions are accurately merged into the 'Default' version, maintaining data integrity and consistency. 

Network Topology 

Every UN data have network topological rules that enforces connectivity and information related to features. While network topology must be enabled to access ArcGIS Utility Network's advanced analytical capabilities, disabling network topology is recommended when modifying large amounts of data or adding rules or assigning associations to features. When network topology is enabled, all of the modified features must be validated in order to use the advanced analytical capabilities of ArcGIS Utility Network. 

Refer to this Esri article for more information on Network Topology and Validating Network Topology

Brief Information on this Tutorial

Deviating from the typical synchronization process, this tutorial will demonstrate FME capability of synchronization involving both traditional and branched versions. Specifically, synchronizing a traditionally versioned dataset, where the Utility Network is not enabled, to a named branched version of the UN dataset and  'Default' version, where Utility Network is enabled. Although Esri recommends disabling the network topology when processing large data, this tutorial assumes that the network topology will remain enabled throughout the process.

Due to this deviation, the workflow of this tutorial will be following this diagram:

As a result, two parts will be involved in the UN data synchronization. To simply describe the two parts involved in this workflow,

1. First part synchronizes traditionally versioned dataset with the named branched version
2. Second part synchronizes the named branched version to the 'Default' version

We recommend separating these two parts into two workspaces due to the need to run Part 1 synchronization first, followed by data validation (network topology) before executing Part 2 synchronization. However, for this demonstration, we will use a single workspace containing both parts.

Requirements

• FME Form Workbench installed and licensed
• Esri ArcGIS Pro or ArcGIS Enterprise installed and licensed
• Basic understanding of Utility Network data structure and dataset, FME transformers, and FME and geodatabase
• Utility Network-configured ArcGIS Portal Feature Service item
• Familiarity with Branch Versioning and Versioning
• Following FME & Esri ArcGIS Utility Network Tutorials:
  1.  Reading Utility Network Data from ArcGIS Portal Feature Service and Writing to ArcGIS Portal Feature Service
  2. Reading Utility Network Data from ArcGIS Portal Feature Service and Writing to File Geodatabase
  3. Retrieving Updated/ Inserted/ Deleted Features in ArcGIS Utility Network Data
  4. Updating/ Inserting/ Deleting Features in ArcGIS Utility Network Data

Sources

• Synchronization_UN_data.fmw contains the completed workspace for this tutorial. Portal URL, Credentials, Feature Service, Layers/Feature Types, and Branch Versioning for the Portal Feature Service Readers and the FeatureWriters will need to be re-configured to make the workspace usable.

Step-by-step Instructions

Part 1: Simple Synchronization - Traditional Version to Branched Version

First workspace synchronizes a traditionally versioned dataset, in this case File Geodatabase is used as a placeholder, to a branched version of UN dataset.

The following screenshot demonstrates a synchronization process from File Geodatabase to Portal Feature Service for a single feature type. Specifically, synchronizing the Editor's/Technician's version of File Geodatabase to the 'QA' version stored in the Portal Feature Service for Water Device. 

As seen in the screenshot, synchronization from File Geodatabase to Portal Feature Service consists of five sections. These five sections are the Readers, Attribute Standardization, ChangeDetector, Attribute Rectification, and the Writer. 

Readers

In the first section of this synchronization process, two readers are present: a Esri ArcGIS Portal Feature Service Reader and a File Geodatabase Reader. The Portal Feature Service reader retrieves the 'DEFAULT' version of a feature type and the File Geodatabase reader retrieves a traditionally versioned feature type, in this case Water Device feature type.

Attribute Standardization, ChangeDetector, and Attribute Rectification

In the Attribute Standardization section, the BulkAttributeRenamer and the StringPadder transformers standardize the data from the two readers to increase the accuracy of the ChangeDetector. Specifically, the BulkAttributeRenamer is used to standardize the attribute names—in this case, converting them all to lowercase— and the StringPadder is used to standardize the date values for the File Geodatabase data to match the same length and width of the date values in the Portal Feature Service.

In the ChangeDetector section, the ChangeDetector is used between the original and revised data to identify which features have been updated, inserted, or deleted. In the configuration for the ChangeDetector, the attribute named 'SUPPORTINGSUBNETWORKNAME' from the 'Default' version from the Portal Feature Service is excluded because this is a topology layer that is not present in the Revised version of the data from the File Geodatabase.

In the Attribute Rectification section, the AttributeManager, TestFilter, and another AttributeManager are used to rectify the features to match the destination's schema definition. This means that all the attribute names that were converted to lowercase must be rectified, which the first AttributeManager corrects. The TestFilter is used to delineate the features that were updated, inserted, or deleted. Then, the second AttributeManager is applied to the features coming out from the update output port to remove the 'ASSETGROUP' and 'ASSETTYPE' attributes. 'ASSETGROUP' and 'ASSETTYPE' were removed because due to network topology preventing modifications to these two attributes.

Writer

In the last section, all the data that has been updated, inserted, or deleted will be written to the 'QA' version using a FeatureWriter transformer. This individual transformer can write multiple feature types within the same destination. While in the previous tutorial series we predominantly used specific writers that separated per feature type selected for writing data, this approach aggregates all the feature types into a single transformer. This transformer can be used and further configured to generate a report that can later configured to be used to pinpoint when the synchronization has occurred for rectifying data if there are issues with the synchronization process.

Since this tutorial assumes you are not disabling the network topology for the UN data, all the modified features in the 'QA' version must be validated before running the second workspace, Part 2.

Part 2: Simple Synchronizing Branched Version to 'Default' Version

Second Workspace  synchronizes branched versioned UN data to the 'Default' version. 

The following screenshot demonstrates a synchronization process from Portal Feature Service to Portal Feature Service for a single feature type. Specifically, synchronizing the 'QA' version stored in the Portal Feature Service to the 'Default' version for Water Device. 

As seen in the screenshot above, synchronization from Portal Feature Service to File Geodatabase consists of four sections. These four sections are the Readers, ChangeDetector, Attribute Rectification, and the Writers.

Readers 

In the first section of this synchronization process, two Esri ArcGIS Portal Feature Service Readers are present: One retrieves the 'Default' version of a feature type and other one retrieves the 'QA' version, in this case Water Device feature type.  

ChangeDetector and Attribute Rectification

In the ChangeDetector section, the ChangeDetector is used between the original and revised data to identify which features have been updated, inserted, or deleted. In the parameter configuration for the ChangeDetector, all of the attributes are included to compare the 'QA' version to the 'Default' version. 

In the Attribute Rectification section, first AttributeManager removes 'ASSETGROUP' and 'ASSETTYPE' attributes for features that are updated. As stated earlier, network topology prevents modifying these two attributes.

Writer

In the last section, the FeatureWriter transformer is used to reconcile and post all the updated, inserted, and deleted features to the 'Default' version. 

Data Attribution

Water Distribution Utility Network Foundation data is copyright 2024 Esri. Licensed under Apache-2.0. Source: ArcGIS Solutions.