Search

FME & Esri Utility Network Migration Training

Liz Sanderson
Liz Sanderson
  • Updated

Introduction

Welcome to the FME Form for Esri’s ArcGIS Pro Utility Network Migration Training Resource Center. FME Workspaces are available to help migrate your Esri ArcGIS Geodatabase to the new ArcGIS Pro Utility Network (UN) data model. This is a self-directed resource for understanding the FME Utility Network Migration workspaces. An overview of the ArcGIS Pro Utility Network can be found here.

This self-directed resource has three parts:

  • Part 1: FME Workbench Basics. What you need to know about FME Workbench and Data Inspector to create and modify FME Workspaces used for Utility Network migration.
  • Part 2: Key Transformers. Understand the key transformers used in the Utility Network migration workspaces.
  • Part 3: Utility Network Migration FME Workspaces. Review of the Utility Network migration workspaces and how to modify them to meet your needs.

 

Recording

The course recording is available for free at the FME Academy Training Archive, corresponding to the parts of the course listed below.

 

Before you start

Prerequisites

Downloads

 

Tutorials

Part 1: FME Workbench Basics

Introduction to FME: In this part of the training, you’ll learn the basic operations of FME Workbench (or in the Interoperability Extension) that you need to know so that you can edit, run and test the migration workspaces. If you are already an advanced user of FME you can skip Part 1.

Because this course was developed on what was formerly known as FME Desktop (now called "FME Form"), you may still see references to "FME Desktop" throughout the course material.

See the tutorial FME Desktop Data Integration.

If you require additional resources to become more familiar with FME, please find additional training courses here.

The migration workspaces also use the following functionality that you need to be familiar with:

 

Part 2: Key Transformers

In the following section, some of the key transformers that you will encounter throughout the Migration Workspaces are listed, along with a description of the Transformer’s function. Please refer to our documentation found here or our Knowledge Centre found here if you require additional resources.

Attribute Handling

  • AttributeManager/AttributeCreator
    • The Attribute Manager and AttributeCreator are used for renaming, removing and setting attribute values. The AttributeCreator is also used to set the values using if-then conditionals.
    • Advanced logic/conditionals are supported in the TestFilter, AttributeCreator and the AttributeManager. Working with Conditionals example.

Schema Mapping

  • Database Joiner
    • The DatabaseJoiner is used to join attributes from an external feature to a feature already in the workspace. The DatabaseJoiner is able to query an external table and retrieve attributes associated with that feature.
  • FeatureMerger/FeatureJoiner
    • The FeatureMerger is used to merge two features that already exist in the workspace together. The transformer requires two inputs, a Requestor, and a Supplier. The Requestor is the feature that will receive the new attributes and/or geometry, while the Supplier will provide the attribute and/geometry to be merged.
  • SchemaMapper
    • The SchemaMapper is used to convert the existing schema of a feature into a new structure through the mappings defined in an external lookup table. The lookups are defined by a series of conditions/filters, and a series of actions that will be performed when those conditions are met. This will allow for new Attribute creation, assignment, and renaming.
  • AttributeValueMapper
    • The AttributeValueMapper is used to compare attribute values to a lookup table, and assigning a new value where a match is found. As such, this can be used to map a subtype or domain. The matched value can be written to a new attribute, or overwrite an existing attribute. This can be used as an alternative to the SchemaMapper, however, multiple AttributeValueMappers can often be replaced with a single SchemaMapper.

Geometry Handling

  • PointOnLineOver layer
    • The PointOnLineOverlayer performs a spatial join between a point and the line it is connected to. Each line will be split at the closest place to any point(within the specified tolerance) - attributes can then be shared between the two features. Attributes can be shared between the Point and Lines through the creation of a List for the desired Output(Point attributes on the Line feature or Line attributes on the Point feature).
    • The SpatialRelator, TopologyBuilder, and the NeighborFinder transformers are also used to create spatial joins in the migration workspaces
  • AnchoredSnapper
    • The AnchoredSnapper will perform a snapping operation on features that match the input specification if they are within the specified tolerance. This is primarily used as a cleaning operation, or to ensure that features are correctly snapped to a line. This transformer is similar to the Snapper transformer, however, the AnchoredSnapper uses Anchor features that are considered accurate and thus will not move during the translation.
  • Clipper
    • The Clipper is used to perform a geometric clipping operation. A Spatial Join can be performed to share attributes between the Clippee and the Clipper features.

Advanced Transformers

  • List Transformers.
    • Several list handling transformers are used in the migration workspaces: ListHistogrammer, ListSorter, ListElementCounter, and ListSearcher
  • VariableSetter/ VariableRetriever
    • The VariableSetter is used to set a Variable to the Specified Value, this variable can later be read back in using the VariableRetriever.
    • The VariableRetriever will read the specified Variable (set from the VariableSetter) and place the value into the specified attribute.

 

Part 3: UN Migration Workspaces

Links to the domain specific (gas, and water) Utility Network Migration Tools can be found here. This will direct you to the Esri Utility Network Migration Tools where you can download the resources you need for the migration from ArcGIS to the Utility Network. This includes:

  • Documentation
  • Domain-specific migration workspaces and related schema mapping spreadsheets
  • Sample data

Refer to the PDF documentation for your domain-specific migration tools located within the Documentation directory of your downloaded Migration Tool package such as "Gas Utility Network Migration Tools Documentation" or "Water Utility Network Migration Tools Documentation".

 

Planning

There are a couple of important questions to ask yourself, before starting your migration.

  • Is your source ArcGIS data ready to be migrated?
  • How are you going to perform this migration? Are you going to migrate circuit by circuit, or as a single Bulk translation?
  • How will you validate the migrated data?
  • Have you allocated the appropriate resources for migration and validation?

Plan a trial migration: a recommended approach is to take a single circuit, or pressure zone then validate, migrate, load, and assess the results. Determine if the current configuration matches your desired or expected results.

 

Pre-Migration

  • Deploy ArcGIS Configuration(Asset Package located within ArcGIS Pro Project).
  • Perform a data quality check on your existing ArcGIS Geodatabase using the Esri Data Reviewer.
  • ArcFM - if using ArcFM be sure to drop the ArcFM Objects prior to your migration.
  • Schema mapping: modify the schema mapping spreadsheet to match your source data and any changes you need to make to the Utility Network data model.
  •  

Review the Key Input & Output Datasets

  • Input datasets
    • Source Geodatabase(File or ArcSDE)
    • Schema Mapping Spreadsheet (Microsoft Excel)
      • Asset Group/Asset Type Assignment
      • Domain Mapping
      • Attribute Mapping
    • Assembly Builder XML Template (Basic Representations Only)
    • Asset Package Geodatabase Template
  • Output datasets
    • File Geodatabase Asset Package
    • Error Geodatabase
      • Redline File Geodatabase provided with error features

Simple Migration

This section summarizes the steps to modify the migration workspace. For full details, please refer to the migration documentation provided with the Migration Tools.

Understanding the Migration Workspace

The Simple Migration Workspaces are divided into several fundamental sections that are indicated by bookmarks in the workspace:

  • Reader Feature Types
  • Schema Mapping
    • Asset Group and Asset Type Assignment
    • Domain Mapping
    • Attribute Mapping
  • Associations, i.e.:
    • Pump Station containment
    • Casing containment
  • Special Cases
  • Validation and Reporting
  • Writer Feature Types

Customizing the Migration Workspace

You will need to customize the migration workspace to suite your specific source data and any customization you make of the asset package. the following sections outline how to modify the workspaces.

  • Update Schema Mapper Excel File
    See Schema Mapper Spreadsheet section of the Domain specific documentation.
    • Update the Asset Group/Asset Type Assignment Sheet
    • Update the Domain/Subtype Mapping Sheet
    • Update the Attribute Mapping Sheet

NOTE: Attributes are Case Sensitive!!

  • Update the Geodatabase Reader
    See Update Source Feature types section of the Domain specific documentation.
    • Add a new Esri Geodatabase Reader (ArcSDE Geodb or File Geodb)
    • Import Feature Classes from Source ArcGIS Geometric Network Geodatabase
    • Change the SourceDataset_GEODATABASE_FILE parameter to the Source Geodatabase
    • If any additional Feature Classes need to be added, use the Import/Update Feature Type function.
  • Workspace modification
    • Prepare the Asset Package via Change Asset Package Spatial Reference tool
      See the Asset Package section of the Domain specific documentation.
    • Asset Package Updates (New Domains or Attributes added - clarify if this is done via Asset Package, or done via the UN and then export Asset Package!)
    • Update Writer Feature Types (if changes were made to the default Asset Package, Writer Feature Types will be required to be updated to import the updated schema)
      See Update Target Feature Types section in the Domain specific documentation.
    • Update Writer Geodatabase Template with updated Asset Package.
  • Workspace Results
    • Asset Package File Geodatabase
      • Feature Classes (Devices, Lines, Assemblies, etc.)
      • C_Associations Table
  • Error Report
    • Redline Geodatabase
      The Redline Geodatabase will write out features that errored out through the translation, this will allow you to visualize the problem feature and determine why it failed. The Redline Geodatabase may need to be altered to write out your desired attributes based on your mappings.
      • Feature Class(Point, Line, Polygon)
      • Subtype
      • Source ObjectID
      • GlobalID
      • Error Message
      • Error Transformer(currently only Electric workspace)
    • Excel File
      The Excel file generates a similar report to the Redline Geodatabase, without the spatial component. This can be used to compare against your Schema Mapping Spreadsheet to determine if there is a problem with one of your mappings. The Excel file may need to be altered to write out your desired attributes based on your mappings.
  • More Debugging Tips n' Tricks
    • Reducing the amount of data (testing specific features)
    • Using Feature Caching
    • Using Data Inspector

Post-migration

After a successful translation to the Target Asset Package, follow the remaining steps as outlined in the Domain specific documentation provided by Esri.

Note:Using {} - GUID vs GlobalID

  • FME is unable to write GlobalIDs, due to the process of enabling GlobalIDs on a Feature Class. As such, FME will write to a GUID field in the Utility Network Asset Package. Esri has tools designed to migrate these GUIDs to a proper GlobalID data type

 

Course Materials & Resources

Course materials including example workspaces, example datasets and slides are here. The materials contain:

  • Simple Migration: examples using Water migration workspaces & sample data
  • Basic Migration: examples using Electric migration workspaces, sample data and some guidance slides
  • Draft Documentation for Electric Migration

Please note that the electric migration workspaces used in these examples are 'beta' release only and should not be used in a production environment.

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.