FME & CIM XML for Electric Utilities

Introduction

CIM  (Common Information Model) is an open standard developed by the electric power transmission and distribution electric utility industry. One of the CIM’s key objectives is to provide a way for application software, such as GIS, ADMS, SCADA, to exchange information about the configuration and status of electric utility networks.

The CIM defines a common set of electric data objects for both transmission and distribution. There are a number of IEC standards related to CIM and they are listed here. The CIM data model is extensive and complex and covers all aspects of electric transmission and distribution. Typically, the CIM XML data model is narrowed down with a ‘profile’ that limits the data model for a specific data exchange. For example, you might have a profile for transmission or distribution or even a profile that limits the exchange data to the facilities managed in a GIS. Ideally, an organization would adopt a single CIM profile that the majority of its applications can work with.  CIM is usually serialized as RDF XML. 

FME has always been able to read and write CIM XML data exchange documents. However, the introduction of the XSD-Driven XML reader & writer format to FME greatly simplifies working with CIM XML (and XML in general). To produce CIM XML in earlier versions of FME, users would have to use the XMLTemplater to create the CIM Classes and populate the appropriate attributes. This approach is summarized in some of the earlier presentations on CIM XML. The disadvantage of this approach meant that a change in the CIM ‘profile’ would require detailed changes to the XML Templates. With the advent of the XSD-Driven XML format, all you need is an XSD that represents the XML data model you need to create. The CIM Classes are then represented in FME as FME Feature Types, just like the majority of other FME formats. 
 

CIM XML ETL

As with all ETL (Extract, Transform & Load) workflows, there are three main steps. In the context of creating CIM XML, these are:

• Extract: reading the source data - usually the electric utility facilities data that is housed in the utility GIS. Examples are Esri Utility Network, Esri Geometric Network, GE Smallworld, Intergraph G/Tech. 
• Transform: map the source data model to the target (CIM) data model. The CIM target data model is usually constrained by a specific  ‘profile’. The profile might be specific for an application (i.e. OMS system) or domain (transmission vs. distribution). The source and target data models are often widely divergent, so the mapping of feature classes, attribute name, values, domains and classes might be detailed and complex. Some objects might not exist in the source data, so these have to be created during the ETL workflow, for example, CIM Terminals and ConnectivityNodes
• Load: serialize the CIM XML using the FME XSD-Driven XML writer

Depending on the source data, additional steps might be added:

• Source data cleanup - i.e. removing unused attribution
• Building topology - source data might not have all the network topology elements required, for example, connectivity nodes. Complex edges might need to be split. 
• Key Mapping: converting derived UUIDs into UUIDs that conform to the CIM profile in use to create persistent IDs.

Writing CIM XML Examples

Safe has created CIM XML writing examples for both the Esri Geometric Network data model and the Esri Utility Network data model. These can easily be restructured for other GIS data sources such as Hexagon G/Technology or GE Smallworld.
 

Esri Utility Network data model to CIM XML example

The example workflow (attached FME 2022.2) is based on EPRI IOP (2022) sample data but defined in an Esri Utility Network database. To gain access to Esri Utility Network datasets (i.e. a feeder), Esri has an export tool that generates a JSON document that represents the Utility Network data for the region of interest. The JSON exports can also be automated either using FME or the Esri Data Interoperability Extension. This JSON document of the UN data is the starting point for the FME conversion to CIM XML. For Esri Geometric Data Models, FME can read directly from the Esri Geodatabase. For this example, the output CIM XML is defined by a CIM profile developed during the EPRI IOP event. 

The workspace for Utility Network source data is divided into several sections that are indicated by bookmarks and annotations in the workspace:

• JSON Interpretation (bookmark “JSON processing”)
  • JSON is read as a text file and attribution is created. Attributes are supplied in two forms - fieldValues and networkAttributeValues. 
  • Bookmarks - JSON Cleanup. GlobalID cleanup - also annotated Schema Mapping: 
    • New JSON attributes must be exposed in this bookmark
    • Clean-up source data: remove unused attribution and set some fundamental attributes, i.e. GlobalID
• Create UN Feature Classes (bookmark “Geometry builders + UN Feature Classes”)
  • Esri UN feature classes are created for features, connectivity, and associations. 
  • Also creates geometry - which is not needed for CIM, but helps in troubleshooting.
• UN Connectivity & Associations: (bookmark “UN Connectivity”)
  • Joins Edges & connectivity objects
  • Joins UN assemblies and connectivity objects. 
  • Joins assemblies to devices
  • There may be duplication of connectivity records in the UN JSON, so these resolved by aggregating the connectivity records.
• Lookup Tables: (bookmark “Lookup tables”)
  • Domain Mapping from Excel spreadsheet lookup tables: Phases, Voltage, etc.
• Final Schema Mapping (bookmark “CIM XML & Schema Mapping”)
  • This bookmark normalizes all the Esri UN features into the CIM XML classes and does the schema mapping.  ‘Intelligent’ CIM mRIDs are created - see note below). 
  • Coordinate System, Locations, Position Points are all in on bookmark: “Location”
  • Terminals and ConnectivityNodes ("Terminals & ConnectivityNodes"" bookmark) are largely derived from the UN connectivity. There is pretty well a one-to-one equivalence between  Esri UN junctions and CIM connectivity nodes. So anywhere there is an existing Esri junction, those are mapped directly to the equivalent CIM Connectivity Node. Esri UN Junctions are usually used at junctions of conductors. In some cases, junctions are not required in Esri UN, for example, where a conductor connects to a device (i.e. switch). In these cases, FME's UN to CIM XML migration workspace will insert a new connectivity node at the conductor line end and then add the logic to connect the line end to the connectivity node to the device i.e.: 

• Special Cases & Supplementary Data: i.e. asset management is added for Transformer Info classes from a secondary data source (simulated in this workspace as an Excel spreadsheet). Usually, this would be read form an asset management database -  "Asset Database" bookmark
• Topology: topology comes from the UN connectivity classes which are used to create the CIM Terminals and ConnectivityNodes using the from / to connectivity - bookmark “Terminal & ConnectivityNode”.
• Boundary Classes: CIM XML supports boundary classes where different datasets can be joined into a model. In this example, the Feeder Breaker is in a separate CIM XML file (transmission) so a boundary node must be included to link to the Breaker ("Boundary Classes" bookmark).
• Writer Feature Types: This is where the XML is serialized. These need to be kept up-to-date with the CIM profile. Use Update Feature Types to keep attribution up-to-date. Instructions for adding a new XSD-Driven XML Writer are in the workspace.

Note: Derived ID’s
FME derives UUIDs from the original Esri UN GlobalId.
For example, ACLineSegment:

Original GlobalId: {2451835f-7ab0-49e0-9399-917b96c39e51}

Derived segment GlobalId = urn:uuid:2451835f-7ab0-49e0-9399-917b96c39e51_434_1000

Generating CIM XML XSD

The CIM XML workspace needs an XSD file to define the CIM RDF schema (equivalent to the RDF Schema, but NOT the same as the Enterprise Architect XML XSD). FME can generate an appropriate XSD file from a sample dataset using the CIM XML to XSD Creator workspace discussed below. 
 

Esri UN to CIM XML workflows

The attached zip file (in the Files section of this article) contains several FME workspaces (FME 2022.2) and sample data. These are summarized below.

Workspaces (in order they are typically be executed):

Esri UN JSON to CIM XML

EsriUN_to_CIMXML.fmw.  Workspace for reading Esri UN JSON and creating CIMXML file against a CIM XML profile
Inputs: 

• CIM XML Instance Set Dataset Name: <any text describing this data  extract> 
• Source Esri Utility Network JSON File: EsriUN_ExampleData.json. UN JSON export from Esri UN 

• Source Asset Info Excel File:

• Source TD Boundary Model XML File:

• Domain Lookup Tables (Schema Mapping): 

• CIM XML Application Schema (XSD): XSD for the current CIM profile: cim.xsd

Outputs:

• Destination CIM XML File:

Other CIM XML Workspace Parameters: 

• CIM XML Resource XML Name Space:  the CIM namespace that matches the profile being used. 
• XSD XML Document:  Adds the CIM XML namespace headers as well as the CIM custom : 

• RDF ID Prefix: urn:uuid: - prefix added to all uuid’s - currently under review by the CIM standards since creates an invalid XML ID)
• Output CIM XML Coordinate System: NAD83 / Illinois East (ftUS) [EPSG:3435]
• Include Point of Common Coupling: (these are the CommonCoupling objects sometime used to model meter connections) Usually No.

UUID Key Mapping

CIMXML_KeyMapping.fmw : FME create 'derived' rdfID's as described above. Some CIM XML applications will only accept a true UUID. The key mapping workspace creates a key mapping table in a SQLite database to create UniqueIDs for each object as well as preserving the original FME generated IDs. This means that subsequent exports of the same data (i.e. the same feeder) will result in persistent UUIDs for the target application. Make a copy of the "KeyMapping - Template.sqlite" database file and then load the copy with new key maps.

CIM XML Validation

CIMXMLFile_Validator.fmw - CIM XML Validation workspace. The validation workspace compliments validation by other applications such as CIMSpy. Validation results output by this workspace: as an Excel spreadsheet contains several  tabs:

• MissingReference: shows if any reference errors are present (generally tied to if the Feeder is not present in the XML file). 
• Reference Detail:  - shows which objects are affected by the above Missing Reference. 
• UniqueObjectClassStats:  Similar to the CIMSPY Object Summary, feature counts of each object. These should match up against the CIM SPY Object Summary. 
• Schema Validation tabs

CIM XSD Creator

cim_sample_to_xsds_with rdf_about.fmw: This workspace will generate the CIM.xsd from a sample CIM XML dataset.

Changing the CIM XML profile

This example UN JSON to CIM XML workflow uses a specific CIM XML profile (data model or schema) developed during the 2022 EPRI IOP project for electric distribution. Ideally, within an organization a single CIM XML profile can be shared across all target applications. But, different organizations or target applications may have different CIM XML data requirements and hence different CIM XML profiles. If the CIM XML profile changes then the UN to CIM XML workflow needs to change. 

• If the data model of the source Esri UN changes then additional attribution and feature classes may be represented in the Esri UN JSON export. Check that any new, required, attributes are exposed in the "JSON processing" bookmark
• Update FME cim.xsd 
  • Use a sample of CIM XML data for the required profile to generate a new CIM XML cim.xsd.
  • Make sure you copy the new XSDs to the CIM XML workspace XSD folder
• Update the FME XSD-Driven XML writer feature types in CIM XML workspace
  • Point FME at the revised cim.xsd file in the XSD-Driven XMLwriter
  • Check all attribute values mapped correctly in the appropriate AttributeManagers for each feature class. 
• Re-run workspace
  • Make sure you update all the parameters - particularly the new cim.xsd location
• FME Validation: run the FME CIM XML Validation workspace. This checks for:
  • Summary class count statistics (should match what otters import and the Jun validator)
  • Duplicate primary keys (mRID / rdf:about)
  • Missing references (all rd:resource’s should link to an rdf:about). Exception is the AdditionalEquipmentContainer which is an external reference to the Feeder Class in the TD)
• Key Mapping
  • Run the key mapping workspace
  • Either start from scratch (Truncate option) or update the key mapping if new data is added
• Validate using GMDMValidator available from EPRI or use CIMSpy

Reading CIM XML

CIM XML is usually serialized as RDF XML, which is a very flat, highly normalized XML data structure - not too dissimilar to a normalized relational database. This makes it very straightforward to read CIM XML. In FME, you can use the XML reader to read back the CIM XML and use an  InLineQuerier transformer to join together the necessary CIM XML classes that can then be mapped to the target output schema.