BC MoF Electronic Submission Framework (ESF) Update

Introduction

Here are some technical notes on working with the new ESF XSD reader / writer. The ESF XSD Reader / Writer was recently updated to comply with the BC Ministry of Forests updates to the ESF regulatory framework and accompanying application schemas. The reason the reader / writer was renamed to ESF XSD is that the new reader / writer now is application schema driven and is based on FME's GML core. 

The workspaces attached to this article include basic examples as to how to set up the ESF reader and writer including how to set the ids and parent ids to control the object nesting and relationships for various submission types.

 

Installation Considerations:

It is best if you just install FME to a new directory, say FME2021.0, and leave your older version intact.

Any FME 2021 build > 21276 should have the ESF new reader / writer, associated help doc and latest required fixes.

We have left the old ESF reader / writer, but the description has had the term 'deprecated' added to make it clear that the ESF XSD is the new one.
So the new one is called "BC MoF Electronic Submission Framework - ESF (XSD-Driven)" 
and has the short name: ESF_XSD

The old one has been renamed to:
BC MoF Electronic Submission Framework - ESF (deprecated)
and retains the short name: ESF
 

Workspace Upgrade Considerations:

Note that any old workspace should run as before. You will need to add the new reader or writer to any existing workspace in order to support the new schemas. Since it is a new format you will need a regular FME license to have it available on your formats list. You can contact sales to get some evaluation licenses for the interim so you can start testing.

When upgrading old workspaces to use the new reader / writer: For previously available submission types (FTA etc), there are some feature type definition schema changes due to the schema updates, so you will need check to make sure the new writer works properly with older workspaces, make adjustments as needed, and test to make sure the output is as expected. Also, make sure you make a backup of any workspace before you begin the update process. Once you start upgrading, you will not be able to run that particular workspace on older versions of FME.

For the new ESF XSD Writer, you no longer need template workspaces the way you did with the old ESF writer. The schema is scanned directly from the ESF GML application schemas.

To add a new ESF XSD writer follow these steps:

1. Add Writer - Select ESF
2. Edit Add Writer parameters and select submission type, click OK
3. The Import Feature Types dialog will appear. Click on Parameters to ensure the correct submission type is still selected and then click OK.
4. A dialog will appear with the list of available feature types for that submission. Select the desired output feature types and then click OK.

Step 2: Select submission type


Step 4: Select desired destination feature types

It is important to note that the Writer's feature types are defined by a fixed schema and so need to be imported from the Writer. If you attempt to modify the destination feature type names or fields then the output will no longer be valid.

Note that the writer has built in validation. This means that the output is validated against the underlying ESF gml application schemas (XSDs) using the open source Apache Xerces library. So this is a comprehensive schema validation, but of course does not include any business rule validation such as what users typically implement in their FME workspaces. But it does provide an extra check to help ensure that the output is valid. Note that the writer validation serves the same function as the XMLValidator transformer in schema mode.

We are happy to help you with any technical questions that you have. Feel free to send your workspace and sample source data to Safe Support if you have a particular problem with the upgrade that you need assistance with.

Troubleshooting

Feature Collection error:

You may encounter an error something like the following:

XML Validation: Error in 'FTA_area_101.xml' on line 2, column 500: 'no declaration found for element 'gml:FeatureCollection''

This is most likely caused by bad or missing gml_parent_id's. This error will occur if you are not writing the ESFSubmission root feature or not setting the gml_id gml_parent_ids correctly. Every feature should have its parent feature set as its gml_parent_id according to the parent's gml_id. The gml_ids dont actually appear in the output gml but are used by the writer to control the nesting. Note that the use of gml_id and gml_parent_id in the ESF_XSD writer replaces the use of esf_harvest_application_id, esf_road_tenure_application_id etc
in the old ESF writer.

Another related diagnostic approach. If you open the output GML in an EXL editor, the root of your gml doc should start with:
<?xml version="1.0" encoding="utf-8"?>
<esf:ESFSubmission xmlns:esf="http://www.for.gov.bc.ca/schema/esf" 
...

If it starts with <gml:FeatureCollection..." 
this is most likely because you have one or more 'orphaned features' which means features that do not have a valid parent. The workspaces attached to this article include examples as to how to set the ids and parent ids for various submission types.


Debugging multiple validation errors:
We recommend using a separate submission type per workspace.


Feature Types Not Written:

If you see a warning such as 
<ESF_XSD_1 Writer> - The following feature types were not written, they did not match any feature types from the GML Application Schema: 'ESFSubmission_1, OpeningDefinition_1, Opening_1, ResultsSubmission_1, Standards_1, Tenure_1'

This it looks like the feature types were not imported from the writer, or that perhaps they were renamed. To write out valid ESF GML you need to use the predefined fixed schema feature types from the writer such as those in the attached workspace, not create your own.

Multi-part Geometry

For multi-area polygons, the geometry has to have a name: extentOf. This is an FME ESF XML requirement. This is ONLY required for multi-part geometries, but it won't hurt on simple areas.

Only the 'top' geometry should be named. So when you inspect your data in FME Data Inspector you should see something like: 

You can set the geometry name in the GeometryPropertySetter. By default the GeometryPropertySetter sets geometry names on all the nested geometries. So you need to modify the geometry naming as shown below:

Attached is example workspace (GeometryPropertySetterExample.fmw) that illustrates how to configure the GeometryPropertySetter for ESF XML.

References:

For more on the ESF XSD reader / writer see the help docs at:
https://docs.safe.com/fme/2021.0/html/FME_Desktop_Documentation/FME_ReadersWriters/esf_xsd/esf_xsd.htm

Also, because the ESF XSD reader and writer is built on FME's GML core, it may be helpful to refer to the GML reader / writer docs as well: https://docs.safe.com/fme/2021.0/html/FME_Desktop_Documentation/FME_ReadersWriters/gml/gml.htm