Managing ArcGIS Online Metadata in FME

FME Version

Files

Part 2 Validating Metadata Attributes.fmw
- 30 KB
- Download

Introduction

Available in Esri ArcGISOnline Connector Package v2.5.0+, the ArcGISOnlineConnector can be used to manage metadata in ArcGIS Online. The Get Details Action retrieves metadata from a feature service, and the Set Details Action can replace some existing feature service metadata.

Users may want to use FME to validate their metadata posted to ArcGIS Online or ArcGIS Portal to ensure it meets the requirements set by their organization. Additionally, FME can also be used to send notifications to the data steward if the metadata is not complete or create a validation report.

Requirements

Access to FME Form version 2024.0+
An ArcGIS Online Account or ArcGIS Portal
A Feature Service uploaded to ArcGIS Online that contains metadata

Step-by-Step Instructions

Part 1: Retrieving Metadata from ArcGIS Online

1. Open FME Workbench

In FME Workbench, start a blank workspace.

2. Add a Creator

Search for the Creator transformer and place it on the canvas. The Creator will initiate the workflow.

3. Get the ArcGISOnline Metadata

From there, add an ArcGISOnlineConnector to the canvas and connect it to the Creator transformer.

Note: To use the Get Details and Set Details actions, you will need to upgrade to Esri ArcGISOnline Connector Package version 2.5.0+ and use a minimum build of FME 2024.0.

If you see a yellow bar indicating a new package version is available, click to update to the latest version (2.5.0+).

To configure the parameters of the ArcGISOnlineConnector, click on the exclamation point and select the ArcGIS Connection section to view your options.

If you have previously set up a connection to ArcGIS Online or Portal, you can select it from the options. If you need help creating a connection to Portal, please refer to the guide on How to Create an ArcGIS Enterprise Portal Web Connection (OAuth 2.0).

If you need to create a new ArcGISOnline connection, select Add Web Connection… and create a name for your connection. Then select Authenticate, enter your ArcGIS Online username and password, and click Sign In.

Next, click the drop-down next to Action and select Get Details. Then, click on the ellipsis button next to Path to see a list of folders you have in ArcGIS Online. Double-click a folder that you know contains a Feature Service, click on a Feature Service, and click OK.

Click on the ellipsis button next to Attributes to Add and click Select All, or you can simply just select the attributes you are interested in.

Please note the folders in this screenshot will not match the folders you own.

Leave all other options as the default. Click OK to close the dialog.

4. Inspect the Output from ArcGISOnlineConnector

With Feature Caching enabled, click Run to run the workspace. Once the workspace has finished, click the green eye on the Output of the ArcGISOnlineConnector to inspect the output. This will display the attributes and metadata in the Visual Preview window. The _metadata_xml attribute contains the metadata as XML.

5. Write Out the Metadata Attributes

To write out metadata attributes, you can use a writer such as Microsoft Excel, CSV, etc. The files can then be sent to the data stewards for manual validation.

Note: The ArcGISOnline Connector will only allow one feature service to be read in at a time, so if you would like to add multiple feature services, another ArcGISOnlineConnector is required. Repeat steps 3 and 4 to set up an additional ArcGISOnlineConnector.

Run the workspace and write all of your files out to a folder.

Save and exit the workspace.

Part 2: Validating your Metadata Attributes

Part 2 of this tutorial covers validating your Metadata attributes from the ArcGISOnlineConnector transformer and will use the workspace built in Part 1.

Note: The following screenshots are for demonstration purposes only, and will not match your data. Please follow along using your dataset.

1. Examine Metadata in ArcGIS Online

In your own ArcGIS Online account, select a Feature Service, and select Metadata.

Here you can compare the metadata values in the ArcGIS Online interface to the attributes in FME from the workspace in Part 1.

2. Flatten XML

Back in FME Workbench, we want to expose additional attributes that aren’t exposed in the ArcGISOnlineConnector.

First, ensure that your feature service in ArcGIS Online has a value for the created date and published date. You can view or edit these attributes in the Metadata Editor in ArcGIS Online under citation and citation dates.

First, we need to flatten the XML file from the ArcGISOnlineConnector using an XMLFlattener transformer.

Place an XMLFlattener on the canvas and attach it to the output port of the ArcGISOnlineConnector. For the XML Source Type, select Attribute with XML document, and for the XML Attribute, select the _metadata_xml attribute.

Lastly, type an asterisk * in the Elements to Match.

Then click OK.

Once the XML Flattener is added to the canvas, run the workspace with Feature Caching enabled. This will ensure that you can easily select the cached values in the next step.

Note: While using Feature Caching can allow you to easily select cached values in the next step if you notice that the metadata is not updating at any point during this exercise, try turning Feature Caching off.

3. Expose Flattened Attributes

Next, add an AttributeExposer to the canvas, place it after the XMLFlattener and expose the following attributes:

createDate
pubDate

4. Run the Workspace and View the Results

Run the workspace and view the output of the AttributeExposer.

5. Add Validation Rules

Attributes can be validated using the AttributeValidator. Here, we will do some basic validation.

Attach the AttributeValidator to the Output port of the AttributeExposer.

In the parameters, under Attributes to Validate, select createDate and pubDate. Then, under Validation Rule select Has a Value. Click OK.

Run the workspace again. Here, you should see the file has passed the validation steps.

6. Change Metadata to Validate

Now, try to remove the metadata for the pubDate in ArcGIS Online so that it will not pass the new validation rules. To do this, go to ArcGIS Online and select the Metadata tab.

In the Metadata editor, select the All Metadata tab. Under Citation, select Citation Dates and remove the Published date.

Then click Save at the top right and you should see a popup at the bottom of the screen letting you know that the Metadata has updated successfully.

Back in FME Workbench, rerun the entire workspace. You will see that the AttributeValidator failed, the pubDate is missing, and there is a validation message.

These validation messages can be output to a file, and a report can be generated by adding an Excel or CSV writer. This report can be sent to data stewards.

More details on how to enhance Excel files or use a formatting template can be found in this article. This workspace can be saved and will not be used in Part 3.

Part 3: Updating Metadata in ArcGIS Online

1. Open FME Workbench

In a blank workspace, add a Creator transformer, then add an AttributeCreator. In the AttributeCreator parameters, create a new attribute called Tag and set the value to VancouverParks. This attribute will be used to update the tags in ArcGISOnline.

2. Set the ArcGISOnline Metadata Details

From there, add an ArcGISOnlineConnector to the canvas and connect it to the AttributeCreator transformer.

Note: To use the Get Details and Set Details actions, you will need to upgrade to Esri ArcGISOnline Connector Package version 2.5.0+ and use a minimum version of FME Form 2024.0.

If you see a yellow bar indicating a new package version is available, click to update to the latest version (2.5.0+).

To configure the parameters of the ArcGISOnlineConnector, click on the exclamation point and select the ArcGIS Connection section to view your options.

Next, click the drop-down next to Action and select Set Details. Then, click on the ellipsis button next to Path to see a list of folders you have in ArcGIS Online. Double-click the folder with the Feature Service you want to update, click that Feature Service, and click OK.

Note: In addition to the Update Details, there is a Replace Metadata XML operation. This operation enables you to replace the metadata.xml file associated with the target item. Although this topic will not be covered in this article, it will be discussed in a future article.

Please note the folders in this screenshot will not match the folders you own.

Back in the ArcGISOnlineConnector parameters under the Fields section, click on the drop-down button next to Tags and click on the Tag attribute you created in the AttributeCreator.

Once you have the Tags field set to the new Tag attribute you created, click OK and run the workspace.

Now if we refresh ArcGIS Online, the Tags metadata has been updated.

This article covers the newly added Get Details and Set Details actions in the ArcGIS Online Connector. These actions allow for the retrieval or replacement of metadata from a feature service in ArcGIS Online or ArcGIS Portal.

To use the newly added actions, you need to upgrade the Esri ArcGIS Online Connector Package to version 2.5.0 or higher, and a minimum of FME Form version 2024.0 is required. Additionally, ensure that your connections to ArcGIS Online or ArcGIS Portal are working properly.

Important Considerations:

ArcGIS Online metadata text fields support HTML so adding HTML tags may be useful for the set details action for some metadata attributes, such as the description.
Make sure to choose only a feature service from your ArcGIS Online or Portal connection.
If you notice that metadata isn't updating, consider turning off Feature Caching.
Look for a notification indicating that metadata has been updated successfully in ArcGIS Online to confirm that your changes have been saved.

Additional Resources

Tutorial: Getting Started with ArcGIS Online and Portal