Managing ArcGIS Online Metadata in FME

Sienna Emery
Sienna Emery
  • Updated

FME Version

  • FME 2023.0

Introduction

FME has two custom transformers that can be used to manage metadata in ArcGIS Online: the ArcGISOnlineMetadataFetcher, which is used to retrieve metadata, and the ArcGISOnlineMetadataOverwriter, which is used to replace existing metadata.

Users may want to use FME to validate their metadata posted to ArcGIS Online to ensure it meets the requirements set by their organization. Additionally, FME can also be used to send notifications to the data steward if the metadata is not complete or create a validation report.
 

Requirements

  • Access to FME Form (Formerly FME Desktop)
  • An ArcGIS Online Account

 

Step-by-Step Instructions

Part 1: Retrieving metadata from ArcGIS Online

1. Open FME Workbench


2. Add a Creator

Search for the Creator transformer and place it on the canvas. The Creator will initiate the workflow.


3. List Items in ArcGIS Online

From there, add an ArcGISOnlineConnector to the canvas and connect it to the Creator transformer.

image16.png

To configure the parameters of the ArcGISOnlineConnector, click on the cogwheel and select the ArcGIS Connection section to view your options.

If you have previously set up a connection to ArcGIS Online or Portal, you can select it from the options. If you need help creating a connection to Portal, please refer to the guide on How to create an FME Web Connection for Esri ArcGIS Portal Feature Service (OAuth 2.0).

If you need to create a new ArcGISOnline connection, select Add Web Connection… and create a name for your connection. Then select Authenticate, enter your ArcGISOnline username and password, and click Sign In.

Next, click the ellipsis button next to the User Content Folder to see a list of folders you have in ArcGIS Online. Choose a folder that you know contains Feature Services and select OK.

image9.png

Please note the folders in this screenshot will not match the folders you own.

Leave all other options as the default. Select OK to close the dialog.


3. Inspect the Output from ArcGIS Online

With Feature Caching enabled, click Run to run the workspace. Once the workspace has finished, click the green magnifying glass on the Output of the ArcGISOnlineConnector to inspect the output. This will display the attributes in the Visual Preview window.


4. Test to Confirm that the Items are a Feature Service

Add a Tester to the canvas and attach it to the Output port of the ArcGISOnlineConnector.

Open up the Tester parameters, for the Left Value, from the drop-down, highlight Attribute Value, then select _item_type. Then, for the Right Value, from the drop-down, highlight Cached Values and select Feature Service.

image20.png

Click OK to close the Tester parameters.


5. Get the ArcGISOnline Metadata

Add an ArcGISOnlineMetadataFetcher and connect it to the Passed port of the Tester.

Open the ArcGISOnlineMetadataFetcher parameters to configure it.

Set the Server URL; this will be the URL you use to access ArcGIS Online. If you do not know your URL, go to ArcGIS Online, sign into your account and examine the URL you are redirected to after logging in.

Click the drop-down by Item Id and hover over Attribute Value and select _id.

Finally, select the Web Connection used in the ArcGISOnlineConnector.

image23.png

Click OK to close the parameters.


6. Examine the Results from the ArcGISOnlineMetadataFetcher

Run the workspace. Click on the magnifying glass on the Output port of the ArcGISOnlineMetadataFetcher to view the results in Visual Preview. The _response_body attribute contains the metadata as XML.

image21.png

There may also be features in the rejected port. Investigate the _response_body if these exist. In this example, the metadata does not exist for this item.

image25.png

7. Write Out the XML Files

To write out the XML files, the AttributeFileWriter transformer is used. The files can then be sent to the data stewards for manual validation, or use part 2 of this tutorial to learn how to automatically validate the XML file.

Add an AttributeFileWriter and attach it to the Output port of the ArcGISOnlineMetadataFetcher. Open up the AttributeFileWriter parameters, and set the Source Attribute to _response_body. For Target Filename, open up the text editor and write the XML to a local folder and set the name of the file to @Value(_name)_metadata.xml. For example, C:/Users/Documents/MetadataOutput/@Value(_name)_metadata.xml. Click OK to exit the text editor. The rest of the values can be left to the default then click OK again.

image19.png

Run the workspace and write all of your XML files out to a folder.

image15.png

Save and exit the workspace.

 

Part 2: Validating your XML

 The XML validation can occur with the XML still in the attribute. However, for the sake of simplicity for the tutorial, a workspace has been created to allow you to follow along with each step.


1. Download the Sample XML

Download the CellSignals_metadata.xml XML file found in the files section of this tutorial.


2. Click New to Create a New Workspace


3. Add a Text File Reader

We are using a Text File reader here in place of an XML reader in the workspace. This is because, in the tutorial, we want to emulate validating the XML attribute returned in the ArcGISOnlineMetadataFetcher.

The XML reader would attempt to parse the XML, whereas the Text File reader can be set to read the full file in one attribute.

Add the Text File reader to the canvas, and for Dataset click on the file browser button. The file browser will be set to select .txt files, click the drop-down and select All Files to be able to select an XML file. Select the CellSignals_metadata.xml.

image12.png

Next, open the reader Parameters and set Read Whole File at Once to Yes.

image32.png

Click OK and OK again to create the reader.

 

4. Examine the XML

Run the workspace. Click on the green magnifying glass to open the XML in Visual Preview. The XML is read into the text_line_data attribute. Double-click on the text_line_data to open the data in a larger window. Click on the ABCXYZ button in the bottom left corner and select XML, to view a formatted version of the text. Then, use Ctrl + scroll wheel to zoom in on the text, if you need to enlarge it. More keyboard shortcuts can be found here.

image2.png

The XML will need to be parsed in order to determine if the values are present for validation. By looking at the file, we can find what XML tags we will want to use to create these attributes.

Here we can see the title, publication date and description.

image27.png

Unfortunately, it is not very easy to determine which values here will align with the Metadata found in ArcGIS Online. However, if you are looking to parse your own metadata, it is a good practice to cross-reference the XML file with the values found in ArcGIS Online.


5. Examine Metadata in ArcGIS Online

You won’t be able to follow these next steps exactly because you do not have access to the Feature Service in this tutorial. However, on your own ArcGIS Online account, select a Feature Service, and select Metadata.

image22.png

Here, you can explore the metadata provided in ArcGIS Online and compare it to the XML version. If you did not write out an XML file earlier in this tutorial, the XML file can be downloaded from your ArcGIS Online account if needed

image7.png

Here you can compare the metadata values in the ArcGIS Online interface to the XML. You can even set temporary values, save the file, and download it again to determine where the value will be set in the XML file.

image26.png

6. Parse the XML File

Go back to FME Workbench. Then, add an XMLFragmenter to the canvas and connect it to the Text File reader feature type.

From there, open up the XMLFragmenter to set it up. Set the XML Source Type to XML File or URL. Then, set the XML file to the CellSignals_metadata file.

image28.png

Next, set the elements to match. Here, select the following elements to match:

  • metadata/dataIdInfo/idAbs
  • metadata/dataIdInfo/idCitation/date/pubDate
  • metadata/dataIdInfo/idCitation/resTitle

Then, click OK.

image10.png

This dialog makes it easy to select elements in the XML file. However, it is only available if the file is selected as the source type. It is unavailable if the attribute is selected as the XML Source Type. In this example, we are first selecting the file as the source type to access this dialog, but we will switch it to the attribute shortly.
 

Next, set Reject Features With No Fragments to No.  
 

Select Options… Check Enable Flattening and select OK.

image18.png

Click the drop-down by Expose Attributes. Then, set the following attributes to expose:

  • resTitle
  • pubDate
  • idAbs

This will expose the attributes selected in the elements to match section.

image13.png

Click OK to exit.

Finally, switch the XML Source Type to Attribute with XML document and set the XML Attribute to text_line_data. The end result should look like this.

image24.png

Click OK to exit.


7. Run the Workspace and View the Results

Running the workspace at this stage, creates the following XML fragments.

image3.png

It will be easiest if these attributes are contained in one feature; therefore, we will need to use an Aggregator in the next step.


8. Aggregate the Attributes

Aggregate the attributes, so they can all be found on one feature. This will make it easier to determine if attributes are missing. Especially if multiple XML files are being validated.

Add an Aggregator to the canvas and attach it to the Fragments output port to the XMLFragmenter.

Open the Aggregator to configure the parameters.

The Aggregator has the option to use Group Processing which would be essential if you are validating multiple files, as Group Processing will group features together by an attribute like an id or filename. This enables multiple features belonging to the same file to be grouped together. However, since in the tutorial, there is only one file, we do not need to set it.

Set the accumulation mode to Merge Incoming Attributes. This will ensure that all attributes are kept from the features. Click OK.

image14.png

 

9. Run the Workspace

Run the workspace and review the output of the Aggregator. All the attributes should now be contained in one feature.

image11.png

10. Add Validation Rules

Attributes can be validated using the AttributeValidator. Here, we will do some basic validation.

Attach the AttributeValidator to the Output port of the Aggregator.

In the parameters, under Attributes to Validate select idAbs, pubDate, and resTitle. Then, under Validation Rule select Has a Value. Click OK.

image1.png

Run the workspace again. Here, you should see our file has passed our validation steps.


11. Change XML File to Validate

Now, we will try to validate a new file that will not pass our new validation rules. To do this, download the Bad_metadata.xml file found in the Files section of this tutorial.

To change the XML file, click the drop-down arrow by the Run button ensuring that Prompt for Parameters is checked and select Rerun the Entire Workspace. Select the Bad_metadata XML file, ensuring to select all files instead of just text files.

image31.png

If you select the magnifying glass on the XML Fragmenter, you’ll find that the only matched element is the resTitle element. This means that only the resTitle is present.

image5.png

Now, if you select the AttributeValidator failed port and take a look at the feature information window. Here, we can see a list that contains all of the failure messages.


12. Explode the List

Add a ListExploder to the canvas. Attach it to the Failed port of the AttributeValidator. This will expose all the failed validation messages.

Open up the parameters. Then select _fme_validation_message_list{} as the list to explode.

Run the workspace again, the final output should contain two features with the following validation messages.

image29.png

These validation messages can be output to a file and a report can be generated and sent to data stewards. The next section of this tutorial will go over the report creation.

 

Part 3: Review The Validation Report Workspace

1. Open the Metadata Report Generator

Download the MetadataReportGenerator.fmw found in the Files section of this article.

This workspace combines the workspaces in Part 2 and 3. Here the XMLFragmenter, is changed to fragment the _response_body attribute and Merge Attributes from Input Features is now set to Yes.

image8.png

Additionally, the Aggregator is now set to use the group by function to aggregate the fragments together based on the feature service ID.

image30.png

Finally, an Attribute Manager has been added to clean up the unneeded attributes and a conditional value is set to create a better error message if metadata does not exist in the attribute. Open the AttributeManager and select the value beside the Error Message attribute.

 image17.png

Here, a conditional statement is used to set the attribute value to “Metadata not found” if the response body contains the Error: Metadata for item not found. This same method could be used to change the default error message from the AttributeValidator if necessary.


2. Run the Workspace

Finally, run the workspace, ensuring to modify the user parameters to your own.  

image4.png

After the workspace has been run, single-click the Sheet1 writer feature type to open the popup menu, then click the Folder icon. In the File Browser, open the Metadata Issues Microsoft Excel file.

image6.png

This Excel file will contain a basic list of all metadata that does not meet the requirements set. This Excel file can be enhanced using a template for formatting more details that can be found in this article.

 

Additional Resources

Tutorial: Getting Started with ArcGIS Online and Portal

 

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.