Known Issue: Four New Data Types for ArcGIS Pro not Supported for Esri ArcGIS Online and Portal Feature Service Readers and Writers in FME

Issue

When working with Esri ArcGIS Online and Portal Feature Service readers and writers in FME Form, the following data types are not supported:

• Date only
• Time only
• Timestamp offset
• Big Integer

Specifically, the Feature Schema (User Attributes) for both readers and writers ignores those four data types, affecting both the reading of existing Feature Services and the creation of new feature services with Date only, Time only, Timestamp offset, and Big Integer fields.

Cause

Although Date only, Time only, Timestamp offset, and Big integer data types were introduced in ArcGIS 3.2, only the File Geodatabase (File Geodb) reader and writer currently support these four data types (field types) in FME. The Open API version of File Geodatabase reader/writer does not support them.

As documented in our Esri ArcGIS Online (AGOL) Feature Service User Attributes and Esri ArcGIS Portal Feature Service User Attributes documentation, the currently supported data types for these formats are: Date, Double, GlobalID, GUID, Integer, Single, SmallInteger, and String.

Because the four data types are not currently supported, FME cannot match attributes with Date only, Time only, Timestamp offset, or Big integer data types to the supported data types. This leads to the Feature Schema ignoring attributes with these four data types, resulting in these attributes not being displayed in the Reader and Writer User Attributes.

Due to this limitation, both reading attributes with Date only, Time only, Timestamp offset, and Big integer values and creating Feature Services with attribute data type set to Date only, Time only, Timestamp offset, or Big integer are affected. However, writing features for database operations remains unaffected as long as the feature service defined with these data types already exists.

Dynamic Schema Definition is also affected when using the Writer, as the dynamic schema matches the defined schema of the data types to the Writer's supported data types. In other words, even if you hard-code the Date only, Time only, Timestamp offset, or Big integer data types into the schema, the Writer will not be able to write them.

Workaround

Reading Attributes with Date only, Time only, Timestamp Offset, or Big Integer Data Types

The behavior of attributes with Date only, Time only, Timestamp offset, or Big integer data types varies depending on whether you have opened the Feature Type's Parameters.

• If you open the Feature Type Parameters and click OK, even without making any changes:
  All attributes with Date only, Time only, Timestamp offset, and Big integer data types will become unexposed.
• If you do not open the Feature Type Parameters:
  Attributes with Date only, Time only, Timestamp offset, or Big integer data types will remain visible in both the Visual Preview and Feature Information pane.

Method 1: Reader (Opened Feature Type Parameters) + AttributeExposer

To expose attributes with Date only, Time only, Timestamp offset, or Big integer data types, you'll need to open the Feature Type Parameters, click on User Attributes, and select "All Attributes" for Attributes to Read, then click OK on the Feature Type Parameters.

Then, add AttributeExposer to the canvas, connect it to the reader, and manually specify the attribute names containing Date Only, Time Only, Timestamp offset, or Big integer data types. The attribute names you specify must exactly match the Field Name values to be exposed, including case sensitivity, otherwise there will be no values for the intended attributes.

Method 2: Reader (Unopened Feature Type Parameters) + AttributeManager

Without opening the Feature Type Parameters for the Reader after initial configuration, you can directly connect AttributeManager to the Reader. AttributeManager will automatically assign Date Only, Time Only, Timestamp offset, or Big integer data types with generic FME data types.

This is the recommended method to read features with Date Only, Time Only, Timestamp offset, and/or Big Integer data type attributes.

Method 3: FeatureReader + AttributeExposer

This method is similar to Method 1. When reading ArcGIS Online/Portal Feature Service using the FeatureReader transformer, all attributes with Date Only, Time Only, Timestamp offset, or Big Integer data types will always be unexposed. To expose these attributes, connect the AttributeExposer to the FeatureReader and expose the attributes with these data types. Unlike Method 1, if you run the workspace/FeatureReader, you can select the attributes instead of having to specify them manually.

Writing to Existing Feature Services with Date, Time, Timestamp Offset, or Big Integer Fields

Database operations (update, insert, and delete) for Big Integer data type will not be affected as long as the attribute with Big Integer in the User Attribute in the ArcGIS Online/Portal Feature Service have the exact same name as the destination attribute names. FME honors the attribute type of the existing schema definition in the destination.

Attribute types specified in AttributeManager and AttributeExposer do not enforce attribute type limitations. Instead, attribute types are enforced by writers or by using and configuring an AttributeValidator or Tester, as described in this documentation. 

However, with Date only, Time only, and Timestamp offset data types, you must use the DateTimeConverter transformer to convert your data with these data types to write successfully. Specifically, using the DateTimeConverter transformer, the data from the following data types must be converted to the following ISO format before passing the features to the writer:

• Date only → %Y-%m-%d
• Time only → %H:%M:%S
• Timestamp offset → %Y-%m-%dT%H:%M:%S%Ez

Timestamp offset is not supported for ArcGIS Portal Feature Service with Enterprise 11.4 but supported for ArcGIS Online

Creating New Feature Services with Date only, Time only, Timestamp Offset, or Big Integer Fields

Using the ArcGIS Online/Portal Feature Service Writer, you can not directly create a Feature Service with attributes defined as Date only, Time only, Timestamp offset, or Big integer data types. Attempting to create a new Feature Service will 'automatically' define these data type the following unless specified to other data types on the Writer:

• Date only → String
• Time only → String
• Timestamp offset → String
• Big integer → Integer

To create a Feature Service with the correct Date only, Time only, Timestamp offset, or Big integer data types for fields within layer/table, you must create and use a File Geodatabase as a intermediate dataset to upload to ArcGIS Online/Portal, then publish that File Geodatabase as a Feature Service.

ArcGIS Online and Portal requires File Geodatabase to be zipped to upload the item. Due to this requirement, there are two methods to create a Feature Service with Date only, Time only, Timestamp Offset, and/or Big Integer Fields in the feature layer/table.

Method 1: Manual Zip + ArcGISOnlineConnector

This method is similar to the method of uploading and publishing Shapefile as seen in Part 1 How to use the EsriArcGISOnlineConnector Article. 

1. Write the File Geodatabase with all the correct Attribute and their data type to a path directory 

Ensure that all the attribute types are properly defined before proceeding.

2. Navigate to that path location of the File Geodatabase using File Explorer or Finder and zip it. 

The entire .gbd folder must be contained within the .zip file.

3. Add a Directory and File Pathnames Reader to the Canvas

Add a Directory and File Pathnames Reader to your canvas by clicking on the canvas and typing "Directory and File Pathnames". Select the parent folder containing the zipped file of the File Geodatabase.

In Path Parameters, set Path Filter to "/*.zip" (without quotes) to retrieve all zipped files in the specified folder.

Run the Reader to select the Feature Cache value needed for the next step

4. Add a Tester to the Canvas 

Add the Tester transformer to your canvas then connect this transformer to your Directory and File Pathnames Reader then open its parameters. In the parameters, set the Left Value as "path_filename". For the Right Value, use the downward arrow to access Cached Values and select your zipped File Geodatabase.

5. Add an ArcGISOnlineConnector 

Add the ArcGISOnlineConnector to your canvas, and then connect the ArcGISOnlineConnector to the Tester transformer.

Double-click on the ArcGISOnlineConnector to configure the transformer's parameters. If you have previously set up an ArcGIS Online/Portal connection, you can select that connection here. If you need to create a connection to ArcGIS Online, please refer to How to Create an ArcGIS Online Web Connection (OAuth 2.0), and if you need to create a connection to ArcGIS Portal, please refer to How to Create an ArcGIS Enterprise Portal Web Connection (OAuth 2.0).

1. In the Request section of the Parameters, select Upload for the Action. 
2. In the Data Source section, select the dropdown arrow next to File to Upload, select path_unix 
3. In the Metadata section, you can either specify a name or use the path_filename for the Title field. For Item Type, select File Geodatabase.
   Then in the Tag Field, write a text relevant to your data. 
4. In the Content Destination section, you can specify a specific folder within the content folder. By default, the uploaded file will not be in a specialized folder in the ArcGIS Online/Portal. 
5. You can manually specify the Title.

Click OK to close the parameter dialog. 

6. Add Another ArcGISOnlineConnector

Add a new ArcGISOnlineConnector and connect it to the existing ArcGISOnlineConnector. This new ArcGISOnlineConnector will be used to publish this File Geodatabase as a Feature Service. 

Open up the parameters.

1. In the Request section of the Parameters, select Publish for the Action. 
2. In the Publishing Options section, for the Content Source, select _id

• You can specify name for the Feature Layer Title, otherwise the name used for the title for the previous ArcGISOnlineConnector will be used.

Click OK to close the parameter dialog.

7. Run the Workspace and Check ArcGIS Online or Portal to View Results

Run the workspace and monitor the <rejected> port for any errors. Upon successful completion, confirm that both the File Geodatabase was uploaded and the Feature Service was created in your ArcGIS Online/Portal.

Save and close the workspace. 

Method 2: All in One Workspace

The FeatureWriter transformer contains a Summary port that provides the file path location of the written dataset. This workspace, ArcGISPublisher.fmw, demonstrates creating a Feature Service on ArcGIS Online/Portal using File Geodatabase with non-supported attribute types for ArcGIS Online/Portal in a single workspace.

Open the ArcGISPublisher.fmw workspace.

While the annotations in the workspace detail what each transformer and custom transformer does, the following is a high-level overview, in order:

• FeatureWriter to write File Geodatabase to a path location
• AttributeCreator to create '_data_location_unix' and '_data_name' attributes
• StringReplacer to extract the FGDB File Name with Extension, stores in '_data_name'
• AttributeCreator to create '_data_name_noext'
• StringReplacer to extract the FGDB File Name without the extension, stores in '_data_name_noext'
• StringReplacer to transforms windows path location to a unix path, stores in '_data_location_unix'
• StringReplacer to replaces .gdb with .gdb.zip to the FGDB name, for '_data_location_unix'
• ZipArchiver to zip the FGBD
• ArcGISOnlineConnector to Upload the Zipped FGDB to ArcGIS Online/Portal
• ArcGISOnlineConnector to publish the uploaded FGDB as Feature Service on ArcGIS Online/Portal
• Logger to display error message for ZipArchiver, and the two ArcGISOnlineConnector

To use this workspace, you must have Windows architecture, ArcGIS Online or Portal Web connections configured, and the FeatureWriter transformer configured. While the same settings are required, you can alternatively copy the entire contents and paste them into your workspace!