ArcGIS Online / ArcGIS Portal feature services: common errors writing to existing feature services mismatched schema/data types

Symptom

When writing to an existing ArcGIS Portal or ArcGIS Online Feature Service layer, there are errors reported in FME log files, as reported below. In some cases, the error messages do not provide detailed information to  determine the cause of workspace failure.

An example of an error in the log you may see:

Encountered an unexpected error. The error code from the server was '500' and the message was: 'JSONObject["globalId"] not found.'. Details: ''

Python Exception <Exception>: Second parameter to FMEException() must be a list of strings. becomes ArcGIS Portal Feature Service Writer: Encountered an unexpected error. The error code from the server was '500' and the message was: 'You do not have permissions to access this resource or perform this operation'. Details: '[]'

'addResults' error for a feature in 'DataTypes'. The error code from the server was '10500' and the message was: 'Database error has occurred.'

Cause

The main reason for such error messages are a mismatch between incoming data and the feature service's existing schema. 

Resolution 

This issue has been resolved in FME 2021.0 and newer. We will now trim data for attributes with data type ‘string.’ But other data types such as Date, Integer, or double, please follow the solution provided below.

The workaround is divided into three steps: 1. Check the existing schema, 2. Validate the incoming data and 3. fix/correct the incoming attribute data before writing the existing feature service.

For this example, I will refer to an instance with the following data types

Step 1: Check the writer schema 

Before we begin we need to verify that you have the most up-to-date schema for the feature service(see doc). Go to Writer → Update Feature Type. Select the existing layers for the feature service.

Check the existing attribute schema for the feature service(layer). Right-click on ArcGIS Online/Portal writer → Show Summary Annotation

 

Step 2: Validate attributes to match the existing schema 

For this step, we will use the  AttributeValidator transformer and configure the validation rules for each attribute based on the writer’s schema. Here is an example of what the AttributeValidator could be configured to: 

Add a logger transformer to the “Failed” port to investigate the features that are failing. Example error messages from the sample workspace:

`_fme_validation_message' has value `Attribute 'string1' with value 'yes' fails check for Maximum Length = 2'

`_fme_validation_message_list{0}' has value `Attribute 'string1' with value 'yes' fails check for Maximum Length = 2'

`_fme_validation_message_list{1}' has value `Attribute 'integer' with value 'test' fails check for type is 'INT''

`_fme_validation_message_list{2}' has value `Attribute 'string2' with value 'This is test string with a total length of fiftyone' fails check for Maximum Length = 50'

Step 3: Update the data to match the writer’s schema

 Any features that fail will require the data to be adjusted. Use various transformers such SubstringExtractor, AttributeManager, and DateTimeFormatter to fix the data. Here is an example using AttributeManager

@Evaluate(@int8(@Value(integer)))  -- Convert data to Int8
@Substring(@Value(string2),0,49)   -- Trim string attribute length to 50 characters
@DateTimeCast(@Value(datetime),datetime) -- cast to the datetime temporal type