SHAPEFILE Writer: Truncates attribute names/values to one character

This article is applicable to FME 2019.1 and newer with the SHAPEFILE format introduced to replace the deprecated ESRISHAPE format.

Symptom

Using the SHAPEFILE writer the attribute names and values of my dataset have been truncated to a single character.

SHAPEFILE writer, expected vs actual output

If I inspect the output file in FME Data Inspector using the SHAPEFILE format it appears correct, however opening the file using the ESRISHAPE format or other applications is when the attribute data is missing.

Viewing the translation log warnings are reported similar to:

WARN  |Shapefile writer: A standard Codepage name was not found for encoding 'UTF-16LE', writing encoding name to .cpg. Other applications may not correctly identify this encoding, if problems are encountered, select a different encoding

Cause

If a user has not specified the SHAPEFILE writer’s encoding parameter, by default, it is set to use fme-source-encoding. This means the writer will attempt to detect what encoding to use based on the encodings of incoming attributes. The first one it finds will be the one it uses to write the file out.

When the source data are tagged with an encoding type such as UTF-16LE, this contains embedded nulls and whilst the old writer was fairly loose with encoding, the new writer is unable to identify a standard code page and this can result in problems with the output.

Resolution

Override the SHAPEFILE writer’s decision by explicitly setting the Writer Character Encoding to use something other than fme-source-encoding.

Add Writer Parameters: set Character Encoding

If the writer has already been added to the canvas, you can locate the Character Encoding setting under the writer parameter section in the Navigator.

Navigator Writer Parameters: set Character Encoding