Error reading large number of Raster files

Symptom

FME fails with a file read or write error when reading a large number of source raster files.

Cause

For the best performance, FME's raster processing uses a "pull" model, where the source pixels are read only when required to write a destination pixel. However, this model requires that all of the source images be open at the same time.
The C runtime library used when compiling FME limits the maximum number of open files to 512 on a Windows operating system. Attempting to read more than 512 source raster images may cause FME to fail while attempting to open a file for reading or writing. Even transformers like the RasterCheckpointer that write temporary files may trigger the failure.

Resolution

To work around the problem, create a set of smaller sub-mosaics using a RasterMosaicker, then use the RasterCheckpointer to cache them to disk. The subsequent full mosaic will use the checkpointed sub-mosaics as its image source, rather than the original data, avoiding the need to open a large number of files.

Example:



In this example, the RasterPropertiesExtractor is used to get the extents of the source images. The first RasterMosaicker has its Group By set to _min_y, so it will create a sub-mosaic for each horizontal line of source images. The RasterCheckPointer writes the sub-mosaics out to a disk cache. The second RasterMosaicker uses the smaller number of cached images to create the full mosaic.

Because the sub-mosaics are cached to disk, only the source files required for a single sub-mosaic will be open at any given time. The Group By for the sub-mosaicker should be set to an attribute value that will keep the number of source images in a sub-mosaic less than the 512 file limit, while also limiting the number of sub-mosaics to less than 512. In this example we used each horizontal line of source images, but your source data may require a different strategy.