Working with Meta Raster Format (MRF) in Amazon S3

Introduction

FME supports a Cloud Optimized Image called Meta Raster Format (MRF). Developed by NASA's Jet Propulsion Laboratory, MRF data allows for efficient raster accessing, loading and processing - not at the expense of your computer.
 
High-resolution raster processing can be timely and computationally intensive. Capture technology, like satellite and aerial imagery, continue to develop, becoming more sophisticated with each technological generation. Now, traditional rasters are being rivaled by MRF, a new raster format, one that lies - not in space with the satellites - but in the cloud.
 
Cloud compatibility is increasing in popularity. On a global scale, business models are starting to incorporate cloud environments because it saves money, improves performance, and reduces storage demand; allowing companies to outsource their resources and data in a safe way.
 
MRF uses a technique called tiling. This allows areas of interest or tiles to be individually extracted from raster datasets without reading data row-by-row.  These tiles are then compressed based on use-case, as different compression types improve different things (eg. download speed).
 
MRF datasets are comprised of 3 files:

1. The metadata (.mrf) XML-formatted file.
2. The data file (extensions differ, depending on the compression used), which contains the raster files that form the MRF. Examples include JPEG and TIFF.
3. An index (.idx), which represents the raster tiles on a grid.

 
FME needs all 3 files to complete a successful translation. When an MRF file is selected in the Meta Raster Format (MRF) Reader, FME will look for the index and data files automatically. Check out our documentation to learn more about the MRF Reader and Writer. 
 
To understand how FME processes MRF data in a cloud environment, try exploring the two exercises below:

1. Uploading to Amazon S3 - writing out to an MRF dataset
2. Downloading from Amazon S3 - reading in an MRF Dataset

Step-by-step Instructions

Uploading to Amazon S3

Download MRF-AmazonS3-Exercises.zip to follow along with these exercises from home. In this scenario, a series of orthoimages (geotiff) will be converted into MRF and uploaded to Amazon S3. 

1. Open FME Workbench on your machine. Create a New workspace.
2. Add a GeoTIFF Reader to the canvas. Point the dataset to the provided GeoTIFF file(s), and set the reader coordinate system to LL84.

3. Next, add a FeatureWriter to the canvas. Within the parameters window, use the arrow to expand the Creation Options section. For Compression Strategy, change the default to TIFF. Click Ok to apply the changes.

Downloading from Amazon S3

In this exercise, we’re going to read in an MRF dataset living on Amazon S3 into an FME Workspace.

1. Open FME Workbench on your machine. Create a New workspace. 

2. Type on the canvas and add the Meta Raster Format Reader. In this instance we will be connecting to Amazon S3 cloud environment. Click the drop down arrow beside the dataset text box, navigate to Select File From Web, Browse Amazon S3. This will open another window.

3. Once again, we are going to set Credential Source to Web Connection, and add our Account. Select your corresponding Region field, and set the Bucket and Path fields. Recall from the uploading exercise, the MRF data was uploaded to a folder called UploadData. 

Within the Path window, select all files to be downloaded (Use SHIFT keyboard shortcut if selecting all files). Click Ok to apply the changes.
 

4. Once you have returned to the MRF Add Reader window, click the Parameters button. Next, we are going to select the Z-slices to Read. In a MRF context, Z-slices represent an optional third dimension, dimension Z. Click the Ellipses beside Z-Slice field.

5. Select All and click Ok.

 

6. Run the workspace. 

 

Data Attribution

The data used here originates from data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.