MinIO and FME

Liz Sanderson
Liz Sanderson
  • Updated

FME Version

  • FME 2019.x

Introduction

MinIO is an open source object storage server that is compatible with Amazon’s S3 API. As both FME Desktop and Server support Amazon S3 integrations this article will show how you can connect to MinIO from within FME.

 

FME Desktop

To connect to data stored in MinIO you will need to download the Amazon S3 Connector package from the FME Hub .

This FME package contains the S3Connector transformer and the Amazon S3 web filesystem (meaning readers and writers can access files from S3 compatible storage directly). 

The S3Connector is able to access MinIO to upload, download, delete files and folders or list file/folder information from a MinIO bucket.


Connect to MinIO

In order to connect to objects stored in MinIO you will need to set the Alternate Endpoint to your MinIO installation. 

This is applicable to the S3Connector transformer, Amazon S3 Compatible Storage web connection and the Amazon S3 web filesystem:


 

FME Server

FME Server supports using Amazon S3 as a trigger for Automations. However, as there is not parameter for alternate endpoints this will default to communicating with Amazon S3. In order to use changes within a MinIO bucket to trigger an Automation, MinIO has to be configured to Publish events via Webhooks. To set this up, refer to the MinIO documentation.


Create a Webhook Triggered Automation

In FME Server Automations, add a webhook trigger. Once the Automation is saved, the webhook url will be available. You need to use this in the MinIO event publication configuration.


 

Connecting the webhook output to a Log action will print the event message to the Automation log:

Event message: {"EventName":"s3:ObjectCreated:Put","Key":"jentest/Screen Shot 2020-09-11 at 12.34.17 PM.png","Records":[{"eventVersion":"2.0","eventSource":"minio:s3","awsRegion":"","eventTime":"2020-09-17T01:59:10.655Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"minioadmin"},"requestParameters":{"accessKey":"minioadmin","region":"","sourceIPAddress":"::1"},"responseElements":{"x-amz-request-id":"16356F9A31909120","x-minio-deployment-id":"170c2d8b-1799-47ea-a003-f31953442a62","x-minio-origin-endpoint":"http://192.168.1.69:9000"},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"jentest","ownerIdentity":{"principalId":"minioadmin"},"arn":"arn:aws:s3:::jentest"},"object":{"key":"Screen+Shot+2020-09-11+at+12.34.17+PM.png","size":15810,"eTag":"0237cc0024d1370f7cabd1ab339c2c70","contentType":"image/png","userMetadata":{"content-type":"image/png"},"sequencer":"16356F9A32143458"}},"source":{"host":"::1","port":"","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36"}}]}

 

Using the Log action will allow you to copy the event JSON. In the webhook action you can ‘Import Webhook Keys from JSON’. This makes it easier to work with information sent in the webhook message.

 


Three keys are imported: EventName, Key and Records.


Logging these three keys would give you the following information:

EventName: s3:ObjectCreated:Put

Key: jentest/Screen Shot 2020-09-11 at 12.34.17 PM.png

Records: [{"s3":{"bucket":{"name":"jentest","arn":"arn:aws:s3:::jentest","ownerIdentity":{"principalId":"minioadmin"}},"s3SchemaVersion":"1.0","configurationId":"Config","object":{"size":15810,"eTag":"0237cc0024d1370f7cabd1ab339c2c70","contentType":"image/png","userMetadata":{"content-type":"image/png"},"key":"Screen+Shot+2020-09-11+at+12.34.17+PM.png","sequencer":"16356F9A32143458"}},"awsRegion":"","eventVersion":"2.0","responseElements":{"x-minio-origin-endpoint":"http://192.168.1.69:9000","x-amz-request-id":"16356F9A31909120","x-minio-deployment-id":"170c2d8b-1799-47ea-a003-f31953442a62"},"eventSource":"minio:s3","eventTime":"2020-09-17T01:59:10.655Z","requestParameters":{"accessKey":"minioadmin","sourceIPAddress":"::1","region":""},"eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"minioadmin"},"source":{"port":"","host":"::1","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36"}}]



These keys can then be used in downstream actions. An example would be to pass those key values to a workspace using Published Parameters, and an S3Connector would allow you to download the data for processing:





Combining the webhook trigger in Automations with the Amazon S3 package should allow you to automate workflows using FME and MinIO.

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.