AWS Credentials in FME

Dami Obasa
Dami Obasa
  • Updated

FME Version

  • FME 2023.0

Introduction

FME can integrate with many AWS services like S3, SQS, SNS, etc. Find out more in the Working with Amazon Web Services (AWS) and FME article.

This article shows the different ways FME provides AWS credentials when using AWS transformers or Triggers and Actions in FME Flow Automations.
 

Using AWS Credentials with AWS 

FME natively supports different ways to authenticate and authorize with AWS. The most common ways are:

  1. AWS Access Key ID and AWS secret key within transformers (FME Form) or automation triggers/actions (FME Flow)
  2. Environment variables

FME does not natively support IAM roles, but there is a workaround.  

AWS transformers in FME use Boto3, the Python SDK for AWS. Both Boto3 and the AWS CLI evaluate credentials in different orders. If FME isn’t using the credentials you expect, check which credentials you’ve provided and how compared to the order. The Boto3 order is available here and the order for the AWS CLI is available here.
 

Example Using Secret and Access Keys

AWS Access and Secret Keys Within a Transformer

This is the most common method as it allows you to input the Access Key ID and Secret Access Key generated from AWS directly into the transformer. An example can be seen in the screenshot below using the S3Connector transformer. When you open the transformer, you will see the parameters allowing multiple options for passing credentials to the transformer. Select “Web Connection” as the credential source, add a web connection, then input the AWS Access Key ID and Secret Access Key. 

S3 connector.png

 

Environment Variables

You can install AWS CLI on your Mac or Windows system, then configure the AWS CLI  to use the AWS credentials you want FME to use.

AWS credentials.png

IAM Role Attached to an EC2 Instance


There are cases where users would like to use AWS services in FME with IAM roles instead of distributing credentials. This is a security best practice, as the role uses temporary credentials that are not associated with a specific user. Permissions for the AWS service(s) that you want to interact with need to be attached to the role.

  FME does not natively support the use of roles, however, there is a workaround using the SystemCaller to interact with AWS using the CLI.

The example below uses FME Form installed on an EC2 instance. To use with FME Flow, the same workspace can be published to FME Flow where the engine(s) are running on an instance with the IAM role attached. The example uses SNS and the SystemCaller will be used to publish a message to a topic.

 

Prerequisites

  • Set up SNS or whichever AWS service that is being used
  • Create an IAM role that has access to your SNS topic 
  • Set up an EC2 instance you want to install FME on.
    • FME Form can be installed on this EC2 instance. If using FME Flow, the EC2 instance hosting the engine will need the role assigned, as the engine will be running the SystemCaller.
  • Attach the IAM role to your EC2. This should inherit the role and you should not need to specify your AWS Access Key ID and Secret  Key when logged into the virtual machine.

    First, install and test if the AWS CLI works to publish your message
    Run this command either in “CMD” for Windows or “Terminal” for Mac:
“aws sns publish --topic-arn (Enter ARN here) --message "Hello World!"


The output should be JSON with a Message ID.
 

Build the workspace in FME Form

  1. Connect to the EC2 and install and license FME Form if it is not already installed.
  2. Add and connect the “Creator” and “SystemCaller” transformers.
  3. Open the  SystemCaller parameters and then paste the command you would like to run into the Command Line parameter.. 

Example: This command sends an SNS notification to print "Hello World! My name is Dami"
 

  Command:  

aws sns publish --topic-arn arn:aws:sns:ca-central-1:441807460849:Dami-sns-topic --message "Hello World! My name is dami"


Systemcaller.png

  1. Run the workspace. 
  2. To verify this works as expected, check the translation logs for a “Message ID” as this indicates the message and notification (if configured) has been sent 
Translation.png

 

Additional resources 

Please consider posting to the FME Community Q&A if you have further questions or issues that are not addressed in this article. There are also different support channels available. For issues or limitations specific to AWS authentication, authorization or services, please refer to their documentation.

Cloud computing with AWS

Working with Amazon Web Services(AWS) and FME 

AWS security credentials

Launch an Instance with IAM role

How AWS utilizes Boto3 for configuring credentials 

How to configure AWS CLI

Install or update the latest version of the AWS CLI

Creating IAM roles

How to publish to a topic

Was this article helpful?

Comments

0 comments

Please sign in to leave a comment.