Introduction
If you are familiar with the FME Flow architecture, you’ll be familiar with the FME Flow Engines… the data processing component of FME Flow. The Remote Engines Service enables the deployment of FME Flow Engines in environments where traditional distributed engines are not feasible. The FME Remote Engines Service for Snowpark enables FME Flow Engines to operate within the Snowflake environment, ensuring secure and proximity-based data processing.
This type of deployment is best suited for reading and writing data in the Snowflake environment. Another use case would be pulling data from external sources to write to the Snowflake database, where the data may be enriched with sensitive information before being written to the Snowflake tables.
Available Versions of FME Remote Engines Service
The FME Remote Engines Service is currently available for FME Flow 2025.1. This is the inaugural version of the FME Remote Engines Service for Snowflake. Currently, we are offering a Trial and Production version of the Snowflake Native App.
Trial Use
If you have found the FME Remote Engines Service trial on the Snowflake Marketplace, you can launch it free for 30 days. In order to use the FME Remote Engines Service, you must also have FME Flow available and licensed. FME Flow is required to orchestrate when the Remote Engines Service will run jobs. FME Form, the authoring tool, is also necessary in order to create workspaces and publish them to FME Flow. When the 30-day trial expires, the App will be removed from your account.
Production Use
If you wish to use the FME Remote Engines Service in a production-related environment, please contact your Safe Software Account Executive. This will allow our team to confirm the requirements and ensure the best experience.
This article will outline the necessary steps to add an FME Remote Engines Service App to your Snowflake environment and connect it to an existing FME Flow licensed environment.
Requirements and Prerequisites.
1. The Remote Engines Service App must have been previously shared with your Snowflake Account (unless using the Marketplace Trial). For more information, contact your Safe Software Account Executive.
2. Must have a similar version of FME Flow available to register the Remote Engines Service with. This environment must be licensed and have internet access to communicate with the FME Flow Remote Engine Service in Snowflake. This can be an on-premise environment, a cloud environment or an FME Flow Hosted environment. Note: If using FME Flow Hosted, please review this article for licensing the Remote Engine Service after it has been configured.
3. It is recommended that you have at least a two-engine license: one engine that can operate in the primary FME Flow environment and another engine for the FME Flow Remote Engines Service. Remote Engines can use Standard and CPU Engines.
4. Must have access to the FME Flow superuser or similar level privileged user in the existing FME Flow environment in order to register the FME Remote Engines Service.
5. You must familiarize yourself with the FME Flow Core’s password policy to ensure the Remote Engines Service is given a secure password (Part 5). If you are unsure about the password policy, visit the FME Flow Documentation here for more information.
- “For new installations of FME Flow, Password Policy is enabled by default for an eight-character minimum that must not contain the Username.”
- When entering a password for the Remote Engines Service in Part 5, it must meet the FME Flow Core’s password policy; otherwise, it will not be accepted, and you will not be able to register the Remote Engine Service with the FME Flow Core.
6. Must have access to Snowflake Snowsight with an ACCOUNTADMIN Role to replicate, launch, and configure the Snowpark Container Services.
7. You must have a Snowflake user account of type=service configured with key pair authentication. The private key must not have a passphrase associated with it.
-
If you do not have access to the OpenSSL command line, you may use https://www.cryptool.org/en/cto/openssl/ tool with the following commands:
1.
openssl genrsa -out privkey.pem 20482.
openssl rsa -pubout -in /privkey.pem -outform PEM -out /pubkey.pem
Step 1 creates the private key file (without a passphrase), and Step 2 creates the public key.
The public key will be used in Snowflake to set the RSA_PUBLIC_KEY of the service user.
- Example SQL: ALTER USER myappuser SET RSA_PUBLIC_KEY='MIIBIjANBgkqhkiG9w0...AB';
8. Once the FME Remote Engines Service App has been launched in the Account, the service user will be granted privileges to access the App (Part 4).
9. The public and private keys of this service user will be used in the FME Flow Core to register the connection to the FME Remote Engines Service (Part 5).
10. You must know the Databases and Schemas the App will access. Once the FME Remote Engines Service App is launched, these privileges can be applied to the App. More information in Part 7.
Building the FME Remote Engines Service in Snowflake
Part 1: Obtaining FME Remote Engines Service for Snowpark
FME Flow Remote Engines is available as a trial and as a private listing. You can launch the trial directly from the marketplace. To obtain the private listing, you can contact your Safe Software Account Executive to have it shared with your Snowflake Account.
The trial is designed to test and validate the FME Remote Engines Service with your account. The trial version will be updated regularly.
The private listing is meant for development to production environments and is version-dependent based on your FME Flow environment. It will also have a longer life cycle that aligns with your FME Flow upgrade cycle. This ensures nothing changes in your FME Remote Engines Service App until you decide.
If you chose the trial, your Snowflake Administrator will receive an email with the subject Snowflake Marketplace: Request Fulfilled for Listing 'FME Remote Engines Service (Private Trial)' from no-reply@snowflake.net indicating that the trial is now available in your account.
If you chose to have the private listing shared with your account, your Snowflake Administrator will receive an email with the subject Snowflake Marketplace: Request Fulfilled for Listing 'FME Remote Engines Service 2025.1' from no-reply@snowflake.net indicating that the private listing 'FME Remote Engines Service 2025.1' has been shared with your account. The link in the email received can be used to access the listing.
You may also return to Snowflake, log into your Snowflake Account (as an ACCOUNTADMIN),
Click on Data Products > Private Sharing > Shared with You, and you should see the Listing.
Replication of FME Remote Engines Service
1. To begin the installation of the application, click the Get button.
The Getting App Ready dialog will appear.
2. This will begin the replication from Snowflake AWS West Region to your region. Please allow up to two hours for this process to complete. Please monitor your email for notifications from no-reply@snowflake.net.
You will receive an email indicating that the App has been installed and is ready. This can take up to 30 minutes, depending on the region the application is being replicated to. Please wait for the App to replicate and for the email to be sent to the email address listed in the pop-up before proceeding to the next step.
3. Once you receive the email, you can revisit the Data Products > Private Sharing > Shared with You page and see that the listing is available.
Click on the Get button again.
If this is the first time installing a Snowflake App, you may be asked to complete your Snowflake Profile by providing your name and email address. If this happens, you must validate your email address with the link sent to your email.
4. Agree and continue to the required permissions for Snowflake Native Apps.
5. Click the Get button to agree with the terms and conditions and start the installation of the App to your account.
Optional: You may expand Options to see the name of the App that will be created.
6. Next, you should see confirmation that the App is being installed.
Next, there will be a confirmation that the App was successfully installed.
Congratulations, you have successfully installed the FME Remote Engines Service into your Snowflake Account.
Part 2: Deploying Remote Engines in Snowpark
In this part, we will deploy the FME Remote Engines Service into Snowflake’s Snowpark Container Services. Once deployed, the FME Remote Engines Service will be available to register with the FME Flow environment (Part 5).
1. Next, you can click on the Configure button in the previous step or go back to Data Sharing > Shared with You and under Privately Shared Listings, click on FME Remote Engines Service (Private Trial).
2. Click on the Expand Button to open the App
Review the External access integrations to see the two endpoints that will be accessed.
The integration provides access to the FME Licensing service at fme-licensing.safe.com:443 and a Simple Notification Service (SNS) on AWS, both of which are related to licensing. These two endpoints are the only external access the App has at launch time. They are required for the FME Remote Engines Service to process jobs.
3. Click on the Save button to dismiss the integrations dialog.
4. Click on the Activate button to activate the App.
You’ll be presented with the Activating FME Remote Engines Service (Private Trial)
This will take several minutes to complete. You can revisit this page later.
Wait for the Application to finish deploying. This will take around 10 minutes. You may revisit the Data Sharing > Private Sharing > Shared with You page.
- On the Shared with You page, click on the ‘Expand’ button to check the deployment's progress.
5. If this App has been deployed, you’ll be presented with this page.
You can also get to the App using the Catalog > Apps link
The App will be listed under "Installed Apps".
The presence of the Launch app button indicates your Application has deployed correctly.
Part 3: Accessing the Container
Continuing from Step 2, you’ll now have the option to launch the application. When you complete this step, you’ll be able to log into the Web UI of the FME Remote Engines Service.
1. If the Launch app button is now present, click it. If it is not available, please revisit the steps in Part 2. After launching the app, sign into your Snowflake account.
2. Next, the FME Remote Engines Web UI will be presented. Please log in with the default install credentials
- Username: admin
- Password: admin
Please note the URL in the browser address bar. This will be used in Part
5 to
register the FME Remote Engines Services.
Ex:
https://ktiros-mycompany-mycompany-az-east.snowflakecomputing.app/fmeserver/
It is recommended that you change the password to something more secure before connecting the FME Flow Core to the FME Remote Engines Service. You can do this by accessing the User Profile icon at the top right corner. When doing this, please enter a new password that matches the password policy on the FME Flow Core system.
Part 4: Granting Permissions to the Application
Next, in Snowsight, using the appropriate privileged user (ACCOUNTADMIN), it is necessary to grant access to the Application Role to a Role that the Service user will be assigned. The FME Flow Core will use this Service user to access the FME Remote Engines in Snowpark.
Example:
GRANT APPLICATION ROLE FME_REMOTE_ENGINES_SERVICE_PRIVATE_TRIAL.APP_USER
to ROLE SUPPORT_RL;
The Application Role FME_REMOTE_ENGINES_SERVICE_PRIVATE_TRIAL.APP_USER was created during the replication/launching of the FME Remote Engines service in your Snowflake Account.
If you are unsure of the Application name, you can run:
SHOW APPLICATIONS;
This will return the names of the Applications that are installed. The default name will be “FME_REMOTE_ENGINES_SERVICE_PRIVATE_TRIAL” unless changed in part 2, step 8.
SHOW APPLICATION ROLES IN APPLICATION FME_REMOTE_ENGINES_SERVICE_PRIVATE_TRIAL;
This will return the name of the application role and should be APP_USER.
With the correct Application Name and correct Application Role, the statement above will grant the APPLICATION ROLE to the role SUPPORT_RL (your role name will vary), which has already been granted to the Service user, SUPPORT_KEY. The Service user SUPPORT_KEY will be used in Part 5 to connect the FME Flow Core to the Remote Engines Service. Your Service user name will vary.
Part 5: Connecting FME Flow to the Remote Engines Service
Before beginning, ensure you have superuser permissions in FME Flow. Ensure you have previously created the service user with key pair authentication and correctly granted the appropriate application role to the service user in Part 4.
1. Log in to FME Flow as a Superuser
2. Create a new queue for the remote engines service. Go to Engine Management in the Web UI and click on the Queue tab. Create a new queue to assign engines to. You’ll assign this queue to the Remote Engines Service running in Snowpark.
Make a note of the queue name as it will be needed in Step 6.
3. Configure the Remote Engines Service. Go to Engine Management in the Web UI and click on Remote Engines Service.
4. Click on the Create or the Create Remote Engines Service Connection button. The following page will present.
5. Enter all details in the Create Remote Engines Service Connection page.
- Provide a new Name for the Connection
- Provide URL - the URL is the same URL as the App URL.
- Provide the Username and Password for the FME Flow Remote Engines as set in the Launching Container section unless changed since launching.
Next, provide the Snowflake Authentication details, and click the check box to enable the Snowflake Authentication.
- Provide Account (take from URL)
- Ex. mycompany-mycompany-az-east
- Provide Username
- Provide Role (this is the role used in Part 4, “Granting Permissions to the Application”)
- Endpoint is similar to the URL without https:// in the value.
- Ex. fme123-mycompany-mycompany-az-east.snowflakecomputing.app
- Private Key as generated.
- Public Key as generated and applied to the Snowflake User (Username).
Example Details
6. Select the Queue created in Step 2 above. Click on the Test button to ensure the information is correct.
You should see a Successful pop-up in the top right of the Web UI.
If you have any errors, please review the Troubleshooting section at the bottom of this article.
7. Next, click on the Save button to save the FME Remote Engines connection information. The Status will be set to “Initializing”.
After a few minutes, the Status will change to “Ready,” and you’ll be ready to test submitting a job to the FME Remote Engines Service.
Part 6: Testing the FME Remote Engines Service by Running a Job
Next, let’s test that we can submit a job to the Remote Engines. Ensure you run a Samples workspace and select the Job Submitter Service.
As seen in the screen grab below, ensure you set the Service to Job Submitter and in the Advanced section select the Job Queue assigned to the Snowflake Remote Engines Service. In this case, the example job queue is set to Snowflake_RE.
Once the job has been submitted, you can review the Jobs page to see the job request in the Completed Jobs tab. The Remote Engines Service Job log can be found at the REMOTE LOG URL as seen below.
When you submit a job to the Remote Engines Service, the FME Flow environment displays the following logging, indicating that the job has been submitted to the Remote Engines Service and where to review the remote job log.
The Job Log is only accessible using the FME Remote Engines Service Web UI.
You can now view the job submitted to the FME Remote Engines Service.
Reviewing Jobs submitted to the Remote Engines Service
When a job is submitted to the Remote Engines Service, the job request is passed to the Remote Engines Service App in the Snowflake environment. The FME Flow Core cannot view the job logs of the FME Remote Engines Service directly.
Therefore, to view job logs for jobs run on the FME Remote Engines Service, you must log into the Remote Engines Service App Web UI. This URL is the same URL presented in Part 3 and used in Part 5 to register the FME Remote Engines Service. You can also view the Job Log URL in the FME Flow environment by reviewing the job submission (see the previous section).
Log in to the FME Remote Engine Services and click on the Jobs page (left side of Web UI).
Click on the Completed Jobs to view the jobs that have run on the Remote Engines Service. From here, you can view the job status, view the job log, or download the job log.
Confirm that the Samples Job ran on the Remote Engines Service (as seen below).
You may click on the line item to view the Job Log.
A successful run confirms that the FME Flow environment can submit jobs correctly to the Remote Engines Service in Snowflake.
Part 7: Accessing Snowflake Data with the FME Remote Engines Service
For the FME Remote Engines Service App to access any data in the Snowflake Databases, it is necessary to grant the appropriate permissions to the Application. This may include:
- Databases
- Schemas
- Tables/Views
- Warehouses
- Stages
- Roles
Examples of some grants are as follows.
1. GRANT ALL ON DATABASE SUPPORT_DB TO APPLICATION FME_REMOTE_ENGINES_SERVICE_PRIVATE_TRIAL;
This grant allows the App to access the database, SUPPORT_DB.
2. GRANT ALL ON SCHEMA SUPPORT_DB.SUPPORT TO APPLICATION FME_REMOTE_ENGINES_SERVICE_PRIVATE_TRIAL;
This grant allows the App to access the schema SUPPORT in the database SUPPORT_DB.
3. GRANT ALL ON ALL TABLES IN SCHEMA SUPPORT_DB.SUPPORT TO APPLICATION FME_REMOTE_ENGINES_SERVICE_PRIVATE_TRIAL;
This grant allows the App to access the tables in the schema SUPPORT within the database, SUPPORT_DB.
4. GRANT USAGE ON WAREHOUSE SUPPORT_WH TO APPLICATION FME_REMOTE_ENGINES_SERVICE_PRIVATE_TRIAL;
This grant allows the App to make use of the compute warehouse, SUPPORT_WH
5. SHOW GRANTS TO APPLICATION FME_REMOTE_ENGINES_SERVICE_PRIVATE_TRIAL;
This statement allows you to review the grants that have been applied to the App.
Part 8: Authoring Workspaces for Remote Engines Service
As with all Workspaces that run in FME, it is necessary to develop the workspace with FME Form. Once you have tested the Snowflake Database Connection in FME Form and FME Flow, you are now ready to publish the workspace using the Snowpark Container Services authentication type. It will be necessary to ensure that, before publishing to FME Flow, any workspace you wish to run in FME Remote Engines Service in Snowflake, you swap the database connection to a Snowpark Container Services authentication type. This will be necessary for any Snowflake Readers, Snowflake Writers, SQLCreator or SQLExecuter using Snowflake, FeatureReader and FeatureWriters Transformers that are using Snowflake.
Note that the FME Remote Engines Service does not, by default, have access to external systems and services. To grant access to external systems and services, see Part 10.
Publishing and Running the Workspace
1. Publish the Workspace to the FME Flow environment as usual, ensuring the correct database connection is selected and uploaded during the publishing. Once the workspace has been published, it can be tested by running the workspace.
2. Ensure the correct Queue is assigned when submitting the job. When the job is submitted, the Web UI confirms on the Competed Jobs page that it has been submitted to the FME Remote Engines Service.
You can easily access the Remote Engines Service Web UI by visiting the Completed Jobs page, locating the job, and then clicking on the URL. This will take you to the login page for the FME Remote Engines Service Web UI, where you can review the job log.
Reviewing the job log for the remote engines was covered in Part 6.
Confirm the results of the workspace by opening the job log and confirming the correct database connection type was used. Example:
Snowflake Reader: Creating connection using connection string 'jdbc:snowflake://mycompany-az-east.east-us-2.azure.snowflakecomputing.com' and properties '{schema="SUPPORT", tracing=OFF, application=SafeSoftwareFME, GEOGRAPHY_OUTPUT_FORMAT=WKB, warehouse="SUPPORT_WH", authenticator=oauth, account=MYCOMPANY_AZ_EAST, db="SUPPORT_DB", GEOMETRY_OUTPUT_FORMAT=EWKB, token=************}'
In the connection string above, you’ll see the following connection parameters: authenticator=oauth and token. There will be no user parameter present. This confirms the workspace connected to the database from within the Application in Snowpark.
3. A successful connection will confirm that the FME Remote Engines Service has been able to connect to the database using the provided connection parameters.
Jobs will have different permissions requirements; some will be read-only, while others will have drop and create on tables. Regardless of the workspaces, the Application must be granted the necessary permissions to execute the jobs. Review Part 7 for more information.
Considerations
1. Creating a database connection for use in Snowpark
- When creating a database connection for use with the FME Remote Engines Service, ensure that the Authentication Type is Snowpark Container Services. Also, ensure that the Warehouse, Database, and Schema are all correctly set.
2. Do not set the Role in the database connection.
- The Snowpark Database Connection will rely on grants given directly to the FME Remote Engine Service Application, as explained in Part 7.
- The Container does not use a username or password to access the databases.
- Running a workspace with the FME Remote Engines Service that uses a Snowflake Database Connection with an Authentication Type other than Snowpark Container Services will fail and is not supported.
- In FME Form, you can swap a Database Connection by right-clicking on it and replacing it with the Database Connection that contains the Snowpark Container Services authentication type.
3. When authoring a workspace, consider that your client system will use a different database connection than when it is published to the FME Flow environment, where it will run on the FME Remote Engines Service within Snowpark. Therefore, permissions and privileges can vary; therefore, work closely with the Snowflake Administrator and DBA.
Part 9: Snowflake Database Connections with Remote Engines
When creating a database connection that will be used by the FME Remote Engines Service in Snowflake, ensure that you set the Authentication type to Snowpark Container Services, as seen below. This connection type is only compatible with the FME Remote Engines Service App in Snowflake.
NOTE: Leave the Role parameter empty.
Part 10: Modifying the Egress Rules
When launching the FME Remote Engines Service App, one Egress Access Integration is created. The integration provides access to the FME Licensing service at fme-licensing.safe.com:443 and a Simple Notification Service (SNS) on AWS, both of which are related to licensing. These two endpoints are the only external access the App has at launch time.
If other external endpoints are required, for example, another public API is needed to pull data from, it will be necessary to create a new network rule, a new external access integration, grant this access to the FME Remote Engines Service App, and call a procedure to enable the external access integration.
1. From within Snowsight, create a new network rule (run as ACCOUNTADMIN)
CREATE NETWORK RULE IF NOT EXISTS consumer_network_rule MODE = EGRESS TYPE = HOST_PORT VALUE_LIST = ('0.0.0.0:443');
Replace the value list contents with a list of DNS names/ports to authorize. These can be IP addresses or DNS names.
For example: ('10.10.10.1:443', 'arcgis.com:443')
For more information on creating network rules and value list examples, please refer to this link.
2. Next, create the External Access Integration (run as ACCOUNTADMIN)
CREATE EXTERNAL ACCESS INTEGRATION IF NOT EXISTS consumer_external_access ALLOWED_NETWORK_RULES = ('consumer_network_rule') ENABLED = TRUE;
GRANT USAGE ON integration consumer_external_access TO APPLICATION <APP_NAME>;
Where <APP_NAME> will be FME_REMOTE_ENGINES_SERVICE_PRIVATE_TRIAL (unless a different name was entered in Part 1, step 7).
3. Next, enable the External Integration
CALL <APP_NAME>.APP_PUBLIC.SETUP_EXTERNAL_INTEGRATION('consumer_external_access');
The External Integration can be altered in the future if required. More information can be found here.
Part 11: Deleting and removing the FME Remote Engines Service App
If you are required to upgrade the version of the FME Remote Engines Service App, you will need to delete the old App. This will also be required when upgrading the FME Flow Core. You may also need to delete it if you decide not to use it anymore.
Follow these steps to remove the old App:
1. Sign in as AccountAdmin to Snowsight Account, where your FME Remote Engines Service was deployed. Open a new worksheet. Ensure you are using the correct role.
2. List Applications
- Execute:
SHOW APPLICATIONS;
3. Expected return: FME_REMOTE_ENGINES_SERVICE_PRIVATE_TRIAL (unless a different name was entered in Part 1, step 7).
4. Drop the FME Remote Engines Application
-
Execute:
DROP APPLICATION FME_REMOTE_ENGINES_SERVICE_2025_1 CASCADE; -
Confirmation:
FME_REMOTE_ENGINES_SERVICE_2025_1 successfully dropped.
5. List External Access Integrations to be sure this has also been removed.
-
Execute:
SHOW EXTERNAL ACCESS INTEGRATIONS; - You should not see anything; however, if you encounter a line item such as
FME_REMOTE_ENGINES_SERVICE_2025_1_WEB_EAI_EXTERNAL_ACCESS, you will need to drop it manually (see the next step). It is expected that this integration will be dropped when the App is dropped in the previous step.
6. Drop External Security Integration if necessary
-
Execute:
DROP EXTERNAL ACCESS INTEGRATION FME_REMOTE_ENGINES_SERVICE_2025_1_WEB_EAI_EXTERNAL_ACCESS; -
Confirmation:
FME_REMOTE_ENGINES_SERVICE_2025_1_WEB_EAI_EXTERNAL_ACCESS successfully dropped.
7. Revisit the FME Flow Core environment and remove the FME Remote Engines Service from the Environment created in Part 4.
If you plan on reusing the service user for a new FME Remote Engines Service deployment, you do not need to delete the user referenced in Part 4. This user can be reused when deploying a new FME Remote Engines Service.
Troubleshooting Remote Engines Service in Snowpark
Can’t register Remote Engines
Example 1: Error 404 Not Found
The user must be assigned the correct role for this step to work.
GRANT APPLICATION ROLE FME_REMOTE_ENGINES_SERVICE_2025_1.APP_USER to ROLE FME_APP_ROLE.
This role, FME_APP_ROLE, is then assigned to the user being configured to access the Remote Engine Service.
The above grants will allow the FME Flow Core environment to connect to the Remote Engines Service and submit jobs.
Example 2: Error 404 Not Found
Probable causes are related to the URLs used to register the Remote Engines Service. Carefully review the URLs and the Account Name value used in Part 4, Step 3. The Account Name should be contained within the URL provided by the Snowflake App when you launched the FME Remote Engines Service App in Part 3: Launching Container.
Illegal Argument Exception: Illegal base64 character
This is likely caused by the private key requiring a passphrase or another issue related to decrypting the private key. Review how the public and private keys were created, ensuring that you followed the instructions in the Section above, called Requirements and Prerequisites, bullet 7.
Invalid Credentials
This error could be caused by a bad username and password provided for the connection to the Snowflake Account hosting the FME Remote Engines Service, as specified in the Snowflake Authentication panel in Part 5, step 5.
You may also encounter this error if you attempt to reuse an existing Remote Engines Service configuration with a new container instance. Please remove the configuration from FME Flow and recreate it from scratch. When creating a new configuration, several background configuration actions are set during the establishment of a new connection. When editing an existing connection, the same events do not run, and thus, the edited configuration will fail to connect to the new container.
Remote Job Submission Failed
When submitting a job to the Remote Engines Service, the 207 response is received, accompanied by the ‘can’t find key to decrypt’ message. Please remove the Remote Engines Service configuration from the FME Flow Core environment and recreate it.
Contact Us and Provide Feedback
As this is a new offering from Safe Software, please don't hesitate to contact our support team or your Safe Software Account Executive.
This article may contain variations in the included screenshots, or the Snowflake Web UI may look different. This may be related to changes and updates to the Snowflake environment. We intend to update this article as needed and welcome any comments.