Using a Private SSH key with FTPCaller to Authenticate an SFTP Connection for FME Form and FME Flow on Windows Systems

Introduction

SFTP systems may support authenticating through the Secure Shell (SSH) protocol via SSH keys instead of a user and password. The FTPCaller transformer supports pushing and retrieving data to/from a target SFTP server using identity keys (private keys) when the Authentication Type parameter is set to ssh-agent / pageant (sftp only).

On Linux and macOS systems, this process is supported by ssh-agent helper application, which is usually already installed and may be running by default. On Windows, users can either use the PuTTY authentication agent application Pageant or ssh-agent (bundled as part of OpenSSH for Windows) if the client machine is Windows Server 2022, Windows Server 2019, or Windows 10 (build 1809 and later).

This tutorial will contain steps required for FME to establish a SFTP connection via private SSH keys for both FME Form (formerly FME Desktop) and FME Flow (formerly FME Server) on a Windows system.

Step-by-step Instructions

Option 1: PuTTY's Pageant

Pageant is part of the free and open-source PuTTY suite and can be downloaded from here. Ensure the application is installed before attempting the steps in this section.

1. Make or have your IT create an SSH key pair on a Windows system. You can do this with the PuTTYgen application (PuTTY's key generator). For this tutorial, use RSA format for the keys.
If you plan to automate the loading of the private key into ssh-agent in an environment where user-input is not possible, such as FME Flow, the key pair cannot be generated with a passphrase.
 

2. Edit the public key to remove the header and comments for an RSA formatted key.  Your public key should look like this:

ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAj90HZMbr8gx/7asV0KECuKGkT7UwjnkL2LPVQqpDKNsC6fl1t1hTIp8Ngy0RBrr7NFZDabcbxlLvwhkrcBIUWDSn3EDriu3z/dS8nFck4a1clQZILc3tXMK34wyZsKAOoCFIC9va86Vwl/xSF4ofT1Jw+vlJn3A9uevT7swJaaOk4hWkZYn8jZ/2T2AfUyxZ0WRfaRZj2t5RdJhAefzlkhbMbmoAXtqEKXJkJMgVRWv9PZgRht/Gbw+O8gGnwMI2mhpVqc4O1wvj1oeo3crg8MZHecyjYCOrbBY6uD/SEsijQEeADfe4EYe7CfP/GlEC5EliiaqdoYbxacav3ookgQ==
  rsa-key-20210917

3. Upload the public key to the SFTP server and append the public key to SSH authorized_keys file for the user you are going to connect as. For example, if you are connecting to the SFTP server as the user sftpUser, the public key needs to be appended to sftpUser's authorized_keys file and not sftpUser2.

4. Open FME Workbench

5. Add a Creator and a SystemCaller to FME to launch Pageant and load the private keys.
Note: if preferred, the step of launching Pageant and loading the private key can be done manually on FME Form.

Modify this command to point to the Pageant.exe located on your machine if the installation path is different and the location of your private key, then add the following line in the SystemCaller:

start "" "C:\Program Files\PuTTY\pageant.exe" --encrypted C:\Users\rmosley\Desktop\safe4.ppk

Note: the --encrypted command option adds a private key to Pageant without decryption and works for keys created with and without a passphrase. If the key was created with a passphrase, Pageant will prompt for the passphrase using a dialog when the client application (eg. FTPCaller) attempts to use the key.

6. Add an FTPCaller to the workflow and change the authentication type to use ssh-agent / pageant (sftp only) and enter the SFTP username. Enter the URL to the folder or file you want to access on the destination SFTP server.

7. You should now be able to access the destination SFTP server via SSH key authentication when the workspace is run.

Note: It is recommended to remove any loaded private SSH keys once it is unneeded to reduce security risk. All loaded private keys can be deleted by killing Pageant with the following command in a SystemCaller at the end of the workspace:

taskkill /IM pageant.exe /F

Option 2: OpenSSH for Window's ssh-agent

OpenSSH for Windows is now supported on certain Windows versions such as Windows Server 2022, Windows Server 2019, Windows 10 (build 1809 and later). In order for a client machine to connect to a destination SFTP server using the SSH protocol, OpenSSH Client must be installed on the client machine.

OpenSSH Client is an optional feature and may not be installed by default. Ensure it is installed by going to Settings > Apps & features > Manage Optional Features before attempting the steps in this section.

1. Make or have your IT create an SSH key pair on a Windows system. You can do this with the ssh-keygen application (OpenSSH’s key generator). Please see here for more details on user key generation. The public key will have a .pub file (eg. id_rsa.pub) and the private key will not have a file extension (eg. id_rsa).
If you plan to automate the loading of the private key into ssh-agent in an environment where user-input is not possible, such as FME Flow, your key pair cannot be generated with a passphrase.

2. Upload the public key to the SFTP server and append the public key to SSH authorized_keys file for the user you are going to connect as. For example, if you are connecting to the SFTP server as the user sftpUser, the public key needs to be appended to sftpUser's authorized_keys file and not sftpUser2.

3. By default, OpenSSH for Window’s ssh-agent service is disabled and does not start automatically. Configure ssh-agent to start automatically by running services.msc as an Administrator to open Windows Services dialog window and changing the OpenSSH Authentication Agent service Startup Type to Automatic.

OpenSSH Authentication Agent is disabled on startup by default.
 

Manually configure OpenSSH Authentication Agent to start automatically.

4. Either restart the client machine to start ssh-agent or manually start ssh-agent by running the command "ssh-agent" in a Powershell or Command Prompt window.

5. Open FME Workbench

6. Add a Creator and a SystemCaller to the workspace.
Note: If preferred, the step of loading the private key can be done manually if the workspace is only to be run on FME Form.

Modify the following command to the location of your private key, then add the line in the SystemCaller.

ssh-add C:\Users\rmosley\Desktop\safe4

7. Add an FTPCaller to the workflow and change the authentication type to use ssh-agent / pageant (sftp only) and add the SFTP username. Enter the URL to the folder or file you want to access on the destination SFTP server.

8. You should now be able to access the destination SFTP server via SSH key authentication when the workspace is run.

     Note: OpenSSH for Window permanently stores loaded private keys in the Windows registry as noted here. So private key files remain loaded even after closing FME Form or restarts of the machine where FME Form is installed. It is recommended to remove loaded private key when the private key is not required to reduce security risk.

Delete all identities from ssh-agent with the following command in a SystemCaller at the end of the workspace:

ssh-add -D

If you have access to the private key file’s corresponding public key, it is possible to use the following command to only remove the identity associated with the private key instead.

ssh-add -d \\path\To\Public\Key\File.pub

Troubleshooting

Unprotected private key file warning when adding SSH key to ssh-agent

You may run into the following error when adding your private key to ssh-agent:
Warning: Unprotected private key file!
Permissions for 'C:\path\to\private\key' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.


To resolve this error, right-click on the private key and select Properties from the drop-down menu. Select the Security tab and select the Advanced option.


Disable inheritance and select ‘Convert inherited permissions into explicit permission on this object’ in the resulting dialog.


Select the Users group entry and remove from permission entries.


The results should look similar to this after performing the above steps.


You should now be able to add the private key successfully. Depending on your configuration, you may need to remove more groups from the permissions entries table in Advanced Security Settings in order for the private key to be successfully added.

FTP info: No identity would match

The FTPCaller fails with the following error: "FTP info: No identity would match" after the SystemCaller loads the key successfully.

This error can be seen if both Pageant and ssh-agent is running and the SSH key is added to ssh-agent. The FTPCaller checks Pageant for identities when both ssh-agent and Pageant is running. Shutdown Pageant and ensure only ssh-agent is running.

FME Flow

The workspaces can be published to FME Flow and run. There are a few restrictions and recommendations on FME Flow:

• Key pairs generated with a passphrase cannot be used with FME Flow as users cannot interact with the passphrase input dialog on Flow.
• There may be caching issues affecting key loading behaviour on FME Flow if SystemCaller continues to fail loading the SSH key. Try rebooting the machine FME Flow is installed on instead of restarting the Flow instance.
• It is recommended to remove any loaded private SSH keys once it is unneeded to reduce security risk. See the relevant sections below for suggestions on how to delete identities.

PuTTY's Pageant

• Manually launching and adding the keys will not work on FME Flow even if you run Pageant as the FME Flow Service Account. You must load the keys with the SystemCaller within a workspace run by FME Flow.
• Restarting the engines will remove the private keys for that process and Pageant must be restarted and loaded with the correct keys. However, the automatic restart of the engine processes handled by FME Flow will not affect the SSH keys.
• Delete all identities from pageant by killing Pageant with the following command in a SystemCaller at the end of the workspace:

taskkill /IM pageant.exe /F

OpenSSH for Windows

• In FME Flow, warnings about unprotected private key files are logged in the fmeprocessmonitorengine.log, located at Resources > Logs > engine > current.
  • If running FME Engines with the default local system account and the unprotected private key file appears, try changing the SSH key’s owner to the server machine's Administrators group.
  • If running FME Engines under a different account and the unprotected private key file warning appears, please ensure that the account running your FME Flow services (Core, Web Application, and Engine) is the same as the owner of the private key.
• OpenSSH for Window permanently stores loaded private keys in the Windows registry as noted here, so key files remain loaded even after manual restarts of FME engines, FME Flow, or machine where Flow is installed. It is recommended to remove loaded private key to reduce security risk.
• Delete all identities from ssh-agent with the following command in a SystemCaller at the end of the workspace:

ssh-add -D

• If you have access to the private key file’s corresponding public key, it is possible to use the following command to only remove the identity associated with the private key instead.

ssh-add -d C:\path\To\PublicKeyFile.pub