FME Flow Troubleshooting: Proxies

Full Guide: FME Flow Troubleshooting Guide

Are you encountering issues connecting to FME Flow through a proxy server?

A proxy server is a computer on your LAN that allows you to connect to external servers without compromising the security of your internal network. If you are consuming FME Flow Web Services and your organization uses a proxy server on your network, you can configure FME Flow to connect through the proxy.

A HTTP Status 404 Not Found error can indicate that you need to set up FME Flow for your proxy server. Please read below for common troubleshooting tips, questions, and resources.

Initial Troubleshooting

Confirm the following steps from the documentation :
- Check that the proxy has been set up for the account you wish to use (Local, Non-Local)
- Restart FME Flow Engines
- Check that the engines are aware of the proxy server
Confirm the authentication mechanism being used by the proxy setting configuration (NTML, Basic, or Digest).
When executing the provided commands, they must be run as an Administrator.

Common Issues

“I want the FME Flow Engines to use a proxy but not the system-wide proxy.”

In this linked documentation, we mention how to add a custom proxy. This will override any system proxy that may be set.

The following syntax specifies how to map access to one or more URLs through a custom proxy. Run the following command, repeating all parameters for each URL you want to map through the custom proxy, as shown:

fme.exe APPLY_SETTINGS SYSTEM "Proxy/Custom Proxy Map" "<Source_URL>,<Proxy_URL>,<Proxy_Port>,<Requires_Authentication>,<UserName>,fme_encrypt(<Password>),<Authentication_Method>[,<Source_URL>,<Proxy_URL>,<Proxy_Port>,<Requires_Authentication>,<UserName>,fme_encrypt(<Password>),<Authentication_Method>]"
Where:

<Source_URL> is the URL you want to access. The URL must be an exact match with the URL in the request; for example, http://www.lib.uwaterloo.ca/locations/umd/digitization_project/Roads_NTDB.zip.
<Proxy_URL> is the URL of the proxy server on your system that will handle requests to the <Source_URL>.
<Proxy_Port> is the port number on the proxy server through which requests are handled.
<Requires_Authentication> is true or false, depending on whether the proxy server requires authentication to access. If true, provide <UserName>, <Password>, and <Authentication_Method> (Basic, Digest, or NTLM).

You'll notice the use of SYSTEM in the syntax above. This controls where in the registry the reg key and value are stored - under HKEY_LOCAL_MACHINE [HKLM] and is associated with the FME Flow software and will only affect the FME Flow and the proxy it will use.

“Instead of defining an FME-wide proxy address, I would like it to be set in the HTTPCaller, so different HTTP Callers could use different proxies.”

We do not have the ability to define proxy settings per workspace, per repository, or per feature. An alternative could be to define an exclusion list using wildcards for any internal URLs.

“In FME Flow, I am trying to use a basic proxy and version control but I am getting the error below. How do I fix this?”

Could not push to remote repository._3586 Remote ref update was rejected: https://github.com/test.git: 407 Proxy Authentication Required

To be able to push to an external Version Control while using the Basic Proxy, the following configurations must be made:

Open <FMEFlowDirectory>\Server\processMonitorConfigCore.txt
Look for # Start FME Flow Core
Add this to the START SYSTEM variable: -Djdk.http.auth.tunneling.disabledSchemes=
Place this string anywhere near the -DJava arguments
Restart Server

“HTTPCaller and FMEFlowJobSubmitter not respecting proxy exceptions if Python is used in the workspace”

If you have enabled a proxy in the UI and set up exceptions for the FME Flow, FME Flow will respect them, unless there is Python in the workspace.

If you open fmeFlowConfig.txt, under the heading FME SERVER SETTINGS START, add the following line: ENABLE_ENGINEPROXY=false

“How can I run jobs when my FME Flow Core cannot send traffic to my Load Balancer directly?”

FME Flow can be configured with a proxy server to enable the Web Application/Core/Engines to send outgoing traffic. However, the FME Services used to submit jobs were designed to remain internal, so they are not included in this proxy configuration. In some installations, your Load Balancer may be able to send traffic to the core, but the core may not be permitted to send traffic back to the Load Balancer. In this case, you will be unable to submit jobs via the run workspace page.

The file ‘hosts’ is an operating system file that maps hostnames to IP addresses. For each core machine, add a line to the host file mapping the Load Balancer's hostname to the current machine. The mapping consists of two values: “127.0.0.1”, which will reference the local machine, and the name of the load balancer. This entry will allow the calls needed to submit a service (run a job) to stay local to the machine.

On Windows, this file can be found at C:\Windows\System32\drivers\etc\hosts.

For Linux, it is /etc/hosts.

Example line to add to hosts file: 127.0.0.1 MyLoadBalancerName

"I'm seeing authorization and permission errors when I have FME Flow transformers in my workspace"

In certain proxy environments, FME Flow fails to run FME Flow transformers. The error message returned may look something like this:

FMEFlowNotifier: Unauthorized request by user due to lack of proper permissions for connection: <My FME Flow Connection Name>. Please see: http://fme.ly/zn8 FMEFlowLogFileRetriever: user not authorized to perform this action (HTTP 403)

To resolve this, go into FME Flow proxy settings (System Configuration > Network & Email > Proxy) and create an exception for the FME Flow URL using the fully-qualified domain name. In the "No Proxy For" parameter, include, for example: <hostname>.<domain>.* *.<domain>

"I've set up FME Flow to use a reverse proxy and when I try and run jobs I receive a timeout error"

By default, a reverse proxy is configured with a connection timeout; if an open connection exceeds this timeout, the connection will be dropped. When running a job under one of the services where the web UI must wait for the job to complete to receive a response (e.g., Data Streaming or Data Download Service), the job maintains an open connection. Therefore, if the job duration exceeds the reverse proxy's connection timeout, the reverse proxy will not be able to return a response, and the Web UI will display an error. Depending on the Reverse Proxy (or Load Balancer/Application Gateway, etc) in question. The error code and message will differ. Note that even if this error occurs, the job doesn't actually stop running and will be reported as completing successfully in the Job history.

To confirm this is the problem, you can create a workspace with a Creator > Decelerator and set the delay on either side of the configured connection timeout. E.g., if it is 2 minutes by default, create one workspace with a 110-second delay and a second workspace with a 130-second delay. Publish the workspaces under the Data Streaming service, and then run each of them on FME Flow, and you should only observe the timeout error on the longer job.

To resolve this problem, increase the client-side connection timeout to exceed the job duration.

Additional Resources

Are you still experiencing issues?

Please consider posting to the FME Community Q&A if you are still experiencing issues that are not addressed in this article. There are also different support channels available.

Have ideas on how to improve this?

You can add ideas or product suggestions to our Ideas Exchange.

Search