Using Python with FME Flow Hosted

Introduction 

Python is a programming language that can be used within FME Flow Hosted (formerly FME Cloud) to accomplish tasks either before or after FME runs or to perform tasks within FME that are not possible with standard FME tools and transformers.
This article will dive deeper into an overview of Python with FME Flow Hosted along with some important FME Flow Hosted notes to consider. It will cover the standard and non-standard Python libraries in FME Flow Hosted along with uploading non-standard Python libraries to FME Flow Hosted. 

 
Note: Before getting started, users are highly advised to review the Python and FME Basics article series, as it provides a lot of details needed to better comprehend this article. 
 

Using Python with FME Flow Hosted

FME Flow Hosted runs on Ubuntu (Linux); therefore, it comes by default with different versions of Python. 

To check the current compatibility of FME Flow Hosted with Python, navigate to Resources > Engine > Plugins > Python. This should allow you to view the current Python versions available. See the Python Compatibility documentation for more information. 

There are four main places where Python can be used in FME:

• Startup Python Script
• Scripted Parameter
• PythonCaller and PythonCreator Transformers
• Shutdown Python Script

*Important Note: There are numerous FME transformers for performing almost every task you can think of without resorting to scripting. If you are thinking about writing a Python script to do something in FME, you should make absolutely sure there is not an existing transformer available already. Please feel free to ask the FME Community.
 

Pre-installed Libraries

FME Flow Hosted comes with pre-installed third-party libraries, for an up-to-date list of these libraries, please see the FME Flow Hosted 3rd-Party Library Support article. 
 

Uploading a Custom Python Library to FME Flow Hosted 

1. Identify and download the custom Python library you want to add to FME Flow Hosted.
2. Navigate to the FME Flow Hosted Python library (Resources > Engine > Plugins > Python)
3. Upload the downloaded custom library to the FME Flow Hosted directory 
4. After this is done, restart the FME engines (You can do this by navigating to engine management, then set the number of hosts to 0 before setting it back to the previous engine count or you can just reboot the entire instance)

 
Note: FME Flow Hosted does not support the Python interpreters included with ArcGIS products. ArcPy cannot be used on FME Flow Hosted.
 

Step-by-Step Instructions

In the example below, we will be uploading a custom Python library called “Pyfiglet” and then calling it. 

Note: If you have file paths hardcoded in your configuration, FME Flow Hosted will not be able to access this resource so you will need to upload data to the Resources > Data > <new folder> and then change your Python to look in that location.  

We recommend using FME Flow (formerly FME Server) parameters to reference where your data is. Once your data is uploaded to FME Flow, you can select one item and view its properties. This will give you the path to your data, and you can use it within your Python script. The FME (Flow) parameters can be resolved in FME Python using the syntax:

fme.macroValues['<paramName>']

For Example:

fme.macroValues['FME_SHAREDRESOURCE_DATA']

We recommend making sure that all of the files are present in Pyfiglet\_libs\tslibs 
 

Part 1: How to Install the Pyfiglet Library 

1. Download Library
Identify and download the Pyfiglet library you want to upload to FME Flow Hosted
Note: We advise using pip to download Pyfiglet. 
In the example below for Mac and Windows, we will be downloading the files to our FME Form (formerly FME Desktop) Python directory.

On Windows:

fme.exe python -m pip install Pyfiglet --target C:\Users\<user>\Documents\FME\Plugins\Python\python38

On Mac:

Pip3 install Pyfiglet -–target /users/<user>/Library/Application\Support/FME/Plugins/python/python38

2. Navigate to Directory
Navigate to the FME Flow Hosted Python directory (Resources > Engine > Plugins > Python > Python38).

3. Upload Library
Upload the Pyfiglet folders content from the directory where it was downloaded to the FME Flow Hosted Python directory
Windows: 

C:\Users\<user>\Documents\FME\Plugins\Python38

Mac:

/users/<user>/Library/Application\Support/FME/Plugins/python/python38

Note: You can also follow the steps in the documentation or copy the Pyfiglet files from the location where it was downloaded and paste the contents to the FME Flow Hosted Python directory 

Once uploaded, it should look like this in FME Flow Hosted:
 

4. Restart Engines
After uploading, restart the FME engines. You can do this by navigating to the Engine Management page, then setting the number of hosts to 0 before setting it back or you can reboot the entire instance. 
 

Part 2: Using the Pyfiglet Library in FME Flow Hosted

In the example below we will run a workspace to print  “Hello” to a log output 


1. Open FME Workbench
Create a new workspace in FME Workbench. Add a Creator and a PythonCaller transformer to the canvas. Connect the PythonCaller to the Creator. 

2. Open PythonCaller Parameters
Open the PythonCaller parameters. Copy then paste the following script: 

import fme
import fmeobjects
import Pyfiglet

class FeatureProcessor(object):

    """Template Class Interface
When using this class, make sure its name is set as the value of the 'Class
  to Process Features' transformer parameter.

    ""
def __init__(self):

        """Base constructor for class members.""
            text = Pyfiglet.figlet_format("Hello")

        print(text)
        pass

    def input(self, feature):

        """This method is called for each FME Feature entering the 
PythonCaller. If knowledge of all input Features is not required for 
processing, then the processed Feature can be emitted from this method 
through self.pyoutput(). Otherwise, the input FME Feature should be 
cached to a list class member and processed in process_group() when 
'Group by' attributes(s) are specified, or the close() method.

:param fmeobjects.FMEFeature feature: FME Feature entering the 

  Transformer.
"""
  self.pyoutput(feature)

 def close(self):

        """This method is called once all the FME Features have been processed
from input().

 """
  pass

   def process_group(self):

        """When 'Group By' attribute(s) are specified, this method is called 
once all the FME Features in a current group have been sent to input().
  FME Features sent to input() should generally be cached for group-by 
processing in this method when knowledge of all Features is required. 
The resulting Feature(s) from the group-by processing should be emitted 

        through self.pyoutput().

 FME will continue calling input() a number of times followed

        by process_group() for each 'Group By' attribute, so this 

        implementation should reset any class members for the next group.

        """

        pass

  def has_support_for(self, support_type):

        """This method returns whether this PythonCaller supports a certain type.

        The only supported type is fmeobjects.FME_SUPPORT_FEATURE_TABLE_SHIM.

        

        :param int support_type: The support type being queried.

        :returns: True if the passed in support type is supported.

        :rtype: bool

        """

        if support_type == fmeobjects.FME_SUPPORT_FEATURE_TABLE_SHIM:

            return False

         return False

3. Publish the Workspace to FME Flow Hosted
Save the workspace, then publish the workspace to your FME Flow Hosted Instance. 

4. Run the Workspace 
Run the workspace in FME Flow Hosted, then click View Details to view the "Hello" in the log, as seen below:

Additional Resources 

• Importing Custom Python Modules to FME Flow
• FME Flow Hosted 3rd Party Library Support
• Python Compatibility
• Using Python with FME Flow
• Startup and Shutdown Python Scripts