Send Email from Automations

Introduction

In this exercise, we’ll set up a workflow where users email Shapefiles to FME Server for processing. We will reply to the sender’s email address if the attached file type is incorrect or if the file fails the first step of data processing. We will also send a responsible party an email with job results, log, and where to download the JSON files whenever the job runs.
The sample workspace reads in a zipped Shapefile and outputs GeoJSON of the input features, with output filename based on input. It is important to archive (zip) your Shapefile(s) (any single-file archive supported by FME will do) for use in automations because the attachment file path passed from the Email (Received) trigger to actions downstream will only point at the first attached file. If multiple files must be accessed from the same incoming email, you can create an Automation to download all files to a directory and then another Automation to watch for and route the files created there.
As an advanced exercise, at the end of this tutorial, learn how to use Global Keys within your automations to add more flexibility.
This exercise is a continuation of the Run a workspace in Response to Incoming Email article.
 

Requirements

• Completion of Run a workspace in Response to Incoming Email tutorial. 

Please note that the if you are importing the project as a non-admin user that you will need to have an Admin grant permissions to the project and contents before you can proceed. Please see Known Issue: Importing a fsproject File Requires an Admin to Grant Permissions to the Project for more information. 

Step-by-Step Instructions

Before you begin, please ensure you have completed Run a workspace in Response to Incoming Email as this tutorial has steps to step up the Email Trigger. The following tutorial cannot be completed without this trigger setup. 

1. Assemble an Automation
Log into FME Server, then go to Automations > Manage Automations. Open the Run Workspace From Email automation created in the previous tutorial. Stop the automation if it is running. 

 
 
2. Add a Filter
Click on the plus sign in the bottom left corner of the automation, and drag out another Action (orange). Connect the new Action component between the Email Trigger and the Run Workspace Action. Remove the direct connection between the original components by clicking on the connecting line and pressing the Delete key.

 
Once the Action has been connected, it can be configured as a Filter. A Filter Action tests whether incoming messages contain a specified string and outputs the messages that match the success port. All other messages are routed to the failure port.
Because the automation only passes along the file name and path of the first email attachment, we will need the workspace to determine whether what has been archived is truly a Shapefile. However, it is helpful to do the first pass here to filter out any wrong files before spending more FME engine time on them.
Our workflow requires that input data meet two criteria: one, the input must be an Esri Shapefile; and two, for the automation to pass a reference to the Shapefile reliably, it must be bundled as a single archive. We will configure this exercise to pass .zip archives, but additional filters could be added downstream of the first filter’s failure port also to route e.g., .shz or .7z archives, into the workspace. If a Shapefile is emailed as a set of separate constituent files (.shp, .shx, .dbf…), whichever was stored first by FME Server will be the only file path read into the automation.

Note: As of FME Server 2020.0, there is now an email.attachments Output (Event) Key, which creates a list of the attachments. This list will need to be processed within an FME Workspace first before processing it further in FME Server. See the Advanced Exercise for details on how to complete this. 
 
Change the Action to Filter if you haven’t done so already. Then click the drop-down for the Key field and select Email Attachment. This setting corresponds to the email.attachment output key of the Email Trigger. In the Contains field, type .zip (omit wildcard characters). Click Apply. 

 
 
This workflow will proceed smoothly if the input file is structured as expected, but we don’t want users to send files off and get no feedback, nor do we want a process administrator to have to track down every user that submits bad data manually.
 
3. Send the User an Email if Attachment File Type is Incorrect
Now that we have our workflow in place, let’s automatically address some of the most common areas the process might break down. The first problem case is where the user emails an incorrect file type (or fails to archive the Shapefile components to a single file).
Drag out an External Action component and connect it downstream of your Filter’s failure port.


In the parameters, set the Action to Email (send). All emails sent will be via the SMTP protocol through an external account. Get SMTP Server parameters from your email provider, or use Load Template (as below) to fill in the blanks for some common providers.
 

SMTP Account (optional) and SMTP Password (optional) are required if the email server requires authentication (this is likely). Enter the details of an account you can check.
 
Click the drop-down arrow button to the right of Email To and select Email > Sent From Address. This will direct the reply email back to the address that sent the triggering email. Under Email From, enter the same address you used in the SMTP Account (optional) above.


It’s ok if all of these email fields use the same address (Just make sure the subject lines of all emails are distinct enough that you can troubleshoot!).
Type an Email Subject like “Incorrect attachment sent to FME Server.”
Finally, to generate an informative message to the user who submitted a rejected email attachment, click the drop-down arrow to the right of the Email Body field and select Text Editor.
 

 
You may click automation Output Key values on the left to add them to a typed email body on the right, like so:


 
Paste-able version:

Dear Data Provider,

The file ({email.attachment}) emailed from this address to {email.to} under the subject {email.subject} for shapefile processing did not match the file input requirements.

Be sure that you have attached a shapefile as a .zip archive and that no other attachments are sent.

Thank you,
Zipster

 
When you have composed an email body, click OK, then Validate and Apply. If your parameters don’t validate, check your email account and password.
You are now prepared to automatically inform users if they submit an unexpected file type to this workflow! Next, we’ll notify the user if the file submitted was a valid single-file archive, but the processing workspace failed (likely because the archived content did not contain complete Shapefiles).
 
4. Send the User an Email if the Processing Workspace Fails
Let’s duplicate the Email external action we just created. Single-click on the Email action to highlight it, then copy and paste it. (ctrl+c / ctrl+v). Connect the duplicated Email action to the failed output port on the ShapefileToGeoJSON.fmw workspace action. 
 

 
This email will also be directed at the user. Still, it must be distinct from the previously configured one (otherwise, we would just drag another connection to the same Email component). Change Email Subject to FME Server failed to process an emailed file. 
We could also use the Text Editor here to add in some specifics, such as the subject line of the sender’s original email.


For the Email Body, again, use the Text Editor to combine values from the upstream messages in the automation and static text to inform the user about what happened without going into unnecessary detail. For example:
 

Dear Data Provider,

The email sent from this address ({email.from}) at {email.sent} to {email.to} contained a .zip archive that was unable to be processed by FME Server. Please check that the .zip contains all the component files of any shapefiles present, and resubmit as necessary.

Contact ResponsibleParty@domain if you require assistance.

Thank you,
Zipster

 
Click OK at the bottom of the editor, then Validate and Apply.
 
We have empowered the user to troubleshoot file submission issues without tying up administrator time! Of course, we still want the person responsible for administering this data workflow (you) to be alerted when new data arrives (especially when there is a problem with translation). As the party responsible for managing the automation, you will also benefit from more detailed information.
 
5. Send the Process Administrator an Email When a Job is Run
In this step, we will configure a third and final email, this time to ourselves as the manager of this particular automation. This email will include status details and an attached log for every job submitted by the automation so that we know at a glance when new data has been successfully integrated and when a job failure may need attention.
As in the previous step, copy your Email external action to avoid re-entering email account information. Now, connect it to both the success and the failure ports of the ShapefileToGeoJSON.fmw workspace action.


Whenever the workspace runs, we want to be informed, so change Email To from Sent from address to your own address (even if you are coincidentally sending and receiving with the same account). We will format the email to make it easy to tell at a glance when the job was successful and when a failure may require some troubleshooting.
 
Use the Text Editor to change the Email Subject to New Results from {automation.name}.
 
Use the drop-down arrow to populate Email Attachment (optional) with Workspace > Job Log. Now we can review a job’s log immediately, without logging into the FME Server web interface.


For the Email Body, use the Text Editor to compose a message with enough detail for troubleshooting, but that can be read at a glance if the job was successful. For example, try pasting in:
 

Dear Me,

You are informed that at {time}, {automation.name} was triggered by the receipt of {email.attachment}. Job {job.id} was initiated in response, and concluded at {job.timeFinished} with status

{job.status}

and message "{job.statusMessage}".

The job log for {job.workspace} is attached for review.

Have a lovely day,
You

 
Click OK, then Validate and Apply.

 
Great! We have a data transformation workflow set up to empower the end-user to troubleshoot many data submission issues and inform the workflow manager of new data and possible issues as they arise. The only thing left is to test that it works.
 
6. Save Your Work and Start the Automation
Click Menu above the Automations canvas, then Save As. Choose a name and add some descriptive tags, then click OK. Start Automation in the upper right.
 
7. Test Email and View Log
We have three cases to test:

1. The user submits a file that is not a .zip. The user is emailed requesting that they check their file attachment and resubmit.
2. The user submits a .zip, but it contains either an incomplete Shapefile or solely non-Shapefile content. The user is informed that the process failed (and again is encouraged to verify the submitted file format), and you are notified that the workspace failed.
3. The user submits a .zip of complete Shapefiles. You are informed that the process has run successfully, and new output data is available.

 
You may use the same email account to test all messages involved. Just be sure that you have configured distinct content for the emails (as above). Do the following:

1. Send an email to the address you are monitoring with Shapefile somewhere in the subject line. Attach to it any arbitrary file that isn’t a .zip.
2. Send an email to the address you are monitoring with Shapefile somewhere in the subject line. Attach to it a .zip of non-Shapefiles. In a Windows environment, you can zip up files by selecting them in the file explorer (ctrl + click), then right-click > Send to > Compressed (zipped) folder.
3. Send an email to the address you are monitoring with Shapefile somewhere in the subject line. Attach the zipped Shapefile from the Downloads section above or any other .zip containing a Shapefile.

 
Wait for at least one polling interval, then check the Automation Log. One way to do this is through Menu > View Log File when your automation is displayed on the screen. You can also navigate to Automations > Manage, check the automation’s box, and select Actions > View Log File. It can be helpful to click the clock icon above the log to show timestamps for each log entry.
 
In the Automation Log, you should see entries that look something like the following:
 
For failure at the filter:



For failure in the workspace:


 
For success:


 
For individual job logs, when your automation canvas is displayed, go to Menu > View Triggered Jobs.
 

Advanced Exercise

In this advanced exercise, you will take the automation from above and add a Global Key. Global Keys are JSON formatted key-value pairs that can travel between components in an automation. For more information or a beginner example, please see Working with User and Global Keys in an Automation.
 
1. Create Global Key
Continuing in the same automation from the previous section. Before you begin, stop the automation so it can be edited. Now we need to create a Global Key that will be used throughout the workspace. On the top menu bar inside the automation, click on the globe icon.

 
Click on the plus sign in the Global Keys Editor to create a new Global Key. Set the Global Key to AttachmentFolder and the Value to:

$(FME_SHAREDRESOURCE_DATA)/Attachments/

 
Click OK to save the Global Key. This key can then be used anywhere in the automation to refer to that folder path.

 
2. Apply Global Key to Email Attachment
Now we need to apply the Global Key. Open the Email IMAP (or SMTP) Action parameters. For Download Attachments To, click on the drop-down arrow then expand Global, and select global.AttachmentFolder.  

 
 
When an email is received with an attachment, all attachments will be downloaded and saved to the folder location set up in the Global Key. This is a useful method for larger automations where the folder location might change frequently.

 
3. Set Up Workspace
We can also apply our Global Key to the workspace as a folder location for the processed JSON files. Open the ShapefileToGeoJSON Workspace Action parameters, and click the drop-down next to Destination GeoJSON File. This time select the Text Editor.
 
In the Text Editor, erase any existing value, then on the side menu, expand Global and then select global.AttachmentFolder. Now we can append any folder names we wish, and they will be created. After {global.AttachmentFolder} add:

/ProcessedFiles/GeoJSON

 
So the value in the Text editor should read:

{global.AttachmentFolder}/ProcessedFiles/GeoJSON

 
Now click OK to close the Text Editor, and then click Apply to save the changes to the Workspace Action.
 
4. Save and Start the Automation
Save the automation and then start it. Once started, send an email containing a correct Shapefile and then view the newly processed GeoJSON file in the folder we set up with the Global Key:

$(FME_SHAREDRESOURCE_DATA)/Attachments/ProcessedFiles/GeoJSON

Conclusion

Congratulations! You created an automation that transforms emailed data and sends emails to the data submitter or workflow manager when further action may be required. You’re ready to add email notifications with automated messages to your own workflows!
 

Data Attribution

The data used here originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.