Including Files with the Data Download Results

Introduction

When using the Data Download service in FME Flow, additional files can be added. In this tutorial, we configure adding an FME Flow job log and additional files with our download. Attaching the job log can be done using the transformer or an HTTPCaller transformer connected to the FME Flow REST API V4.

Step-by-step Instructions

Example 1: Including the Job Log File using the FMEFlowLogFile Retriever

The FMEFlowJobLogRetreiver transformer has been deprecated as of FME 2024.1. If you’re using FME 2024.1 or newer, see Example 2: Including the Job Log File using the FME Flow REST API V4.

1. Open Template Workspace

In FME Workbench, open the Download_Logfile-Begin.fmwt workspace. This workspace writes a test message into a text file.

2. Add and Configure the FMEFlowLogFileRetriever Transformer

Add an FMEFlowLogFileRetriever transformer to the canvas and connect it to the Creator transformer.

In the parameters, set the Connection to your FME Flow web connection. See Connecting to FME Flow for instructions on how to create the web connection if you haven’t done so. 

Then set the Job ID to the FME Flow Job ID Parameter $(FME_JOB_ID). The parameter can be selected by clicking on the drop-down next to Job ID then going to User Parameter > FME Flow Parameters > FME_JOB_ID.

Finally, change the Output Log Attribute to text_line_data. We need to change this attribute so that our writer, in the next step, can read it. Click OK to save the parameters.

3. Add a Text File Writer

Now that the log file has been retrieved, we need to write it out. Add a Text File writer to the canvas. For the Dataset, browse to where you would like it to be saved, then name it job_log.txt, we will modify the filename in the next step.

Once the Text File writer is added to the canvas, connect it to the FMEFlowLogFileRetriever transformer.

 4. Create Fanout

When the workspace runs on FME Flow, the data download file will contain the job_log.txt. Although job_log.txt is not a very descriptive name, we can update it to include the log number and change the file extension to .log. In the Navigator window, expand the job [TEXTLINE] writer. Then double-click on Fanout.

In the Edit Job [TEXTLINE] dialog, enable Fanout Expression, then set the Fanout Expression to job_$(FME_JOB_ID).log. 

Now, the file will be named according to the number of the current job, such as job_1234.log.

5. Publish to FME Flow

Save the workspace, then publish it to FME Flow. When registering for services, ensure that Data Download is selected. Click the Edit button and then for Include Writers in Download, select both TEXTLINE writers.

6. Run the Workspace and View the Download

Log into your FME Flow. Run the newly published workspace.

After it has finished running, click on the data download link to download the file.

Once downloaded, browse to the file and inspect the output to confirm the log file.

Example 2: Including the Job Log Using the Flow REST API V4

This functionality requires V4 of the FME Flow REST API. V3 and earlier do not contain the endpoints needed to access job logs.

1. Open Template Workspace

In FME Workbench, open the Download_Logfile-Begin.fmwt workspace. This workspace writes a test message into a text file. 

If you’ve already completed Example 1, you can replace the FMEFlowLogFileRetriever transformer with the HTTPCaller. 

2. Add and Configure the HTTPCaller

Add an HTTPCaller transformer and attach it to the Creator transformer.

Open the HTTPCaller settings. 

For the Request URL, navigate to your FME Flow REST API V4. Scroll down and expand Jobs and then expand the GET /jobs/{id}/log request used to retrieve a job log. 

Select Try it out, fill out the parameters, and click Execute to test running the API call here. 

The response will include the Request URL. Copy the Request URL and paste it into the HTTPCaller Request URL parameter. Then replace the job ID with $(FME_JOB_ID). Set the HTTP Method to GET. 

Alternatively, you can enter the Request URL by inputting your FME Flow hostname in the following URL:

https://<YourFMEFlowHostname>/fmeapiv4/jobs/$(FME_JOB_ID)/log

Select Use Authentication and either select your FME Flow web connection or input your FME Flow credentials. 

Investigate the Request Preview to make sure the call looks correct. You can select Send Test Request to test executing the request and examine the response. Once you’ve tested the request, leave the remaining parameters as default and click OK to close the HTTP Caller settings. 

3. Run to the HTTPCaller and Inspect the Response

Run the workspace to the HTTPCaller and inspect the data coming out of the Output port of the HTTPCaller. In the Visual Preview window, click the ellipsis for the _response_body attribute to inspect.

The entire response body is the job log file.

4. Add a Text File Writer

Add a Text File writer to the canvas. For the Dataset, browse to where you would like it to be saved, then name it job_log.txt, we will modify the filename in the next step.

Once the Text File writer is added to the canvas, connect it to the HTTPCaller output port.

Open the new Text File writer to examine the parameters. Navigate to the User Attributes tab and set the text_line_data value to the _response_body attribute containing the job log.

5. Create Fanout

When the workspace runs on FME Flow, the data download file will contain the job_log.txt. Job_log.txt is not a very descriptive name; however, we can update it to include the log number and change the file extension to .log. In the Navigator window, expand the job [TEXTLINE] writer. Then double-click on Fanout.

In the Edit Job [TEXTLINE] dialog, enable Fanout Expression, then set the Fanout Expression to job_$(FME_JOB_ID).log. 

Now, the file will be named according to the number of the current job, such as job_1234.log.

5. Publish to FME Flow

Save the workspace, then publish it to FME Flow. When registering for services, ensure that Data Download is selected. Click the Edit button and then for Include Writers in Download, select both TEXTLINE writers.

6. Run the Workspace and View the Download

Log into your FME Flow. Run the newly published workspace.

After it has finished running, click on the data download link to download the file.

Once downloaded, browse to the file and inspect the output to confirm the log file.

Example 3: Including Additional Files in the Zip File with the File Copy Writer

Sometimes one may want to include additional files in the Data Download zip file that do not undergo any transformations within the workspace. One method is to read in and write out the same file type with a Reader and Writer and configure the Writer output to be included in the Data Download results as we did with the log file in the previous examples. Alternatively, one can copy files into the Data Download zip folder using a File Copy Writer, which the following instructions go through. 

1. Open Template Workspace

In FME Workbench, open the Download_Logfile-Begin.fmwt workspace. This workspace writes a test message into a text file. If you completed Examples 1 or 2, you can remove the transformers and writers you added to the template workspace. 

2. Add and Configure an Attribute Creator

Add a second AttributeCreator to the canvas and attach it to the Creator output port. Open the AttributeCreator settings and create the following attributes:

• filecopy_source_dataset (this is the full source dataset path of the file to be copied)
• filecopy_dest_filename (optional: to change the file name in the Data Download results, specify the new file name)

In the Value column, fill out the file path to the file to be copied and, optionally, the new file name. 

Click OK to close the transformer.

3. Add a File Copy Writer

Add a File Copy Writer to the workspace. In the Dataset parameter, select a file location to copy the file to, however this will not be used when the workspace is run with the Data Download service on FME Flow. Next, name the folder to include the copied file. Attach it to the output port of the AttributeCreator created in Step 2. 

If you specified a new file name in Step 2, open the File Copy Writer settings, and under Format Attributes, select the file_copy_dest_filename attribute to be exposed. 

Click OK to save and close the writer settings.

When the attributes are expanded on the canvas, they will be green if they match and line up between the AttributeCreator and the File Copy Writer. 

4. Publish the Workspace to FME Flow

Run the workspace on FME Form to test and then publish the workspace to FME Flow using the Publishing Wizard. When registering for services, ensure that Data Download is selected. Click the Edit button and then for Include Writers in Download, select both the TEXTLINE and FILECOPY writers. 

5. Run the Workspace and View the Download

Log into your FME Flow. Run the newly published workspace.

After it has finished running, click the data download link to download the file.

Once downloaded, browse to the file and inspect the output to confirm that it has been successfully copied into the Data Download result.