Handling Local Timezones, UTC, Daylight Savings Time, and Leap Units

DateTime Variations

Dates and times (together known as DateTimes) do vary over different regions of the world and in response to adjustments to match solar time. The primary time standard is called Coordinated Universal Time (also known as UTC). However, local time standards can be affected by:

• Local Time Zones
• Daylight Savings Time
• Leap Year and Leap Second Adjustments

This page includes a series of examples to show how FME can be used to manipulate and handle datetime variations caused by timezones and other adjustments.

The examples follow below and all include workspaces and the data to run them. They cover these topics:

• How to create a zoned timestamp
• How to add a time zone to an existing datetime
• How to resolve a datetime with time zone to UTC time
• How to apply time zones to datetime arithmetic
• How to create a timestamp including daylight savings time
• How to apply daylight savings time to datetime arithmetic
• How to manually add or subtract daylight savings time
• How to create a timestamp with leap year or leap seconds included
• How to repair datetimes with leap seconds or invalid leap years
• How to apply leap years to datetime arithmetic

The examples are all stand-alone; in other words, you don't have to carry out all of them, you can just pick and choose which examples you want to try.

Local Timezones

In theory, time is defined at each part of the world by the offset of its local noon from the local noon at zero degrees longitude. In practice, each part of the world falls inside a designated timezone.

Each timezone has an offset that is a number of hours (or minutes) plus or minus from UTC.

For example, Vancouver in Canada is in the Pacific Time Zone (PST or Pacific Standard Time). The PST zone is eight hours behind UTC time and this offset is generally formatted in dates as "-08:00". So at noon UTC it is "040000-08:00" in pacific time.

Similarly, Japan - which uses Japanese Standard Time (JST) - is nine hours ahead of UTC, which is represented as "+09:00". So at noon UTC it is "210000+09:00" in Japan.

FME is capable of interpreting time offsets in datetimes, and taking them into account in arithmetic calculations. It can also add or remove offsets from a datetime, resolve an offset to UTC, or resolve UTC to a local offset.

Getting a Zoned Timestamp

To create a new time stamp the DateTimeStamper transformer is the one to use. There is a parameter to choose whether you require local time or UTC, and another to denote that the time zone offset is required.

Here a user records the current date in UTC. A second DateTimeStamper records the current datetime in the local timezone:

The DateTimeConverter is used to convert the results to a more human-readable format. The output looks like this:

It's easy to see that the difference between local and UTC time matches the offset difference (7 hours).

Variations

Caution is required when FME is run on a virtual machine, such as with Amazon AWS. For example, I am based in Winnipeg (Central time), processing data from Vancouver (Pacific time) using an Amazon virtual machine based in Virginia (Eastern time). The timestamp from this setup will be Eastern time.

So your computer's local timezone may not be the same as your local timezone, which may itself be different from the timezone of the data!

Additionally, if you want both UTC and local time, then - unlike the example above - it's best to obtain one datetime and then convert it to local time (i.e. resolve the offset). Otherwise, there might be slight differences in the seconds fields, when fractions of a second are involved, and the two times won't match.

The FME DateTime function equivalent to the DateTimeStamper transformer is @DateTimeNow()

Example

Follow these steps to try the attached example.

Open the workspace, ensure Feature Caching is active, and press the Run button. Inspect the cached results.

Notice that the two times differ by the same amount as the time zone offset. Of course, the amount of difference will depend on the timezone in which you run the workspace!

Experiment with the parameters of the DateTimeStamper transformers to see what they do. Include seconds in the DateTimeConverter output format, to see what tiny difference (if any) exists between UTC and Local time as a result of being created by different transformers.

Adding Timezone to a Date

In this example, the user has a datetime attribute. Its current format is:

%Y-%m-%d %H:%M:%S

For example:

2019-07-15 15:14:30.59

There is no time zone offset information on this attribute. However, the user knows that all records (city of Vancouver) relate to their local time zone (Pacific).

The user wishes to add a time zone to their records. Because there is no specific transformer to carry out this operation, the user must make use of an FME DateTime function.

The user places an AttributeManager transformer to create a new attribute called ZonedTime:

In the Text Editor dialog the user has entered the following content:

@TimeZoneSet(@Value(IssuedDate),local,auto)

The TimeZoneSet function sets the timezone on the IssuedDate attribute. The "local" argument tells FME that this data is in the user's (or their computer's) local timezone.

The output looks like this:

Where time zone has been added to the date as a UTC offset. Also notice that the datetime has been converted to FME format. A DateTimeConverter can be used to convert back to the prior format.

Variations

The "local" argument in the TimeZoneSet argument can be replaced by an offset such as "-08:00". However, a local timezone is preferred. That's because daylight savings can be taken into account (see below). The same reason means adding the offset with a StringConcatenator is similarly not recommended.

However, sometimes a set of spatial data represents features spread across multiple time zones. For example, I have a dataset representing the capital cities of the world. I cannot assign a fixed time zone offset to each because each may fall in a different time zone.

In that scenario, there are two transformers that may help. The Geocoder transformer (a package from the FME Hub) and the TimeZoneExtractor transformer (a transformer from the FME Hub) both use a Google web service to convert the position of the data into a known time zone offset. That offset could then be applied using the above function, with the time zone offset in place of "local".

Example

Follow these steps to try the attached example.

Open the workspace template, ensure Feature Caching is active, and press the Run button.

Inspect the cached results to show that the IssuedDate attribute does not have a time zone offset, but that the ZonedTime attribute has a time zone offset.

As before, the actual timezone depends on the timezone in which you run the workspace! Because the data comes from the city of Vancouver, you should perhaps experiment with using a fixed offset (such as -08:00).

Resolving Timezone Offsets

In this example, the user has a datetime attribute. Its current format includes a local time plus offset, such as:

20190715151430.59-08:00

They wish to resolve the offset to get time in UTC, such as:

20190715231430.59+00:00

Because there is no specific transformer to carry out this operation, the user must make use of an FME DateTime function.

The user places an AttributeManager transformer to create a new attribute called UTCTime and opens the Text Editor dialog:

In the Text Editor dialog the user has entered the following content:

@TimeZoneSet(@Value(ZonedTime),utc,convert)

The TimeZoneSet function sets the timezone on the IssuedDate attribute. The "utc" and "convert" arguments tell FME that this data should be converted to UTC.

The output looks like this:

Notice that the offset has been resolved by subtracting it from the zoned time.

Note that daylight savings differences are taken into account when converting zones in this way. FME has a large database of daylight savings and time zone information that is updated on a yearly basis and will consult this when making the conversion. It will know (for example) that 19900331153143 UTC (15:31 on March 31st, 1990) was before daylight savings started for that year, and that the Canada-Central equivalent is 19900331093143-06:00 (09:31-06:00); whereas 19900401153143 UTC (15:31 on April 1st, 1990) was after daylight savings so the Canada-Central equivalent is 19900401103143-05:00 (10:31-05:00).
 

Variations

The "source" datetime must include a time zone offset. If it does not, then the feature will be rejected.

An alternative method using transformers would involve extracting the offset (DateTimeConverter) and then subtracting it from the original date (DateTimeCalculator). The TimeZoneSet function is preferable because it carries out the same task in a single step, using functionality designed for this specific purpose.

Example

Follow these steps to try the attached example.

Open the workspace template, ensure Feature Caching is active, and press the Run button.

Inspect the cached results to show that the ZonedTime attribute has a local offset, while the UTC time attribute represents the same time but in UTC format.

Arithmetic Calculations and Timezones

Here a user has two dates. They are being created in an AttributeCreator transformer, but could just have easily been read from a source dataset:

One datetime (16:43pm) is in UTC time. Another (08:42am) is in PST (Pacific) time. A DateTimeCalculator transformer is set up to calculate the interval between the two datetimes:

The resulting interval is just 60 seconds. Although the difference between times (16:43 and 08:42) is eight hours and one minute, the DateTimeCalculator takes the eight hour time difference (UTC offset) into account. The result is that in reality, just 60-seconds separates the two datetime values.

Variations

In the above example, if both datetimes were unzoned (i.e. lacking an offset) then the time difference would be 28,860 seconds. That's because, if no time zones are defined, then no time zone offsets are applied.

If only one datetime was unzoned (for example, one is PST, and the other is unknown), then the feature would be rejected. FME will not assume that both attributes fall in the same time zone. The interval attribute on the rejected feature will be set to <null>.

The order of the attributes in the DateTimeCalculator is important. In this example, if the UTC date is defined as the Start Datetime, then the difference is -60 seconds. If the PST date is defined as the Start Datetime, then the difference is (+)60 seconds.

The FME DateTime function equivalent to the DateTimeCalculator transformer's Interval mode is @DateTimeDiff()

Example

Follow these steps to try the attached example.

Open the workspace, ensure Feature Caching is active, and press the Run button.

Inspect the cached results to show that the interval is -60.

Experiment with different times and timezone offsets for the two dates.

Daylight Savings Time

In some countries, time moves forward by an hour at the start of the summer months, then back again at the start of winter months. This is known as Daylight Savings Time (DST).

For example, Vancouver time is usually Pacific Standard Time (PST). In summer months it is Pacific Daylight Time (PDT).

Not all countries and locations switch to daylight savings time simultaneously. That's why - for example - if you register for a Safe Software webinar in March or November, it's a wise idea to check the exact time of the event, including the DST status.

Getting a Timestamp with DST Applied

To create a new timestamp the DateTimeStamper transformer is the one to use. There is a parameter to choose whether you require local time or UTC, and another to denote that the time zone offset is required.

Because UTC time never switches to daylight savings time, the offset between local time and UTC can vary throughout the year. This offset is already accounted for in timestamp results.

Here (using the same example as a previous section), a user records the current date in UTC. A second DateTimeStamper records the current datetime in the local timezone:

The results look like this:

Although the difference between PST and UTC is eight (8) hours, here the offset is recorded as seven (7). That's because the offset includes the calculation for daylight savings time (PDT).

If the workspace were re-run in a month without daylight savings, say December, then the UTC offset would be recorded as -08:00.

This illustrates that FME takes care of DST automatically when creating timestamps.

Variations

As with the previous example, caution is required when FME is run on a virtual machine, such as with Amazon AWS. For example, I am based in Winnipeg (Central time), processing data from Vancouver (Pacific time) using an Amazon virtual machine based in Virginia (Eastern time). The timestamp from this setup will be Eastern time.

So your computer's local timezone may not be the same as your local timezone, which may itself be different from the timezone of the data! This means that the daylight savings rules may operate differently in the computer timezone than the local timezone.

The FME DateTime function equivalent to the DateTimeStamper transformer is @DateTimeNow()

Example

Follow these steps to try the attached example.

Open the workspace, ensure Feature Caching is active, and press the Run button.

Inspect the cached results. Notice that the time offset takes into account the daylight savings difference. A local timestamp will be created with the correct daylight savings offset from UTC.

Once more, the actual timezone offset depends on the timezone in which you run the workspace, as well as the time of year!

Datetime Arithmetic with DST Applied

Datetime arithmetic can be carried out with FME, but what happens when one date is within daylight savings time, but the other date isn't?

Here a user has two dates. They are being created in an AttributeCreator transformer, but could just have easily been read from a source dataset:

Both datetimes exist in the Pacific time zone, although neither has the UTC offset applied. Both dates are 10th March 2019. One time is 00:30am and the other is 03:30am.

It's easy to think that the difference between the two times is three hours (180 minutes), and the DateTimeCalculator at first confirms that:

However, daylight savings time began at 02:00am that day, at which point clocks would be turned forward one hour. That means the actual time difference is only two hours (120 minutes).

So, when a UTC offset is applied to these datetimes (using the TimeZoneSet function as described above) then daylight savings time is taken into account to give the correct result:

The output even shows that the UTC offset is different for each datetime.

In short, as long as a UTC offset is available, then daylight savings time is taken into account during any arithmetic operations.

Variations

As previously mentioned, if only one datetime had a UTC offset then the feature would be rejected. FME will not assume that both attributes fall in the same time zone or have the same daylight savings value. The interval attribute on the rejected feature will be set to <null>.

The FME DateTime function equivalent to the DateTimeCalculator transformer's Interval mode is @DateTimeDiff()

Example

Follow these steps to try the attached example.

Open the workspace, ensure Feature Caching is active, and press the Run button.

Inspect the cached results for each DateTimeCalculator to show the difference when UTC offset causes daylight savings to be taken into account.

Manual Addition or Subtraction of DST Hours

To a large extent this isn't needed if you have the UTC offset, because everything is applied automatically, and even without the offset, times created with the DateTimeStamper include the difference. Likewise, if you add the UTC offset correctly, then you shouldn't need to do this.

However, in rare cases that you do need to add or subtract an hour, use the DateTimeCalculator.

Here an FME user is examining records from a log of when an overnight FME process started:

They use an AttributeManager to add the UTC offset to the original datetimes. The datetimes that occurred in the summer months have a different offset because of daylight savings time.

However, the user suspects that the task scheduler used to start the process ignored daylight savings time and ran the process at 11:00pm standard time all year round. This means that the process logs are incorrect for summer months. The datetimes need an hour added to them.

They set up a DateTimeCalculator -using a Conditional parameter - to add one hour to the datetime where the offset ends in "-07:00":

Now the output shows an hour added to those specific datetimes:

Of particular note is that, because the additional hour moved the time to midnight, the date also rolled over; for example, the date used to be 2016-04-01 but is now 2016-04-02.

Variations

Should the situation be reversed, it would be just as easy to subtract an hour as add one. However, as noted, manual additions or subtractions to accommodate daylight savings is likely to be a rare case.

The FME DateTime function equivalent to the DateTimeCalculator transformer's Add/Subtract mode is @DateTimeAdd()

Example

Follow these steps to try the attached example.

Open the workspace, ensure Feature Caching is active, and press the Run button.

Inspect the cached results for each step to show the datetimes at each stage (without UTC offset, with UTC offset, with additional hour).

Pay attention to the conditional parameter in the DateTimeCalculator. This is a way to avoid separating the datetimes using a Tester.

Leap Year and Leap Second Adjustments

A leap year is an adjustment made to keep days in line with astronomical events, such as seasons. They are necessary because the Earth's orbit around the sun is not exactly 365 days, and so astronomical events would drift with respect to the calendar.

In other words, these adjustments fix variability in the length of the year. They ensure that summer and winter do not drift to different parts of the calendar.

A leap second is an adjustment made to keep times in line with the movement of the sun. In particular, they aim to keep noon UTC as close to solar noon as possible. They are necessary because the Earth's orbit has irregularities that would otherwise cause a difference between UTC and solar noon.

In other words, these adjustments fix variability in the length of the day. They ensure that the sun is at its highest elevation at noon.

A leap year occurs on a fixed interval of every four years (with some exceptions). However, because variations in the length of the day are less predictable, the insertion of a leap second cannot be scheduled.

Of course, such adjustments - however small - can have an effect on datetime measurements and arithmetic.

Leap Unit DateTimes

A leap year datetime is represented by the date February 29th; for example:

2016-02-29 12:00:00-08:00

A leap second datetime is represented by the number of sixty seconds past the minute; for example:

2016-12-31 23:59:60-08:00

In other words, where the time would normally progress from 23:59:59 to 24:00:00, at certain points, a leap second may be inserted to give 23:59:60!

The DateTimeStamper transformer is aligned to the system clock, so as long as your computer properly handles leap years and leap seconds, then this FME transformer should return those valid values.

Example

No workspace is included, but simply try using a DateTimeStamper on February 29th, the next time there is a leap year. The upcoming leap years are 2020 and 2024. Be sure not to miss it, or you will have to wait another four years!

You can also try capturing a leap second with the DateTimeStamper. Although these occur at irregular intervals and at different times of day, they are on fixed dates. So although we don't know when the next leap second will be, it will occur on either June 30th or December 31st. The International Earth Rotation and Reference Systems Services (IERS) issues a regular bulletin (Bulletin C) with information about leap seconds. The current bulletin tells us that there will NOT be a leap second on December 31st 2019.

Repairing Datetimes

The DateTimeConverter transformer has a parameter called Repair Overflow. Invalid leap year dates can be corrected by setting this parameter to Yes.

Here a user has two dates. One is a valid leap year date (2016-02-29), and the other is not (2019-02-29):

The user converts the format of these dates, using the Repair Overflow option:

The result is the invalid date is counted as an "overflow" and rolls over to the next valid date:

Variations

A leap second (23:59:60) can also be rolled over to its next valid time if desired, which would be 00:00:00

The FME DateTime functions equivalent to the DateTimeConverter transformer are @DateTimeCast(), @DateTimeFormat(), and @DateTimeParse().

Example

Follow these steps to try the attached example.

Open the workspace, ensure Feature Caching is active, and press the Run button.

Inspect the cached results to prove the repair works.

Try turning off the repair option to see what the result is. Also, experiment with varying the input dates; for example, what happens if you try 20190231 (31st February)?

Leap Unit Arithmetic

FME will automatically account for leap years when carrying out arithmetic calculations on datetimes.

This user has four dates:

Both sets of start and end dates cover the range February 28th to March 1st. However, one range is 2016 (a leap year), and the other is 2019 (not a leap year).

The user calculates the interval between each set of dates - in hours - using the DateTimeCalculator transformer:

For 2019 the difference is returned as 24 hours. For 2016 the leap year has been taken into account, and the difference is returned as 48 hours:

Variations

FME won't calculate time intervals using leap seconds in the same way; i.e. 23:59:59 to 00:00:00 is always one second only, even if a leap second occurred in between. This is for two reasons: partly because there is no standard method to handle leap seconds, but mostly because leap seconds occur simultaneously across the globe, and so are at different times of the day in different locations! For example (thank you Wikipedia), the leap second on December 31, 2005 23:59:60 UTC occurred December 31, 2005 18:59:60 in the US Eastern time zone.

The FME DateTime function equivalent to the DateTimeCalculator transformer's Interval mode is @DateTimeDiff()

Example

Follow these steps to try the attached example.

Open the workspace, ensure Feature Caching is active, and press the Run button.

Inspect the cached results to prove the calculation works.

Experiment with different dates to prove (for example) that February 28th 2016 to February 28th 2017 returns an interval of 366 days.

Data Attribution

Unless otherwise stated, the data used throughout this series originates from open data made available by the City of Vancouver, British Columbia. It contains information licensed under the Open Government License - Vancouver.