Skip to content

Handling Invalid Timestamps

Encountering odd or invalid timestamps in the Parse.ly data pipeline, often due to the reader’s web browser or device clock settings, is an occasional issue. These timestamps appear in the ts_session_current and/or ts_session_last fields. Although we can’t control these user-generated values, it’s possible to correct them before loading into your database using ETL checks and transformations. Here’s a guide on handling invalid timestamps effectively:

  1. Comparing Timestamps: During your ETL process, compare the ts_session_current (browser timestamp) with timestamp_info_nginx_ms (server timestamp).
  2. Transformation for Significant Gaps: If there’s a notable gap between these timestamps, transform the ts_session_current to create a new field, like ts_session_current_trans. Set this new field equal to the minimum timestamp_info_nginx_ms for events sharing the same session_id, visitor_site_id, and apikey.
  3. Handling Acceptable Timestamps: For events within an acceptable time range, set ts_session_current_trans = ts_session_current.
  4. Using the Transformed Timestamp: Future transformations requiring a browser timestamp should use the ts_session_current_trans field.

If you need additional help with handling invalid timestamps in Parse.ly’s Data Pipeline, contact Support.

Lines of JSON from an excerpt of raw data. Most of the lines are blurred out. The clear lines relate to various timestamps. Reviewing these are one step towards finding and handling invalid timestamps.
Reviewing some of the timestamp fields in an entry of raw data

Additional Resources for handling timestamps

Last updated: August 15, 2024