Handling Invalid Timestamps

Encountering odd or invalid timestamps in the Parse.ly data pipeline, often due to the reader’s web browser or device clock settings, is an occasional issue. These timestamps appear in the ts_session_current and/or ts_session_last fields. Although we can’t control these user-generated values, it’s possible to correct them before loading into your database using ETL checks and transformations. Here’s a guide on handling invalid timestamps effectively:

1. Comparing Timestamps: During your ETL process, compare the ts_session_current (browser timestamp) with timestamp_info_nginx_ms (server timestamp).
2. Transformation for Significant Gaps: If there’s a notable gap between these timestamps, transform the ts_session_current to create a new field, like ts_session_current_trans. Set this new field equal to the minimum timestamp_info_nginx_ms for events sharing the same session_id, visitor_site_id, and apikey.
3. Handling Acceptable Timestamps: For events within an acceptable time range, set ts_session_current_trans = ts_session_current.
4. Using the Transformed Timestamp: Future transformations requiring a browser timestamp should use the ts_session_current_trans field.

If you need additional help with handling invalid timestamps in Parse.ly’s Data Pipeline, contact Support.

Additional Resources for handling timestamps

• Timestamp fields in our raw data schema
• Metadata guidelines for providing the correct publication date format
• Admin Guide: Modifying the time zone of a Dashboard
• How we measure Engaged Time
• How to set date/time parameters with Parse.ly API