Reconciling differences between DPL and Dashboard

---

What is Parse.ly Analytics?

Parse.ly Analytics includes products such as the Parse.ly Dashboard and Parse.ly API that provide aggregated and cleansed data.

What is Parse.ly Data Pipeline?

Parse.ly Data Pipeline is the raw, unaggregated, and uncleansed data output for all Parse.ly events.

How can both systems return different data but both be right?

The Parse.ly Dashboard and Parse.ly Data Pipeline use the same underlying raw data, but the Parse.ly Dashboard combines event data and content metadata in a way that is not simple to replicate in the raw Data Pipeline data.

For example, consider a piece of content that was published on August 1, 2020, and tagged with Section A. On August 2, 2020, the tagging changed to a different section. The Parse.ly Dashboard would capture that change because the change was within 5 days after the published date, but the Data Pipeline serves data “as-is” when the event occurs. One way to handle this is to treat metadata as Type I or Type II Slowly Changing Dimension. This solution is detailed further below.

For example, now consider an article published on June 1, 2020, and tagged with Section A. On June 15, 2020, the tagging changed to a different section, and the article was then recrawled. The Parse.ly Dashboard would not update the past 15 days of data and instead only start reporting the new section as of June 15. The same applies to Data Pipeline. But given that the customer is using the most recent metadata for all post traffic in the DPL query, this would make the two systems report differently.

Both systems are right. The Data Pipeline provides more control to decide which “right” is best for the organization and analysts.

Considerations for defining a specific version of the truth in queries:

• Create a metadata dimension as a Slowly Changing Dimension. For all SCDs, multiple versions of the truth exist, and each organization must decide which is best for its business case. This allows reporting on views based on either:
  
  • the metadata at the time the event happened on the site
  • the metadata that is true now
  • the metadata that was true in-between the event happening and now
    
• Always account for time zone differences, even small ones. The Data Pipeline reports all time zones in UTC, and the Parse.ly Dashboard reports in the customer’s time zone.