JSON format

Raw data accessed via S3 (bulk) or Kinesis (streaming) consists of lines of JSON objects, also known as JSONLines. This format is easy to parse in programming languages, cloud SQL engines, and big data tools.

This page describes the schema of these JSON records (keys and values) for interpreting raw events.

Example JSON page view record

The following example shows a page view record from a Parse.ly site with alphabetically sorted keys.

{
  "action" : "pageview",
  "apikey" : "example.com",
  "campaign_id" : "facebook",
  "channel" : "website",
  "display" : true,
  "display_avail_height" : 735,
  "display_avail_width" : 1280,
  "display_pixel_depth" : 24,
  "display_total_height" : 800,
  "display_total_width" : 1280,
  "engaged_time_inc" : null,
  "event_id" : "0xe6508eda93d5598367b18555ae9b828d",
  "extra_data" : {"subscriber_type" : "premium"},
  "flags_is_amp" : false,
  "ip_city" : "Newark",
  "ip_continent" : "NA",
  "ip_country" : "US",
  "ip_lat" : 37.5147,
  "ip_lon" : -122.0423,
  "ip_postal" : "94560",
  "ip_subdivision" : "CA",
  "ip_timezone" : "America/Los_Angeles",
  "ip_market_name" : "New York",
  "ip_market_nielsen" : "501",
  "ip_market_doubleclick" : "3",
  "metadata" : true,
  "metadata_authors" : [
    "Laura Vitto"
  ],
  "metadata_canonical_url" : "http://example.com/2020/08/07/airpods-ftw/",
  "metadata_custom_metadata" : "{"page" : 1,"omnitureData" : {"channel" : "watercooler","content_type" : "article","v_buy" : null,"v_buy_i" : null,"h_pub" : 0.0,"h_buy" : null,"h_pub_buy" : null,"v_cur" : 0.0,"v_max" : 0.0,"v_cur_i" : 0,"v_max_i" : 0,"events" : "event51,event61","top_channel" : "watercooler","content_source_type" : "Internal - Editorial Series","content_source_name" : "Apple iPhone 7 Event","author_name" : "Laura Vitto","age" : "0","pub_day" : 7,"pub_month" : 9,"pub_year" : 2020,"pub_date" : "08/07/2020","sourced_from" : "Internal","isPostView" : true,"post_lead_type" : "No Lead Image","topics" : "Apple,Gadgets,iPhone 7,Watercooler","campaign" : null,"display_mode" : null,"viral_video_type" : null,"standalone_video_show" : null,"b_flag" : false}}",
  "metadata_duration" : null,
  "metadata_full_content_word_count" : 174,
  "metadata_image_url" : "http://a.amz.mshcdn.com/media/ZgkyMDE2LzA5LzA3LzU2L0NyeFhpNjNYRUFBSnZwRS5lNDAyMy5qcGcKcAl0aHVtYgkxMjAweDYzMAplCWpwZw/156d0173/3ae/CrxXi63XEAAJvpE.jpg",
  "metadata_page_type" : "post",
  "metadata_post_id" : "http://example.com/2020/08/07/airpods-ftw/",
  "metadata_pub_date_tmsp" : 1473275118000,
  "metadata_save_date_tmsp" : 1473275204000,
  "metadata_section" : "watercooler",
  "metadata_share_urls" : null,
  "metadata_tags" : [
    "parsely_smart:entity:Breathability",
	"parsely_smart:entity:Lyocell",
	"parsely_smart:entity:Perspiration",
	"parsely_smart:entity:Textile",
	"parsely_smart:iab:Needlework",
	"sleep week 2022",
	"underscored explore",
	"underscored lifestyle"
  ],
  "metadata_data_source" : "crawl",
  "metadata_thumb_url" : "https://images.parsely.com/xY9xNBMulGDKRMzfKaUQzs7A9PA=/160x160/smart/http%3A//a.amz.mshcdn.com/media/ZgkyMDE2LzA5LzA3LzU2L0NyeFhpNjNYRUFBSnZwRS5lNDAyMy5qcGcKcAl0aHVtYgkxMjAweDYzMAplCWpwZw/156d0173/3ae/CrxXi63XEAAJvpE.jpg",
  "metadata_title" : "Everyone has the same fear about Apple's new earbuds",
  "metadata_urls" : [
    "http://example.com/2020/08/07/airpods-ftw/"
  ],
  "pageload_id" : "b510edbe-84eb-47b6-aa35-9843b5d3b579",
  "pageview_id" : "ae2badca-d81f-467d-b5fc-5d8y45f08ff6",
  "ref_category" : "internal",
  "ref_clean" : "http://example.com/",
  "ref_domain" : "example.com",
  "ref_fragment" : "",
  "ref_netloc" : "example.com",
  "ref_params" : "",
  "ref_path" : "/",
  "ref_query" : "",
  "ref_scheme" : "http",
  "referrer" : "http://example.com/",
  "session" : true,
  "session_id" : 6,
  "session_initial_referrer" : "http://example.com/",
  "session_initial_url" : "http://example.com/",
  "session_last_session_timestamp" : 1473271351611,
  "session_timestamp" : 1473277747806,
  "schema_version" : "2.3.0",
  "slot" : false,
  "sref_category" : "internal",
  "sref_clean" : "http://example.com/",
  "sref_domain" : "example.com",
  "sref_fragment" : "",
  "sref_netloc" : "example.com",
  "sref_params" : "",
  "sref_path" : "/",
  "sref_query" : "",
  "sref_scheme" : "http",
  "surl_clean" : "http://example.com/",
  "surl_domain" : "example.com",
  "surl_fragment" : "",
  "surl_netloc" : "example.com",
  "surl_params" : "",
  "surl_path" : "/",
  "surl_query" : "",
  "surl_scheme" : "http",
  "surl_utm_campaign" : "facebook_campaign",
  "surl_utm_term" : "8908",
  "surl_utm_medium" : "partners",
  "surl_utm_source" : "facebook",
  "surl_utm_content" : "sports/baseball",
  "timestamp_info" : true,
  "timestamp_info_nginx_ms" : 1473277850000,
  "timestamp_info_override_ms" : null,
  "timestamp_info_pixel_ms" : 1473277850017,
  "ts_action" : "2020-08-07 19:50:50",
  "ts_session_current" : "2020-08-07 19:49:07",
  "ts_session_last" : "2020-08-07 18:02:31",
  "ua_browser" : "Safari",
  "ua_browserversion" : "9.1.2",
  "ua_device" : "Other",
  "ua_devicebrand" : null,
  "ua_devicemodel" : null,
  "ua_devicetouchcapable" : false,
  "ua_devicetype" : "desktop",
  "ua_os" : "Mac OS X",
  "ua_osversion" : "10.10.5",
  "url" : "http://example.com/2020/08/07/airpods-ftw/#L.eZPflSGqq5",
  "url_clean" : "http://example.com/2020/08/07/airpods-ftw/",
  "url_domain" : "example.com",
  "url_fragment" : "L.eZPflSGqq5",
  "url_netloc" : "example.com",
  "url_params" : "",
  "url_path" : "/2020/08/07/airpods-ftw/",
  "url_query" : "",
  "url_scheme" : "http",
  "user_agent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/601.7.7 (KHTML, like Gecko) Version/9.1.2 Safari/601.7.7",
  "utm_campaign" : "facebook_campaign",
  "utm_term" : "8908",
  "utm_medium" : "partners",
  "utm_source" : "facebook",
  "utm_content" : "sports/baseball",
  "version" : 1,
  "videostart_id" : "be0badca-d81e-467d-a5fc-5d7a45f08ff5",
  "visitor" : true,
  "visitor_ip" : "108.225.131.20",
  "visitor_network_id" : "None",
  "visitor_site_id" : "zp94fd56-a400-8210-4b23-zb4348207c43"
}

These key-value pairs are typically strings, but occasionally also numbers, null, or booleans (true / false).

Base event fields

These required fields come from integration with Parse.ly’s data collection infrastructure, whether that’s:

• basic integration for standard web pages
• dynamic tracking of custom events/data
• mobile SDKs for iOS or Android

These fields appear in every single event, regardless of event type or source. Note that excluding the session_id and the visitor_ip fields is possible, though all Parse.ly integrations attempt to support these fields to the best of their ability.

Timestamp fields

Parse.ly records two raw timestamps per event. One comes from Parse.ly’s data collection servers, and the other comes from Parse.ly’s client-side trackers. These are stored as numbers that represent seconds since the UNIX epoch, also known as UNIX time. Parse.ly’s server clocks are in UTC. Pulling data spanning multiple days may be necessary for timezone conversions.

Parse.ly’s server-side timestamp is generally more reliable than the client-side timestamp. However, the client-side timestamp provides greater precision in certain scenarios.

Parse.ly’s nginx (server-side) timestamp is at second resolution, whereas Parse.ly’s pixel (client-side) timestamp is at millisecond resolution. When a pixel timestamp is within a few seconds of the corresponding nginx timestamp, it is likely more accurate since it represents when the event was sent (at millisecond resolution) rather than when the event was received (at second resolution). With Parse.ly’s standard JavaScript tracker, both nginx and pixel are always captured together, so combining them makes JavaScript tracker-based events as accurate as possible.

In mobile SDKs for iOS and Android, it is common to “batch” events if devices are offline. These are also known as “late-arriving” events. In these cases, neither the auto-generated server-side timestamp (in nginx) nor the auto-generated client-side timestamp (in pixel) can be trusted; instead, the client-side override timestamp may be a more accurate representation of reality. The mobile SDK populates these by filling a ts field in the data key-value object sent with every event.

Event ID

A unique, hex-encoded ID string is also generated for each Event. This property can be used to deduplicate events for easier ingestion and processing.

This unique ID is generated by hashing the values of apikey, action, url, timestamp (internal, generated property), visitor_site_id, and timestamp_info_pixel_ms. To ensure that each event_id is truly unique, all events sent to Parse.ly must provide these required fields (excluding timestamp, which Parse.ly generates) at an appropriate level of cardinality and granularity.

For example, if visitor_site_id is not provided for a series of events, then the only properties able to generate unique values for those events are the event type and the timestamp.

Visitors

The visitor_site_id is set by a first-party cookie and visitor_network_id is set by a third-party cookie.

Session enrichments

Parse.ly’s JavaScript tracker automatically creates useful session information for user session analysis. For one thing, Parse.ly’s session_id also doubles as a “number of visits” value, since it’s an auto-incrementing integer that starts at 1 and moves up by one for every new visit by a visitor with the same visitor_site_id.

Note that these enrichments are performed client-side by Parse.ly’s JavaScript tracker; they will not apply to events received via other integrations.

The other fields stored with the session are described below:

Timestamp enrichments

Based on the timestamp fields, Parse.ly creates an important field called ts_action. This field reinterprets timestamp_info_nginx_ms (Parse.ly’s server time) as a formatted date string that is highly compatible with many systems. For example, it is the same format expected by Amazon Redshift and Google BigQuery’s JSON value parsers.

• ts_action: "2016-06-18 02:03:24"

This value is derived from epoch time 1466215404000; it also lacks timezone information but can be interpreted as a UTC time. Including timezone information, as one might for the “full” ISO8601 standard, makes this string incompatible with some SQL engines, so Parse.ly uses a maximally compatible format instead.

Geo IP enrichments

Based on the visitor_ip field, Parse.ly enriches the following:

URL and referrer enrichments

Based on the url, referrer, session_initial_url and session_initial_referrer fields, Parse.ly provides several enrichments. For illustration, the following examples use values:

Metadata

Whether crawled via JSON-LD or meta tags or passed directly in pixels (as is the case in Parse.ly’s video integration), metadata associated with the url field is passed along in a series of metadata_ fields:

UA and device enrichments

Based on the ua field, Parse.ly enriches the following:

Parse.ly also provides information regarding the display of the device:

UTM parameter enrichments

Based on the url field, Parse.ly enriches the following from its query parameters. Note that UTM parameters are a web-wide de facto standard for campaign tracking, first introduced by Urchin and Google Analytics. Google runs a free tool called the URL builder to build URLs with this format, but many tools will automatically add these parameters to allow for easier tracking, especially in places where HTTP referrers are not automatically set.

In this example, the article URL, http://example.com/1234 was clicked from an email newsletter. It might then have had query parameters like the following (scroll to read):

http://mashable.com/1234?utm_source=newsletter_2016-06-01&utm_medium=email&utm_term=footer&utm_content=template_a&utm_campaign=subscriber_newsletter

Which would be parsed as follows:

UTM parameter tracking is powerful because it allows grouping, rollup, and slice-and-dice of campaigns, which often have associated costs and can be part of an ROI calculation. It also helps tremendously with decoding “direct” traffic; e.g., in many email service providers, the above click from an email newsletter would have no HTTP referrer set, and thus UTM parameters would be the only way to understand this traffic.

Extra data

Arbitrary key-value pairs can be passed via Parse.ly’s dynamic tracking or Parse.ly’s implementation for custom segments. Such custom data may include subscriber information or IDs for use in joining to other data sources. In these situations, key/value pairs will appear as a nested JSON object in the extra_data field.

As part of an ETL process, these fields can be “flattened” into the root document format for inclusion in downstream databases storing Parse.ly raw data.

• "action": "_scroll"
• "extra_data": {"_y": 1430}

In this example, a custom event (_scroll) was sent to Parse.ly’s Data Pipeline with associated custom data {"_y": 1430} representing 1,430 pixels on the y-axis of scroll-depth within the browser. This kind of raw data could be used to implement scroll depth tracking.

Other possibilities

This raw data schema is already quite rich and supports many queries not available in the Parse.ly Dashboard or APIs. Additional possibilities for storing data in raw events include:

• subscriber identifiers, to do detailed loyalty analysis
• more granular information about on-page or in-app activities
• a specialized set of query parameters for social virality modeling
• ad impression or revenue data
• and other custom data

Next steps

Read on for Code Examples.

Or, get help from Parse.ly:

• For existing Parse.ly customers, contact Parse.ly to discuss advanced use cases for raw data.
• For organizations not yet Parse.ly customers, start with the basic integration or schedule a demo to learn about advanced use cases Parse.ly customers have implemented.