API: Content Recommendations

Overview

Related content rather than high click-through rates

At a high level, our system optimizes showing similar content rather than high click-through rates. We chose to take this approach because systems that optimize for click-through rate tend to promote click-bait. You may be familiar with recommendation blocks below articles that display outrageous images or feature headlines like The ten most unexpected mistakes that could kill you today. This sort of content rises to the top because the system that chooses recommendations focuses narrowly on attracting clicks. Parse.ly’s recommendation system takes a different approach by answering the following question: if the reader wants to learn more about the topics covered by the content they are currently reading, what should we recommend?

Note: if you’re interested in displaying content that has received the most traffic recently, you can use our analytics endpoint or our top endpoint.

How do Parse.ly Recommendations work?

When you become a Parse.ly customer, we crawl all of your historical content and continue to crawl all new pieces of content in real time. Each time we crawl a piece of content, we analyze the title and full content to create an embedding and store that in a database. An embedding is a numerical representation of content in a special, high-dimensional space where documents that contain similar meaning are located closely together.

When the Parse.ly API is asked to provide a recommendation for a given document, we simply search through this embedding space for that document’s neighbors. The ordering of the recommended documents simply reflects how close they are to the query document in the embedding space.

How can I filter or tweak the recommendations?

If you notice a problem with the content recommended by the Parse.ly API, you might be able to use filters or exclusion to improve the results.

Filtering

Using filters, you can choose to only recommend content that’s been

• Assigned to a particular section
• Given a particular tag
• Written by a particular author
• Published in a given publication date range

Note: a request to our API can have multiple filters attached to it. See the docs for more information.

Excluding content

If you want to block certain pieces of content from showing up in the recommendations, you can do so by using our /exclude endpoint.  Note that excluded pieces of content will also not show up in any of our API’s endpoints (such as the /analytics or /top endpoints).

What about personalization?

Our approach towards personalized closely resembles our basic approach to content recommendations.  The basic approach, which we have outlined above, is to serve content that’s similar to the currently viewed piece of content.  In the case of personalization, we keep track of the visitor’s history – each piece of content that they read.  When making personalized recommendations, we take into account all of this visitor’s personal history and retrieve content that is similar to all of their previously-read content.  We are careful to not recommend content that the visitor has already read.  For details on how to instrument personalization, check out the docs here.

Recommendations and Personalization Endpoints

GET /profile

Train a user profile for the purpose of personalized recommendations. Use HTTP method:related to return recommendations using the same UUID as specified here.

Query Parameters

Your API secret is not used with this endpoint.

Note: Profile data is kept for 7 days by default. Contact your Relationship Manager for information about extending retention limits.

GET /related

Retrieve a list of Post recommendations for a specified profile UUID (trained in HTTP method:profile) or a specified canonical URL.

If no sufficiently relevant recommendations are found, results are returned sorted by publication date. You can change this behavior by specifying the respect_empty_results parameter. Note that these results will still be limited by any filters specified, so it’s possible to have no results returned if there are none that match the filters, regardless of the value of respect_empty_results.

This endpoint takes one notable optional parameter:

• sort, which determines the posts returned. _score orders by relevancy, and pub_date orders by publish date.

This allows you to find relevant recommendations but organize results to promote, for example, popular social stories.

Query Parameters

Your API secret is not used with this endpoint.

Optional Parameters

The maximum pagination limit is 2000 records. You will receive an error from the API if you attempt to retrieve past record 2000 through any combination of page and limit.

GET /history

Retrieve a list of URLs visited by a user by UUID. This is the training profile being used for personalized recommendations.

Query Parameters

Your API secret is not used with this endpoint.

GET /similar

The /similar endpoint is deprecated and the functionality has now been merged into the /related endpoint. Customers should use the /related endpoint moving forward.

Examples

Contextual

Contextual Recommendations use a URL as input and return a list of relevant Posts as output.

The endpoint for this is /related and the parameter required is url. Let’s say the URL passed is about the Apple iPod. The returned “related” Posts might look like this:

{
  "data": [
    {
      "title": "Ipods And Crime"
      "url": "http://apikey.com/219045/",
      "author": "Kevin Black",
      "section": "Technology",
      "pub_date": "2012-02-20T19:08:00Z",
      "tags": ["apple", "ipod", "crime"]
    },
    // other similar JSON documents...
  ]
}

Personalized

Personalized Recommendations return Posts relevant to the user’s interests, and does not include Posts that the user has already read in the past.

Integrating Personalized Recommendations is slightly more complicated than the above APIs because it requires two steps on behalf of the implementer.

1. Training. This logs that an individual with a specific UUID has visited a specific URL on your site.
2. Recommending. Using a saved UUID for the user, return relevant stories from the Parse.ly API.

To train a user profile for your site, use the /profile endpoint. Pass the uuid and url parameters to log that an individual with that UUID has visited the specified URL.

Generating UUIDs is sometimes a point of confusion for our customers. Parse.ly already generates a UUID for every user using a cookie-based approach. This is accessible on pages with the Parse.ly tracker installed in the field PARSELY.config.parsely_site_uuid. You are free to re-use this UUID for your application if the cookie-based approach is sufficient for you.

However, some customers may have better UUIDs available. If you have e.g. an internal integer ID which also identifies registered/premium users on your site as well as anonymous visitors, this may be a better choice. Parse.ly does not care what the UUID is — it is simply a string used for identification on further calls to the API.

Once a user is trained, you can view that user’s history with /history endpoint. Pass the uuid and you will see the URLs currently associated with that user. This is mainly useful for debugging purposes.

Finally, call the /related endpoint with the uuid parameter to get personalized recommendations for that user. The default strategy uses a document similarity metric weighted toward most recently seen Posts, and a filter excluding all visited Posts; we plan to introduce additional strategies over time.

Personalized recommendations will appear as multiple posts, similarly to Contextual recommendations. For example, if you train a user on a story about 4G data access plans for mobile phones, you may end up with recommendations like this:

{
 "data": [
   {
     "title": "Verizon announces new 4G phone"
     "url": "http://apikey.com/244232/",
     "author": "John Peters",
     "section": "Technology",
     "pub_date": "2012-03-30T08:26:00Z",
     "tags": ["verizon", "technology", "phones", "4g"]
   },
   // other similar JSON documents...
 ]
}