Title: How Headline Testing works
Author: Lucas Radke
Published: June 16, 2025
Last modified: November 6, 2025

---

 1. [User handbook](https://docs.parse.ly/user-handbook/)
 2. [The Parse.ly Dashboard](https://docs.parse.ly/user-handbook/dashboard/)
 3. [Optimization menu](https://docs.parse.ly/user-handbook/dashboard/optimization-menu/)
 4. [Headline Testing page](https://docs.parse.ly/user-handbook/dashboard/optimization-menu/headline-testing-tab/)
 5. How Headline Testing works

#  How Headline Testing works

Parse.ly’s [Headline Testing](https://docs.parse.ly/dashboard/headline-testing-tab/)
uses a real-time optimization technique based on a machine learning method called
multi-armed bandit testing. Specifically, it uses the Thompson sampling algorithm,
which dynamically allocates more traffic to better-performing variants over time.
This means that if one headline performs well early on, more users will see it, 
while poorly performing variants are shown less frequently.

You can [create tests](https://docs.parse.ly/dashboard/headline-testing-tab/create-test/)
with up to 10 headline variants per piece of content. One of these is the original(
the control), and the others are alternatives to evaluate. As users visit the page,
the tool automatically displays different variants to different visitors and collects
data on how often each one is seen (impressions) and how often each is clicked. 
Based on this data, the algorithm adjusts which headlines are prioritized, sending
more traffic to those that perform well.

Not every headline test ends with a statistically confident winner, but that doesn’t
mean the test failed. Valuable insights can still emerge, especially when looking
at the Help Chance metric. This helps you choose headlines that are likely better
than your control, even when the system isn’t confident enough to declare a single
clear winner.

This approach stands in contrast to traditional A/B testing, where traffic is split
evenly between all variants for the entire duration of the test. In a bandit-based
system like Parse.ly’s, the testing process begins with equal distribution, but 
quickly shifts traffic toward better-performing options as more data becomes available.

## Details on the Headline Testing process

 1.  First, a separate JavaScript script must be installed on your website. This script
     is intentionally designed to be as lightweight as possible, separate from the 
     main Parse.ly script, to ensure it loads very quickly and before any page content
     is displayed.
 2.  The script comes preloaded with the list of active headline experiments. The system
     attempts to locate the control headline attached to each article link on the page.
 3.  If the control headline is found, the script replaces it with the assigned variant
     headline for that specific visitor. A [local storage](https://docs.parse.ly/cookies/#h-description)
     key (vipexp-local-state) holds which experiment and variant was shown to a user.
 4.  When the variant headline is displayed the first time, this counts as an impression
     for that visitor.
 5.  If the visitor clicks on the headline the first time, it is recorded as a success
     for that headline variant, and the system tracks this click.
 6.  The selected headline variant is stored in the visitor’s cookies. This ensures
     consistency so that the same visitor will continue to see the same headline variant
     on subsequent visits.
 7.  The system uses a **multi-armed bandit approach**, specifically **Thompson sampling**
     with a **beta distribution**, to determine which headline should be shown more
     frequently. For each variant, the algorithm models the probability of success 
     based on observed data:
 8.   * **Alpha (α)** represents the number of **unique clicks** on a headline variant
        under test.
      * **Beta (β)** represents the number of visitors the headline has been displayed
        to (**impressions**) without clicks.
 9.  Over time, the algorithm dynamically adjusts the probability of showing each variant,
     favoring headlines that perform better while exploring other options to continuously
     learn.

This entire process happens automatically and quickly, enabling real-time optimization
without requiring any manual analysis or adjustments.

## A/B testing vs. multi-armed bandit (MAB)

Parse.ly’s Headline Testing uses a multi-armed bandit (MAB) algorithm rather than
traditional A/B testing. Here’s how the two approaches differ, and why MAB is better
suited for headline optimization:

| Aspect | A/B testing | Multi-armed bandit (MAB) | 
| **Traffic allocation** | Fixed (e.g., 50/50) | Dynamic based on variant performance | 
| **Optimization timing** | After the tests conclude | Ongoing during the test | 
| **Handling poor variants** | Continues showing all variants equally | Gradually reduces traffic to underperforming variants |

Multi-armed bandit algorithms do more than just test headline variants; they optimize
traffic distribution and adapt in real time.

As the system collects performance data, it dynamically adjusts how traffic is allocated,
potentially sending more visitors to the better performing headlines as soon as 
there is enough evidence to support the shift. This allows the best headline to 
reach a larger portion of the audience more quickly, while also ensuring that just
enough exposure is given to the weaker variants.

The multi-armed bandit approach is especially useful in environments where content
has a short lifecycle or where fast optimization is required.

## Exporting Results

You now **export the results of finished headline tests** for further analysis or
reporting.

#### **How to Export:**

 1. Go to any **finished headline test**.
 2. Click the **“Export”** button in the top-right corner of the Evolution chart.
 3. Choose your preferred format:
 4.  * **CSV** for spreadsheet-friendly output
     * **JSON** for structured data and automation workflows

These exports includes available test metrics, such as CTR, views, and variant performance.

## **Our Algorithms and Confidence Grades**

Parse.ly uses Bayesian modeling to estimate the future performance of each headline
variant. These estimates are translated into **confidence grades**, which help you
make data-driven decisions more easily.

Each headline variant receives a **win chance** — the probability that it will continue
to perform best in the future — and a **help chance** — the probability that it 
will perform better than the control.

### **Confidence Grades Explained:**

 * **High**: Win chance of 95% or more. Strong result; safe to implement.
 * **Fair**: Win chance between 70–95%. Worth considering with editorial judgment.
 * **Low**: Win chance between 55–70%. Use cautiously; help chance becomes relevant.
 * **Insufficient**: Win chance below 55%. Results are statistically inconclusive.

### **Outcomes of a Test:**

 * **Winner**: Identified with sufficient confidence.
 * **No clear winner**: Results inconclusive, but data may still be useful.
 * **Not enough data**: Traffic volume too low to evaluate properly.

To improve inconclusive tests, consider running them longer, or reducing the number
of variants.

## **Results Dashboard Enhancements**

Parse.ly has also improved the results dashboard with clearer metrics and helpful
visuals:

 * **Winner badge**: Clearly marks the top headline and its confidence grade.
 * **Popover insights**: Detailed breakdowns of win chance and help chance.
 * **Performance metrics**:
    - **Confidence grade**
    - **Total clicks**
    - **Total impressions**
    - **Click-through rate (CTR)**
    - **Improvement over control (CTR difference as percentage points)**
 * **Evolution chart**: Visualizes how each variant performed over time.

### **Understanding “pp” (Percentage Points)**

The improvement metric in Headline Testing is expressed in **percentage points (
pp)**, not just as a percent. This distinction matters:

 * **Percent** expresses a relative change.
 * **Percentage point (pp)** expresses an absolute difference between two percentages.

For example, if the control headline has a CTR of 4% and a variant has a CTR of 
5%, the improvement is **1 percentage point (pp)**, translating to a **25% relative
increase**. Using percentage points helps avoid confusion and keeps comparisons 
clear and accurate.

Understanding the difference ensures better decision-making when interpreting small
changes in click-through rates.

## **Best Practices**

 * Use your most preferred headline as the control.
 * Focus traffic on fewer variants and longer experiments.
 * Let win chance guide obvious choices; use help chance for nuanced decisions.
 * Weigh CTR improvement against editorial quality — a minor gain may not warrant
   change.

Headline Testing is meant to **support editorial judgment**, not override it. These
new features make it easier to blend data and intuition, driving better decisions.

Last updated: November 06, 2025