What Is an Incrementality Test and Why Does It Matter for Creative?
An incrementality test is a rigorous experimental method to determine whether a specific advertising exposure actually causes a desired outcome (e.g., purchase, sign-up) beyond what would have happened organically. It compares a test group that sees your ads against a holdout group that is randomly excluded from seeing them. The difference in conversion rates between the two groups is the incremental lift—the true value of the ad.
In the creative process, incrementality tests are crucial because they separate correlation from causation. A spike in conversions after a campaign launch might be due to seasonality, PR, or other factors—not the ad itself. By isolating the ad's effect, you can confidently attribute results to your creative choices: the hook, the value proposition, the call to action, or the visual. This allows you to double down on what actually works and cut what doesn't.
How Is an Incrementality Test Used in Practice?
Running an incrementality test typically involves these steps:
- Define the test group and holdout group. Use random assignment to ensure both groups are comparable. The holdout group should be a small percentage (e.g., 5-10%) of your target audience and must not see any ads from the campaign across all channels.
- Run the campaign normally. The test group receives the ads as planned. The holdout group is suppressed via ad platforms (e.g., Facebook's holdout tests, Google's conversion lift studies) or through your own ad server.
- Measure conversions in both groups. Track the same conversion events (purchases, leads, etc.) for both groups over the same time period.
- Calculate incremental lift. Subtract the conversion rate of the holdout group from the test group. A positive lift means the ads drove incremental conversions. A negative lift suggests the ads may have cannibalized organic conversions or had a negative effect.
Incrementality tests are especially valuable for direct-response campaigns where you need to prove ROI. They can also be applied to brand campaigns by measuring lift in brand searches or site visits. However, they require sufficient sample size and careful setup to avoid contamination (e.g., the holdout group seeing ads via other channels).
Common Mistakes and How to Avoid Them
One frequent mistake is using a too-small holdout group, which reduces statistical power and makes it hard to detect a real lift. Another is failing to suppress ads across all channels—if the holdout group sees a retargeting ad from a different platform, the test is invalid. Also, avoid running incrementality tests for too short a duration; conversions may take time to materialize, especially for high-consideration purchases. Finally, remember that incrementality tests measure the net effect of the entire campaign, not individual creative elements. To test specific creative variations, combine incrementality with A/B testing.
Concrete Example
A D2C skincare brand launches a new ad campaign for a serum. They run an incrementality test on Facebook: 90% of their target audience sees the ads (test group), and 10% is held out. After two weeks, the test group shows a 3% conversion rate, while the holdout group shows a 2% conversion rate. The incremental lift is 1 percentage point (50% relative lift). This confirms the ads are driving additional sales. The brand then tests a different creative angle (e.g., before-and-after vs. ingredient-focused) using a similar incrementality setup to see which angle yields higher incremental lift.