The Pre-Launch Experiment Checklist

Part One

Hypothesis & Rationale

A test without a real hypothesis can't teach you anything, even if it "wins." You'll know something moved — but not why, which means you can't repeat it or generalize it.

The format, and why it matters

Every hypothesis should name a mechanism, not just an outcome.

Write it as: We believe [change] for [audience] will cause [metric] to [move] because [evidence]. The because clause is the part people skip — and it's the one that actually makes the test diagnostic. "We believe adding social proof to the checkout page will increase conversion" tells you what you hope happens. Add "...because users cited trust as their top drop-off reason in exit surveys" and now, whichever way the test goes, you're testing a specific belief about *why* customers behave the way they do — not just throwing a change at the wall.

Backed by evidence, not a hunch

Evidence sources worth naming explicitly

Qualitative research (interviews, session recordings), funnel drop-off data, support tickets, competitor analysis, or a prior test's learning all count as evidence. "I have a feeling" doesn't — not because feelings are always wrong, but because a hypothesis with no cited evidence gives you nothing to update when the result comes in.

Isolate one change

Confounding kills interpretability

If you change the headline and the button color in the same test, and the metric moves, you don't know which change did it — or whether it was an interaction between the two. Isolating the change doesn't mean every test has to be tiny; it means you should be able to name, in one sentence, the single thing that's different between control and variant.

Checklist — Part 01

Written as: We believe [change] for [audience] will cause [metric] to [move] because [evidence]

Backed by real evidence (data/research), not a hunch

Change is isolated — testing one thing

Part Two

Metrics Defined Before Launch

Deciding what counts as success after looking at the data is one of the most common — and least visible — ways experimentation programs lose credibility.

The three roles a metric can play

Primary, secondary, and guardrail aren't interchangeable

Primary is the one metric that decides ship or no-ship — there should be exactly one, not three. Secondary metrics are diagnostic: they help explain *why* the primary moved (or didn't), even though they don't decide the outcome. Guardrail metrics exist purely to catch harm — including metrics owned by other teams whose surfaces your test might quietly affect (a classic case: a checkout test that lifts short-term conversion while cannibalizing a different product line's revenue).

Why the threshold has to be set first

Goalpost-moving is a real bias, not just a cliché

If "success" is only defined after you've seen a +2% lift, there's a strong temptation to call +2% a win even if you'd have shrugged off +2% before the test ran. Pre-registering the threshold — "we'll ship if primary improves by at least X%, with no guardrail regressions" — removes that temptation entirely and makes the ship/no-ship call mechanical, not political.

Checklist — Part 02

One primary metric named — the decision-maker

Secondary metrics identified

Guardrail metrics set — including other teams' metrics

Success threshold agreed up front

Part Three

Statistical Setup

This is the part most teams under-invest in, and it's the part most likely to silently invalidate an otherwise well-designed test.

Sample size & MDE

You can't calculate one without deciding the other first

Sample size depends on three inputs: your baseline conversion rate, your desired statistical power (commonly 80% — the chance of detecting a real effect if one exists), and your Minimum Detectable Effect (MDE) — the smallest lift you actually care about catching. The relationship is intuitive once you see it: the smaller the effect you want to be able to detect, the larger the sample you need. Teams that skip setting an MDE upfront tend to either run tests far longer than necessary, or stop early with a sample too small to detect anything but a huge, unrealistic effect.

Test duration

A full business cycle, not "until it looks done"

Customer behavior shifts across the week — weekday and weekend traffic can behave completely differently, and novelty effects (people reacting to *change itself*, not the change's merits) fade over the first few days. A test that runs 3 days might capture a real effect, a novelty spike, or pure day-of-week noise — there's no way to tell them apart. The standard fix is running at least one full week, ideally two, regardless of how "significant" things look on day 2.

The stopping rule

Why "no peeking" is a statistical requirement, not a nicety

Checking for statistical significance repeatedly during a test — and stopping the moment it crosses your threshold — inflates your real false-positive rate well beyond the 5% you think you're running at. Every additional peek is another chance for random noise to cross the line by luck alone. The fix is deciding your stopping rule before the test starts: either commit to a fixed sample size/duration and don't look until it's reached, or use a sequential testing method specifically built to allow early stopping without breaking the statistics.

Checklist — Part 03

Sample size calculated in advance

Minimum detectable effect (MDE) decided

Test duration covers full business cycles (≥1 week)

Stopping rule agreed — no peeking

Part Four

Tracking & QA

A statistically perfect test design still produces garbage results if the instrumentation underneath it is broken. This is the most common invisible killer of experiment validity.

Sample Ratio Mismatch (SRM)

The single most important health check most teams skip

SRM is what happens when the actual traffic split between control and variant differs meaningfully from what you intended — say, 48/52 instead of 50/50. It sounds minor. It isn't: it's a signal that your randomization broke somewhere (a bucketing bug, bot traffic hitting one arm disproportionately, a redirect issue), which means the two groups may no longer be comparable in ways that have nothing to do with your hypothesis. When SRM is present, the whole test's results become untrustworthy — not just slightly noisy. The standard check is a simple statistical test comparing your observed split to the expected one; most experimentation platforms surface this automatically, but it's worth confirming manually if yours doesn't.

Cross-device rendering

A broken variant on one segment can tank your whole result

A layout bug that only appears on one browser or one device size can move your metric for reasons that have nothing to do with what you're actually testing. Checking render quality across your key device/browser segments before launch is cheap; discovering it after the test has been running for a week is not.

Checklist — Part 04

Events fire correctly in both variants — tested

Instrumentation / tracking validated

No sample-ratio mismatch or bucketing issues

Variant renders correctly across devices

Part Five

Alignment & Logistics

The statistics can be flawless and a test can still fail the organization, if nobody agreed in advance what happens next.

Pre-agree the outcome, not just the metric

"What ships if it wins or loses" is a different question than "did it work"

Even a clean, statistically significant win can stall for weeks if stakeholders haven't already agreed what happens next — who builds it into production, on what timeline, with what caveats. Deciding this before launch turns a results readout into a scheduled action, not a fresh negotiation.

Documentation and ownership

Undocumented tests get re-run; unowned tests get ignored

A test that isn't logged in your backlog or results system will likely get re-litigated by someone else in six months who didn't know it already ran. A test with no named owner tends to just sit — nobody's watching it, nobody actions the result, and the learning is lost even if the test itself was good.

Checklist — Part 05

Stakeholders aligned on what ships if it wins or loses

Test documented in your backlog / system

Launch date and owner assigned

Bonus

Common pre-launch failure modes.

If you only have five minutes before a launch, scan this list — it's the fastest way to catch the most common ways experiments quietly go wrong.

The "vibes" hypothesis

No stated mechanism, so a failed test teaches nothing and a won test can't be repeated deliberately.

Underpowered by default

Sample size never calculated; the test just runs "until it feels done," which usually means it's too small to detect anything but a huge effect.

SRM ignored

Nobody checks the traffic split balance, and a silent bucketing bug invalidates the entire test without anyone noticing.

Goalpost-moving

Success criteria get decided after seeing the data, quietly eroding trust in every result that follows.

Orphaned tests

No named owner, no pre-agreed next step — the result sits in a dashboard and nothing is ever actioned.

Glossary

Terms worth knowing.

MDE

Minimum Detectable Effect — the smallest lift you actually care about being able to detect. Set before the test; it drives your required sample size.

SRM

Sample Ratio Mismatch — when the actual traffic split between control and variant differs meaningfully from what was intended, signaling a broken randomization.

Stopping Rule

A pre-committed rule for when a test ends and results are read — set in advance specifically to avoid inflating false positives by "peeking" repeatedly.

Guardrail Metric

A metric that doesn't decide the test's outcome, but exists purely to catch unintended harm — including metrics owned by other teams.

Statistical Power

The probability your test detects a real effect if one truly exists — commonly targeted at 80%. Lower power means real effects can go undetected.

The Tool

The full checklist, all in one place.

Everything above, condensed into one printable page — run through this in the 24–48 hours before any test goes live.

Pre-Launch Experiment Checklist

Print / Save PDF

0 / 18 checked

01 · Hypothesis & Rationale

Written as: We believe [change]... because [evidence]

Backed by real evidence, not a hunch

Change is isolated — testing one thing

02 · Metrics Defined

One primary metric named

Secondary metrics identified

Guardrail metrics set, incl. other teams'

Success threshold agreed up front

03 · Statistical Setup

Sample size calculated in advance

MDE decided

Duration covers ≥1 full business cycle

Stopping rule agreed — no peeking

04 · Tracking & QA

Events fire correctly in both variants

Instrumentation validated

No sample-ratio mismatch / bucketing issues

Renders correctly across devices

05 · Alignment & Logistics

Stakeholders aligned on what ships if it wins/loses

Test documented in backlog / system

Launch date and owner assigned

Next Step

One test is one piece. Here's how to see the whole program.

Adasight's Experimentation Gap Analysis is a structured look across the four places programs typically get stuck — with a clear roadmap for closing what's holding yours back.

Data

Can your team get to the data needed to form a hypothesis?

Insights

Do results turn into a clear "why" — or end in a dashboard?

Experimentation Practice

Is there a clear success metric and process behind every test?

Is your experimentation data structured enough to feed AI?

See the four gaps →

What the audit delivers:

A current-state assessment across process, tooling, and skills.

The specific gaps keeping tests from producing trustworthy results.

A prioritized roadmap ranked by impact and effort.

30 minutes. No pitch — just a clear view of your biggest opportunities.