A/B testing.

02 · THE BRIEFING

1. I
   When to consider it
   You have a UX audit that delivers 5 to 10 hypotheses, but you do not know which to start with. Your team argues over a button colour. Your previous redesigns did not move the conversion rate (visitors who take action). The moment instinct is no longer enough.
2. II
   Why it matters
   Without a test, each redesign is a bet. You ship a new landing page, you push to prod, you wait. Three months later, you still do not know if it is better or worse. The hidden cost: good ideas killed for lack of proof, and bad ones staying live by inertia.
3. III
   What you get back
   A quantified verdict per test, validated on a duration and volume that make the result reliable (statistically). A log of past tests, won or lost, with their real measured lift. An internal testing culture instead of the argument from authority.
4. IV
   How we run it
   We start by reviewing what the audit surfaced, before we draft a single variant. Tools: GA4 Audiences (Google Analytics segmentation) for traffic split and conversion events, Hotjar (heatmap + in-page surveys) to watch behaviour on each variant, Microsoft Clarity (free heatmap + session replay) as backup for suspicious sessions. One hypothesis at a time, never two competing variants on the same funnel.
5. V
   What it unlocks
   A test pipeline that runs without debate, month after month. A product roadmap steered by data, not by opinion. An accumulation of knowledge about what really works with your visitors.

We get back to you within the week · scoping before any quote.

04 · WHAT WE WON'T WRITE IN AN RFP

An A/B test is a business decision tool. Not a design deliverable nor a benchmark.

An A/B test is only useful if you are willing to kill your favourite ideas when the data calls against them. Otherwise, you run measurement theatre. The non-negotiable rule: 14-day minimum duration, sample size calculated before launch, one hypothesis per test. If the ICP (ideal customer profile) is not validated upstream, the right order is first audit the funnel, then test.

• 01

  Data-sourced decision

  Each call rests on a quantified verdict, not on the opinion of the last person who spoke. You validate the real lift, not the placebo effect of the first two weeks live.

• 02

  Knowledge capitalisation

  Lost tests count as much as winners. You build a log of what works and what does not in your sector, your ICP, your key pages. Lasting heritage, not a throwaway deliverable.

• 03

  Exit from opinion battles

  No more meetings on button colour or CTA wording. The data calls, the team moves forward. You free brain time for the calls that are actually worth a debate.

• 04

  Accelerated product roadmap

  You know what to push live, what to kill, what to send back to reflection. No more six-month redesign cycles for an uncertain result. You move forward through sourced micro-wins.

05 · THE PLAY-BY-PLAY

Four steps per test. Four weeks on average. Zero production freeze.

1. 01

   We pick the hypothesis that pays.

   We start from the audit backlog (5 to 10 hypotheses prioritised on impact / effort). We pick one, the one where the business lift × traffic volume potential is highest. We formulate the hypothesis in `if X then Y measured by Z` format. No opportunistic tests outside the backlog.

2. 02

   We size before launching.

   Calculation of the sample size needed to detect the target lift at 95 % confidence. Minimum duration scoping (14 days to cover a full weekly cycle). If your traffic does not let us reach the size in 4 weeks, we revisit the hypothesis or we wait.

3. 03

   We launch and stop touching.

   Variant B in parallel with version A, 50/50 split via GA4 (opens in a new window) Audiences (Google Analytics segmentation). No production freeze. Hotjar (opens in a new window) watches both variants in parallel, Microsoft Clarity (opens in a new window) as backup for suspicious sessions. No modification during the test window. No premature stop, even if A seems to win at 7 days.

4. 04

   We call it, we document, we archive.

   At the end of the window, we read the quantified verdict (real lift, confidence interval, device and source segmentation). Call: push B live, kill B, or relaunch a refined variant. Versioned Notion documentation, log of won and lost tests up to date, brief for the next iteration.

07 · NOT YET FOR YOU IF

Three cases where A/B testing is not the top priority.

• Your traffic is under 5,000 conversions a month.

  At this volume, reaching a statistically honest sample size takes 8 weeks minimum per test. Across 10 hypotheses, two years to validate half. Better to prioritise traffic and the audit first.

• You are not ready to kill your favourite variants.

  If the final call goes back to a leader who will not read the verdict, A/B testing becomes measurement theatre. The cost of a test cycle is not paid back by an already-made decision. Better to clarify governance first.

• Your conversion tracking is broken or fuzzy.

  Without reliable GA4 on conversion events, test verdicts rest on truncated data. You validate a false winner, you push the wrong variant live. The right order is tracking first, test second.

We open your hypothesis backlog. We tell you which ones pay, which ones fall.