MarketingCollectible prompt
A/B test plan with a real hypothesis and sample size
Act as an experimentation lead. Plan one A/B test on {{page_or_feature}} of {{website_url}}. The goal: {{goal}}. The change I want to test, and why I think it will help:
{{test_idea}}
1. Hypothesis, in this form: because we saw a specific observation in data or research, we believe a specific change for a specific audience will move a specific metric, and we'll know at the planned sample size. If my reason is a hunch, say so and suggest the fastest way to get evidence first.
2. Metrics: one primary metric tied to the goal (completed bookings, for example, not button clicks); two or three guardrails that must not get worse, such as refunds, bounce rate or page speed; and how each is measured.
3. Sample size: ask for this page's current conversion rate and weekly visitors, then calculate visitors per variant for the smallest lift worth acting on, at 95% confidence and 80% power. Show the math and the expected run time. Run whole weeks, at least two, to cover weekday patterns. If traffic can't finish it in about six weeks, say so plainly and suggest a bolder change, a before-and-after comparison with caveats, or user testing instead.
4. Setup: a 50/50 split, sticky per visitor, no flicker (assign on the server or at the edge if possible), consent respected, and a sample ratio check to confirm the split works.
5. Decision rule, written before launch: no peeking and stopping early on a good day; what we do if it wins, loses or ties; and which segments we'll check, such as mobile versus desktop, without fishing for a winner.
Return a one-page test plan I can share, with a results template.
Marketingnvoka.com/library/nvoka-ab-test-plan-with-a-hypothesis-and-sample-sizeScan to open
Marketing
A/B test plan with a real hypothesis and sample size
Plan one A/B test properly: hypothesis, metric, sample size, run time and a decision rule set in advance.
Curated by Nvoka
ClaudeChatGPTGeminiClaude Code
Make it yours
Fill in the blanks and change any word. Only your copy changes, never the card.
Fill in the blanks
0 of 4 filledYour prompt
Act as an experimentation lead. Plan one A/B test on {{page_or_feature}} of {{website_url}}. The goal: {{goal}}. The change I want to test, and why I think it will help:
{{test_idea}}
1. Hypothesis, in this form: because we saw a specific observation in data or research, we believe a specific change for a specific audience will move a specific metric, and we'll know at the planned sample size. If my reason is a hunch, say so and suggest the fastest way to get evidence first.
2. Metrics: one primary metric tied to the goal (completed bookings, for example, not button clicks); two or three guardrails that must not get worse, such as refunds, bounce rate or page speed; and how each is measured.
3. Sample size: ask for this page's current conversion rate and weekly visitors, then calculate visitors per variant for the smallest lift worth acting on, at 95% confidence and 80% power. Show the math and the expected run time. Run whole weeks, at least two, to cover weekday patterns. If traffic can't finish it in about six weeks, say so plainly and suggest a bolder change, a before-and-after comparison with caveats, or user testing instead.
4. Setup: a 50/50 split, sticky per visitor, no flicker (assign on the server or at the edge if possible), consent respected, and a sample ratio check to confirm the split works.
5. Decision rule, written before launch: no peeking and stopping early on a good day; what we do if it wins, loses or ties; and which segments we'll check, such as mobile versus desktop, without fishing for a winner.
Return a one-page test plan I can share, with a results template.