Conversion Optimisation: Stop Testing Tactics, Start Testing Assumptions
The Metrics That Actually Tell You Something
Conversion rate is the obvious primary metric, but it is rarely sufficient on its own. The supporting metrics that tell you whether your CRO programme is generating genuine commercial value are the ones most teams underweight.
Revenue per visitor is more useful than conversion rate in isolation because it accounts for changes in average order value that a conversion rate change might obscure. A test that increases conversion rate by 8% but reduces average order value by 15% is not a win. Revenue per visitor catches that. Conversion rate alone does not.
Understanding the difference between click rate and click-through rate matters when you are measuring engagement across different stages of the funnel. These are not interchangeable terms, and conflating them produces misleading performance reads.
Downstream metrics, particularly lead quality for B2B and repeat purchase rate for e-commerce, are the ultimate test of whether a CRO win is real. I have seen conversion rate improvements that looked impressive in the dashboard and were quietly eroding business performance because the users converting were not the users who would become valuable customers. Optimising for conversion without understanding who is converting is a way of making the wrong number go up.
Reducing bounce rate is worth tracking as a secondary signal, particularly when you are making changes to page layout or load performance, but treat it as a leading indicator rather than an outcome metric. Bounce rate tells you something about initial engagement. It does not tell you whether the people who stayed converted, or converted at the value you needed.
For a broader view of how CRO connects to commercial performance across the full marketing mix, the CRO and Testing hub at The Marketing Juice covers the strategic and measurement dimensions in more depth.
The Compounding Logic of Getting This Right
The reason CRO deserves serious attention is not that any individual test will transform your business. It is that a well-run programme compounds. A 10% improvement in conversion rate on a channel spending £500,000 a year is worth £50,000 in equivalent media budget. Do that across three channels, sustain it over two years, and you have created material commercial value without spending an additional pound on acquisition.
But that compounding only happens if the programme is testing the right things. Twelve months of button colour tests will not get you there. Twelve months of assumption testing, built on rigorous qualitative research, structured hypothesis generation, and honest measurement of downstream outcomes, will.
The discipline required is not technical. It is intellectual. It means being willing to challenge what the team believes to be true about its users, being willing to call a test inconclusive when the data does not support a clean conclusion, and being willing to follow the evidence even when it leads somewhere uncomfortable, like the conclusion that the page is fine and the problem is the offer.
That is harder than running another A/B test. It is also the only version of CRO that actually works.
About the Author
Keith Lacy is a marketing strategist and former agency CEO with 20+ years of experience across agency leadership, performance marketing, and commercial strategy. He writes The Marketing Juice to cut through the noise and share what works.
Frequently Asked Questions
When to Bring in External CRO Expertise
There is a version of this conversation that turns into an agency pitch, and I want to avoid that. External CRO expertise is genuinely valuable in specific situations, and genuinely unnecessary in others.
It is valuable when your in-house team has the technical capability to run tests but lacks the experience to design hypotheses that go beyond surface-level optimisation. It is valuable when you need an outside perspective to challenge assumptions that have become invisible through familiarity. It is valuable when you are entering a new market or launching a new product and do not have historical data to anchor your hypothesis generation.
It is less valuable when the real problem is strategic, not executional. If your offer is wrong, your pricing is off, or your audience targeting is too broad, no CRO specialist will fix that. The decision to hire a CRO specialist should follow an honest assessment of where the constraint actually is.
The most effective external engagements I have seen are time-bounded, hypothesis-driven, and structured around knowledge transfer. The goal is not to create a dependency on an external team. It is to build internal capability and leave the organisation with a better understanding of its users than it had before.
The Metrics That Actually Tell You Something
Conversion rate is the obvious primary metric, but it is rarely sufficient on its own. The supporting metrics that tell you whether your CRO programme is generating genuine commercial value are the ones most teams underweight.
Revenue per visitor is more useful than conversion rate in isolation because it accounts for changes in average order value that a conversion rate change might obscure. A test that increases conversion rate by 8% but reduces average order value by 15% is not a win. Revenue per visitor catches that. Conversion rate alone does not.
Understanding the difference between click rate and click-through rate matters when you are measuring engagement across different stages of the funnel. These are not interchangeable terms, and conflating them produces misleading performance reads.
Downstream metrics, particularly lead quality for B2B and repeat purchase rate for e-commerce, are the ultimate test of whether a CRO win is real. I have seen conversion rate improvements that looked impressive in the dashboard and were quietly eroding business performance because the users converting were not the users who would become valuable customers. Optimising for conversion without understanding who is converting is a way of making the wrong number go up.
Reducing bounce rate is worth tracking as a secondary signal, particularly when you are making changes to page layout or load performance, but treat it as a leading indicator rather than an outcome metric. Bounce rate tells you something about initial engagement. It does not tell you whether the people who stayed converted, or converted at the value you needed.
For a broader view of how CRO connects to commercial performance across the full marketing mix, the CRO and Testing hub at The Marketing Juice covers the strategic and measurement dimensions in more depth.
The Compounding Logic of Getting This Right
The reason CRO deserves serious attention is not that any individual test will transform your business. It is that a well-run programme compounds. A 10% improvement in conversion rate on a channel spending £500,000 a year is worth £50,000 in equivalent media budget. Do that across three channels, sustain it over two years, and you have created material commercial value without spending an additional pound on acquisition.
But that compounding only happens if the programme is testing the right things. Twelve months of button colour tests will not get you there. Twelve months of assumption testing, built on rigorous qualitative research, structured hypothesis generation, and honest measurement of downstream outcomes, will.
The discipline required is not technical. It is intellectual. It means being willing to challenge what the team believes to be true about its users, being willing to call a test inconclusive when the data does not support a clean conclusion, and being willing to follow the evidence even when it leads somewhere uncomfortable, like the conclusion that the page is fine and the problem is the offer.
That is harder than running another A/B test. It is also the only version of CRO that actually works.
About the Author
Keith Lacy is a marketing strategist and former agency CEO with 20+ years of experience across agency leadership, performance marketing, and commercial strategy. He writes The Marketing Juice to cut through the noise and share what works.
Frequently Asked Questions
Building a Test Roadmap That Generates Compounding Insight
The best CRO programmes I have seen share a common structural feature: they are designed to generate knowledge, not just results. Each test is scoped to answer a specific question about user behaviour, and the answer, whether the test wins or loses, feeds directly into the next hypothesis.
This is fundamentally different from a test roadmap that is just a list of things someone thought might improve conversion. A roadmap built on assumptions produces a series of disconnected experiments. A roadmap built on questions produces a body of knowledge about how your users think, what they need, and where the real friction is.
A useful CRO checklist helps ensure you are covering the right ground, but a checklist is a floor, not a ceiling. The ceiling is a programme that gets progressively smarter about your specific audience over time.
Practically, this means documenting not just test results but the reasoning behind each test. What assumption were you challenging? What did you expect to find? What did you actually find? What does that tell you about user behaviour that you did not know before? A well-maintained test log is one of the most valuable assets a CRO programme can produce, and it is almost universally neglected.
It also means being disciplined about test sequencing. Run the structural tests before the cosmetic ones. Test the offer before you test the headline. Test the headline before you test the button. The sequence matters because each layer of the problem depends on the one below it.
When to Bring in External CRO Expertise
There is a version of this conversation that turns into an agency pitch, and I want to avoid that. External CRO expertise is genuinely valuable in specific situations, and genuinely unnecessary in others.
It is valuable when your in-house team has the technical capability to run tests but lacks the experience to design hypotheses that go beyond surface-level optimisation. It is valuable when you need an outside perspective to challenge assumptions that have become invisible through familiarity. It is valuable when you are entering a new market or launching a new product and do not have historical data to anchor your hypothesis generation.
It is less valuable when the real problem is strategic, not executional. If your offer is wrong, your pricing is off, or your audience targeting is too broad, no CRO specialist will fix that. The decision to hire a CRO specialist should follow an honest assessment of where the constraint actually is.
The most effective external engagements I have seen are time-bounded, hypothesis-driven, and structured around knowledge transfer. The goal is not to create a dependency on an external team. It is to build internal capability and leave the organisation with a better understanding of its users than it had before.
The Metrics That Actually Tell You Something
Conversion rate is the obvious primary metric, but it is rarely sufficient on its own. The supporting metrics that tell you whether your CRO programme is generating genuine commercial value are the ones most teams underweight.
Revenue per visitor is more useful than conversion rate in isolation because it accounts for changes in average order value that a conversion rate change might obscure. A test that increases conversion rate by 8% but reduces average order value by 15% is not a win. Revenue per visitor catches that. Conversion rate alone does not.
Understanding the difference between click rate and click-through rate matters when you are measuring engagement across different stages of the funnel. These are not interchangeable terms, and conflating them produces misleading performance reads.
Downstream metrics, particularly lead quality for B2B and repeat purchase rate for e-commerce, are the ultimate test of whether a CRO win is real. I have seen conversion rate improvements that looked impressive in the dashboard and were quietly eroding business performance because the users converting were not the users who would become valuable customers. Optimising for conversion without understanding who is converting is a way of making the wrong number go up.
Reducing bounce rate is worth tracking as a secondary signal, particularly when you are making changes to page layout or load performance, but treat it as a leading indicator rather than an outcome metric. Bounce rate tells you something about initial engagement. It does not tell you whether the people who stayed converted, or converted at the value you needed.
For a broader view of how CRO connects to commercial performance across the full marketing mix, the CRO and Testing hub at The Marketing Juice covers the strategic and measurement dimensions in more depth.
The Compounding Logic of Getting This Right
The reason CRO deserves serious attention is not that any individual test will transform your business. It is that a well-run programme compounds. A 10% improvement in conversion rate on a channel spending £500,000 a year is worth £50,000 in equivalent media budget. Do that across three channels, sustain it over two years, and you have created material commercial value without spending an additional pound on acquisition.
But that compounding only happens if the programme is testing the right things. Twelve months of button colour tests will not get you there. Twelve months of assumption testing, built on rigorous qualitative research, structured hypothesis generation, and honest measurement of downstream outcomes, will.
The discipline required is not technical. It is intellectual. It means being willing to challenge what the team believes to be true about its users, being willing to call a test inconclusive when the data does not support a clean conclusion, and being willing to follow the evidence even when it leads somewhere uncomfortable, like the conclusion that the page is fine and the problem is the offer.
That is harder than running another A/B test. It is also the only version of CRO that actually works.
About the Author
Keith Lacy is a marketing strategist and former agency CEO with 20+ years of experience across agency leadership, performance marketing, and commercial strategy. He writes The Marketing Juice to cut through the noise and share what works.
Frequently Asked Questions
The AI and Personalisation Trap
A few years ago, I sat in a presentation from a major network agency selling an AI-driven personalised creative solution. The pitch was compelling: machine learning to serve the right message to the right user at the right moment, with claimed uplifts of 90% reduction in cost per acquisition and three times the conversion rate.
My first question was about the baseline. What were they comparing against? It turned out the control condition was a set of static display ads that had not been refreshed in eighteen months, were running on a broad audience with no segmentation, and had creative that was, to be generous, uninspired.
The AI had not achieved a breakthrough. It had replaced poor creative with slightly less poor creative and applied basic audience logic that any competent media planner would have applied manually. The uplift was real. The attribution was not. You do not get to claim AI success when the control was broken to begin with.
The same logic applies to personalisation in CRO. Dynamic content, behavioural triggers, and predictive segmentation can generate genuine lift when the fundamentals are already solid. When they are applied on top of a page that does not clearly explain the offer, does not load quickly enough, or does not give users a reason to trust the business, they are sophisticated wallpaper. The underlying problem remains.
Personalisation is a multiplier, not a fix. It amplifies what is already working. If what is already working is weak, personalisation amplifies the weakness.
Building a Test Roadmap That Generates Compounding Insight
The best CRO programmes I have seen share a common structural feature: they are designed to generate knowledge, not just results. Each test is scoped to answer a specific question about user behaviour, and the answer, whether the test wins or loses, feeds directly into the next hypothesis.
This is fundamentally different from a test roadmap that is just a list of things someone thought might improve conversion. A roadmap built on assumptions produces a series of disconnected experiments. A roadmap built on questions produces a body of knowledge about how your users think, what they need, and where the real friction is.
A useful CRO checklist helps ensure you are covering the right ground, but a checklist is a floor, not a ceiling. The ceiling is a programme that gets progressively smarter about your specific audience over time.
Practically, this means documenting not just test results but the reasoning behind each test. What assumption were you challenging? What did you expect to find? What did you actually find? What does that tell you about user behaviour that you did not know before? A well-maintained test log is one of the most valuable assets a CRO programme can produce, and it is almost universally neglected.
It also means being disciplined about test sequencing. Run the structural tests before the cosmetic ones. Test the offer before you test the headline. Test the headline before you test the button. The sequence matters because each layer of the problem depends on the one below it.
When to Bring in External CRO Expertise
There is a version of this conversation that turns into an agency pitch, and I want to avoid that. External CRO expertise is genuinely valuable in specific situations, and genuinely unnecessary in others.
It is valuable when your in-house team has the technical capability to run tests but lacks the experience to design hypotheses that go beyond surface-level optimisation. It is valuable when you need an outside perspective to challenge assumptions that have become invisible through familiarity. It is valuable when you are entering a new market or launching a new product and do not have historical data to anchor your hypothesis generation.
It is less valuable when the real problem is strategic, not executional. If your offer is wrong, your pricing is off, or your audience targeting is too broad, no CRO specialist will fix that. The decision to hire a CRO specialist should follow an honest assessment of where the constraint actually is.
The most effective external engagements I have seen are time-bounded, hypothesis-driven, and structured around knowledge transfer. The goal is not to create a dependency on an external team. It is to build internal capability and leave the organisation with a better understanding of its users than it had before.
The Metrics That Actually Tell You Something
Conversion rate is the obvious primary metric, but it is rarely sufficient on its own. The supporting metrics that tell you whether your CRO programme is generating genuine commercial value are the ones most teams underweight.
Revenue per visitor is more useful than conversion rate in isolation because it accounts for changes in average order value that a conversion rate change might obscure. A test that increases conversion rate by 8% but reduces average order value by 15% is not a win. Revenue per visitor catches that. Conversion rate alone does not.
Understanding the difference between click rate and click-through rate matters when you are measuring engagement across different stages of the funnel. These are not interchangeable terms, and conflating them produces misleading performance reads.
Downstream metrics, particularly lead quality for B2B and repeat purchase rate for e-commerce, are the ultimate test of whether a CRO win is real. I have seen conversion rate improvements that looked impressive in the dashboard and were quietly eroding business performance because the users converting were not the users who would become valuable customers. Optimising for conversion without understanding who is converting is a way of making the wrong number go up.
Reducing bounce rate is worth tracking as a secondary signal, particularly when you are making changes to page layout or load performance, but treat it as a leading indicator rather than an outcome metric. Bounce rate tells you something about initial engagement. It does not tell you whether the people who stayed converted, or converted at the value you needed.
For a broader view of how CRO connects to commercial performance across the full marketing mix, the CRO and Testing hub at The Marketing Juice covers the strategic and measurement dimensions in more depth.
The Compounding Logic of Getting This Right
The reason CRO deserves serious attention is not that any individual test will transform your business. It is that a well-run programme compounds. A 10% improvement in conversion rate on a channel spending £500,000 a year is worth £50,000 in equivalent media budget. Do that across three channels, sustain it over two years, and you have created material commercial value without spending an additional pound on acquisition.
But that compounding only happens if the programme is testing the right things. Twelve months of button colour tests will not get you there. Twelve months of assumption testing, built on rigorous qualitative research, structured hypothesis generation, and honest measurement of downstream outcomes, will.
The discipline required is not technical. It is intellectual. It means being willing to challenge what the team believes to be true about its users, being willing to call a test inconclusive when the data does not support a clean conclusion, and being willing to follow the evidence even when it leads somewhere uncomfortable, like the conclusion that the page is fine and the problem is the offer.
That is harder than running another A/B test. It is also the only version of CRO that actually works.
About the Author
Keith Lacy is a marketing strategist and former agency CEO with 20+ years of experience across agency leadership, performance marketing, and commercial strategy. He writes The Marketing Juice to cut through the noise and share what works.
Frequently Asked Questions
Statistical Significance Is Not the Same as Commercial Relevance
This is the part of CRO that the industry consistently gets wrong, and it costs businesses real money.
Statistical significance tells you that the result you observed is unlikely to be due to chance. It does not tell you that the result is large enough to matter, that it will hold when you scale, or that it applies to your full audience rather than the segment that happened to be in the test window. These are different questions, and treating significance as a green light to ship is how you accumulate a portfolio of tests that technically won but commercially flatlined.
When I was judging the Effie Awards, I saw this pattern repeatedly in submissions. Teams would present statistically significant test results as evidence of a CRO success story, and the commercial numbers would tell a completely different story. The uplift was real in the test. It did not survive contact with the full funnel, the full audience, or the full measurement period.
The questions you should ask before calling a test a win: Is the effect size large enough to matter commercially, not just statistically? Does the result hold across different segments, or is it driven by one subset of users? What is the confidence interval, and does the lower bound still represent a meaningful improvement? What does the result look like downstream, not just at the conversion point being measured?
A 2% uplift in form completion rate is not a win if it comes at the cost of lead quality and your sales team is now working twice as hard to close the same number of deals. CRO that optimises one metric at the expense of another is not optimisation. It is displacement.
The AI and Personalisation Trap
A few years ago, I sat in a presentation from a major network agency selling an AI-driven personalised creative solution. The pitch was compelling: machine learning to serve the right message to the right user at the right moment, with claimed uplifts of 90% reduction in cost per acquisition and three times the conversion rate.
My first question was about the baseline. What were they comparing against? It turned out the control condition was a set of static display ads that had not been refreshed in eighteen months, were running on a broad audience with no segmentation, and had creative that was, to be generous, uninspired.
The AI had not achieved a breakthrough. It had replaced poor creative with slightly less poor creative and applied basic audience logic that any competent media planner would have applied manually. The uplift was real. The attribution was not. You do not get to claim AI success when the control was broken to begin with.
The same logic applies to personalisation in CRO. Dynamic content, behavioural triggers, and predictive segmentation can generate genuine lift when the fundamentals are already solid. When they are applied on top of a page that does not clearly explain the offer, does not load quickly enough, or does not give users a reason to trust the business, they are sophisticated wallpaper. The underlying problem remains.
Personalisation is a multiplier, not a fix. It amplifies what is already working. If what is already working is weak, personalisation amplifies the weakness.
Building a Test Roadmap That Generates Compounding Insight
The best CRO programmes I have seen share a common structural feature: they are designed to generate knowledge, not just results. Each test is scoped to answer a specific question about user behaviour, and the answer, whether the test wins or loses, feeds directly into the next hypothesis.
This is fundamentally different from a test roadmap that is just a list of things someone thought might improve conversion. A roadmap built on assumptions produces a series of disconnected experiments. A roadmap built on questions produces a body of knowledge about how your users think, what they need, and where the real friction is.
A useful CRO checklist helps ensure you are covering the right ground, but a checklist is a floor, not a ceiling. The ceiling is a programme that gets progressively smarter about your specific audience over time.
Practically, this means documenting not just test results but the reasoning behind each test. What assumption were you challenging? What did you expect to find? What did you actually find? What does that tell you about user behaviour that you did not know before? A well-maintained test log is one of the most valuable assets a CRO programme can produce, and it is almost universally neglected.
It also means being disciplined about test sequencing. Run the structural tests before the cosmetic ones. Test the offer before you test the headline. Test the headline before you test the button. The sequence matters because each layer of the problem depends on the one below it.
When to Bring in External CRO Expertise
There is a version of this conversation that turns into an agency pitch, and I want to avoid that. External CRO expertise is genuinely valuable in specific situations, and genuinely unnecessary in others.
It is valuable when your in-house team has the technical capability to run tests but lacks the experience to design hypotheses that go beyond surface-level optimisation. It is valuable when you need an outside perspective to challenge assumptions that have become invisible through familiarity. It is valuable when you are entering a new market or launching a new product and do not have historical data to anchor your hypothesis generation.
It is less valuable when the real problem is strategic, not executional. If your offer is wrong, your pricing is off, or your audience targeting is too broad, no CRO specialist will fix that. The decision to hire a CRO specialist should follow an honest assessment of where the constraint actually is.
The most effective external engagements I have seen are time-bounded, hypothesis-driven, and structured around knowledge transfer. The goal is not to create a dependency on an external team. It is to build internal capability and leave the organisation with a better understanding of its users than it had before.
The Metrics That Actually Tell You Something
Conversion rate is the obvious primary metric, but it is rarely sufficient on its own. The supporting metrics that tell you whether your CRO programme is generating genuine commercial value are the ones most teams underweight.
Revenue per visitor is more useful than conversion rate in isolation because it accounts for changes in average order value that a conversion rate change might obscure. A test that increases conversion rate by 8% but reduces average order value by 15% is not a win. Revenue per visitor catches that. Conversion rate alone does not.
Understanding the difference between click rate and click-through rate matters when you are measuring engagement across different stages of the funnel. These are not interchangeable terms, and conflating them produces misleading performance reads.
Downstream metrics, particularly lead quality for B2B and repeat purchase rate for e-commerce, are the ultimate test of whether a CRO win is real. I have seen conversion rate improvements that looked impressive in the dashboard and were quietly eroding business performance because the users converting were not the users who would become valuable customers. Optimising for conversion without understanding who is converting is a way of making the wrong number go up.
Reducing bounce rate is worth tracking as a secondary signal, particularly when you are making changes to page layout or load performance, but treat it as a leading indicator rather than an outcome metric. Bounce rate tells you something about initial engagement. It does not tell you whether the people who stayed converted, or converted at the value you needed.
For a broader view of how CRO connects to commercial performance across the full marketing mix, the CRO and Testing hub at The Marketing Juice covers the strategic and measurement dimensions in more depth.
The Compounding Logic of Getting This Right
The reason CRO deserves serious attention is not that any individual test will transform your business. It is that a well-run programme compounds. A 10% improvement in conversion rate on a channel spending £500,000 a year is worth £50,000 in equivalent media budget. Do that across three channels, sustain it over two years, and you have created material commercial value without spending an additional pound on acquisition.
But that compounding only happens if the programme is testing the right things. Twelve months of button colour tests will not get you there. Twelve months of assumption testing, built on rigorous qualitative research, structured hypothesis generation, and honest measurement of downstream outcomes, will.
The discipline required is not technical. It is intellectual. It means being willing to challenge what the team believes to be true about its users, being willing to call a test inconclusive when the data does not support a clean conclusion, and being willing to follow the evidence even when it leads somewhere uncomfortable, like the conclusion that the page is fine and the problem is the offer.
That is harder than running another A/B test. It is also the only version of CRO that actually works.
About the Author
Keith Lacy is a marketing strategist and former agency CEO with 20+ years of experience across agency leadership, performance marketing, and commercial strategy. He writes The Marketing Juice to cut through the noise and share what works.
Frequently Asked Questions
The Hierarchy of Conversion Problems Worth Solving
Not all conversion problems are equal, and one of the most useful frameworks I have used is a simple hierarchy of what to fix first. It runs from structural to cosmetic, and the commercial impact follows the same order.
At the top: offer and positioning. If the thing you are selling does not clearly solve a problem the visitor has, or if a competitor is solving it more credibly on the next tab, no amount of page optimisation will close that gap. I have seen teams spend a year on CRO when the real problem was a pricing model that made no sense to the buyer. The page was not the issue.
Below that: technical performance. Page speed has a direct and measurable relationship with conversion rates, particularly on mobile. A page that takes four seconds to load on a mid-range Android device is losing a meaningful share of its potential conversions before a single word is read. This is not a creative problem. It is an engineering problem, and it should be fixed before anything else.
Below that: clarity and trust. Does the user understand what they are being asked to do and why they should do it? Do they trust the business enough to hand over payment details or personal information? These are the questions that assumption testing is designed to answer.
At the bottom: cosmetic details. Button colour, font size, image selection. These are worth testing once everything above them is solid. Testing them first is the equivalent of polishing the windows on a building with structural problems.
The TOFU, MOFU, BOFU framework is a useful lens here too, not because the labels matter but because it forces you to think about where in the buying experience a user is when they hit your page. The conversion problem for a user in awareness mode is completely different from the problem for a user who is ready to buy and just needs a reason to choose you over the alternative.
Statistical Significance Is Not the Same as Commercial Relevance
This is the part of CRO that the industry consistently gets wrong, and it costs businesses real money.
Statistical significance tells you that the result you observed is unlikely to be due to chance. It does not tell you that the result is large enough to matter, that it will hold when you scale, or that it applies to your full audience rather than the segment that happened to be in the test window. These are different questions, and treating significance as a green light to ship is how you accumulate a portfolio of tests that technically won but commercially flatlined.
When I was judging the Effie Awards, I saw this pattern repeatedly in submissions. Teams would present statistically significant test results as evidence of a CRO success story, and the commercial numbers would tell a completely different story. The uplift was real in the test. It did not survive contact with the full funnel, the full audience, or the full measurement period.
The questions you should ask before calling a test a win: Is the effect size large enough to matter commercially, not just statistically? Does the result hold across different segments, or is it driven by one subset of users? What is the confidence interval, and does the lower bound still represent a meaningful improvement? What does the result look like downstream, not just at the conversion point being measured?
A 2% uplift in form completion rate is not a win if it comes at the cost of lead quality and your sales team is now working twice as hard to close the same number of deals. CRO that optimises one metric at the expense of another is not optimisation. It is displacement.
The AI and Personalisation Trap
A few years ago, I sat in a presentation from a major network agency selling an AI-driven personalised creative solution. The pitch was compelling: machine learning to serve the right message to the right user at the right moment, with claimed uplifts of 90% reduction in cost per acquisition and three times the conversion rate.
My first question was about the baseline. What were they comparing against? It turned out the control condition was a set of static display ads that had not been refreshed in eighteen months, were running on a broad audience with no segmentation, and had creative that was, to be generous, uninspired.
The AI had not achieved a breakthrough. It had replaced poor creative with slightly less poor creative and applied basic audience logic that any competent media planner would have applied manually. The uplift was real. The attribution was not. You do not get to claim AI success when the control was broken to begin with.
The same logic applies to personalisation in CRO. Dynamic content, behavioural triggers, and predictive segmentation can generate genuine lift when the fundamentals are already solid. When they are applied on top of a page that does not clearly explain the offer, does not load quickly enough, or does not give users a reason to trust the business, they are sophisticated wallpaper. The underlying problem remains.
Personalisation is a multiplier, not a fix. It amplifies what is already working. If what is already working is weak, personalisation amplifies the weakness.
Building a Test Roadmap That Generates Compounding Insight
The best CRO programmes I have seen share a common structural feature: they are designed to generate knowledge, not just results. Each test is scoped to answer a specific question about user behaviour, and the answer, whether the test wins or loses, feeds directly into the next hypothesis.
This is fundamentally different from a test roadmap that is just a list of things someone thought might improve conversion. A roadmap built on assumptions produces a series of disconnected experiments. A roadmap built on questions produces a body of knowledge about how your users think, what they need, and where the real friction is.
A useful CRO checklist helps ensure you are covering the right ground, but a checklist is a floor, not a ceiling. The ceiling is a programme that gets progressively smarter about your specific audience over time.
Practically, this means documenting not just test results but the reasoning behind each test. What assumption were you challenging? What did you expect to find? What did you actually find? What does that tell you about user behaviour that you did not know before? A well-maintained test log is one of the most valuable assets a CRO programme can produce, and it is almost universally neglected.
It also means being disciplined about test sequencing. Run the structural tests before the cosmetic ones. Test the offer before you test the headline. Test the headline before you test the button. The sequence matters because each layer of the problem depends on the one below it.
When to Bring in External CRO Expertise
There is a version of this conversation that turns into an agency pitch, and I want to avoid that. External CRO expertise is genuinely valuable in specific situations, and genuinely unnecessary in others.
It is valuable when your in-house team has the technical capability to run tests but lacks the experience to design hypotheses that go beyond surface-level optimisation. It is valuable when you need an outside perspective to challenge assumptions that have become invisible through familiarity. It is valuable when you are entering a new market or launching a new product and do not have historical data to anchor your hypothesis generation.
It is less valuable when the real problem is strategic, not executional. If your offer is wrong, your pricing is off, or your audience targeting is too broad, no CRO specialist will fix that. The decision to hire a CRO specialist should follow an honest assessment of where the constraint actually is.
The most effective external engagements I have seen are time-bounded, hypothesis-driven, and structured around knowledge transfer. The goal is not to create a dependency on an external team. It is to build internal capability and leave the organisation with a better understanding of its users than it had before.
The Metrics That Actually Tell You Something
Conversion rate is the obvious primary metric, but it is rarely sufficient on its own. The supporting metrics that tell you whether your CRO programme is generating genuine commercial value are the ones most teams underweight.
Revenue per visitor is more useful than conversion rate in isolation because it accounts for changes in average order value that a conversion rate change might obscure. A test that increases conversion rate by 8% but reduces average order value by 15% is not a win. Revenue per visitor catches that. Conversion rate alone does not.
Understanding the difference between click rate and click-through rate matters when you are measuring engagement across different stages of the funnel. These are not interchangeable terms, and conflating them produces misleading performance reads.
Downstream metrics, particularly lead quality for B2B and repeat purchase rate for e-commerce, are the ultimate test of whether a CRO win is real. I have seen conversion rate improvements that looked impressive in the dashboard and were quietly eroding business performance because the users converting were not the users who would become valuable customers. Optimising for conversion without understanding who is converting is a way of making the wrong number go up.
Reducing bounce rate is worth tracking as a secondary signal, particularly when you are making changes to page layout or load performance, but treat it as a leading indicator rather than an outcome metric. Bounce rate tells you something about initial engagement. It does not tell you whether the people who stayed converted, or converted at the value you needed.
For a broader view of how CRO connects to commercial performance across the full marketing mix, the CRO and Testing hub at The Marketing Juice covers the strategic and measurement dimensions in more depth.
The Compounding Logic of Getting This Right
The reason CRO deserves serious attention is not that any individual test will transform your business. It is that a well-run programme compounds. A 10% improvement in conversion rate on a channel spending £500,000 a year is worth £50,000 in equivalent media budget. Do that across three channels, sustain it over two years, and you have created material commercial value without spending an additional pound on acquisition.
But that compounding only happens if the programme is testing the right things. Twelve months of button colour tests will not get you there. Twelve months of assumption testing, built on rigorous qualitative research, structured hypothesis generation, and honest measurement of downstream outcomes, will.
The discipline required is not technical. It is intellectual. It means being willing to challenge what the team believes to be true about its users, being willing to call a test inconclusive when the data does not support a clean conclusion, and being willing to follow the evidence even when it leads somewhere uncomfortable, like the conclusion that the page is fine and the problem is the offer.
That is harder than running another A/B test. It is also the only version of CRO that actually works.
About the Author
Keith Lacy is a marketing strategist and former agency CEO with 20+ years of experience across agency leadership, performance marketing, and commercial strategy. He writes The Marketing Juice to cut through the noise and share what works.
Frequently Asked Questions
Conversion optimisation is the practice of systematically improving the percentage of visitors who complete a desired action on your website or landing page. Done well, it compounds: every percentage point improvement in conversion rate multiplies the return on every pound you spend on traffic. Done poorly, which is most of the time, it produces a library of inconclusive tests and a team that mistakes activity for progress.
The gap between those two outcomes is rarely about tools or traffic volume. It is almost always about whether the team is testing the right things. Most CRO programmes test surface details. The ones that generate real commercial lift test the assumptions underneath the page.
Key Takeaways
- Most CRO programmes fail not because of bad execution but because they test cosmetic details rather than the assumptions driving user behaviour.
- Assumption testing, not button-colour testing, is what separates programmes that compound over time from ones that plateau after the first few wins.
- Your conversion funnel has a weakest link. Finding it requires qualitative research, not just analytics dashboards.
- Statistical significance is a threshold, not a result. A test can be significant and still be commercially meaningless.
- The fastest CRO gains often come from fixing broken basics, not from sophisticated personalisation or AI-driven experimentation.
In This Article
- Why Most CRO Programmes Test the Wrong Things
- What Assumption Testing Actually Means
- How to Find the Real Constraint in Your Funnel
- The Hierarchy of Conversion Problems Worth Solving
- Statistical Significance Is Not the Same as Commercial Relevance
- The AI and Personalisation Trap
- Building a Test Roadmap That Generates Compounding Insight
- When to Bring in External CRO Expertise
- The Metrics That Actually Tell You Something
- The Compounding Logic of Getting This Right
Why Most CRO Programmes Test the Wrong Things
Early in my agency career, I watched a client celebrate a 12% uplift in click-through rate from a button colour change. The test was clean, the sample size was sufficient, and the result was statistically significant. The problem was that click-through rate on that button was not the constraint. The page after it was converting at 2.3%. The button test was measuring the wrong thing entirely.
This is the most common failure mode in CRO: optimising a metric that is not the bottleneck. It happens because button colours are easy to test, easy to explain to stakeholders, and produce results quickly. Testing a fundamental assumption about why users are not converting requires more rigour, more patience, and more willingness to sit with an uncomfortable answer.
The industry has a structural incentive problem here. Agencies get paid to run tests. In-house teams get credit for shipping experiments. Neither group is particularly rewarded for concluding that the problem is not on the page at all, that it is in the offer, the pricing model, or the audience targeting. So the tests keep coming, the roadmap stays full, and the conversion rate moves in fractions.
If you want to understand what a rigorous CRO programme actually looks like from the ground up, the full picture is covered in The Marketing Juice CRO and Testing hub, which covers everything from audit methodology to commercial measurement.
What Assumption Testing Actually Means
Every page you build rests on a set of assumptions about your users: what they already know, what they are worried about, what they need to see before they will act. Most of those assumptions were never validated. They were inherited from a brief, borrowed from a competitor, or made by someone who has not spoken to a customer in three years.
Assumption testing means surfacing those beliefs explicitly and designing experiments to challenge them. Not “does the green button outperform the orange button” but “do users on this page already understand what the product does, or are we losing them because we assumed prior knowledge they do not have?” Not “does a shorter form convert better” but “is form length actually the friction point, or is it that users do not trust us enough yet to hand over their details?”
The distinction matters because the test design is completely different. A button colour test takes two days to set up. An assumption test about trust might require adding a social proof section, restructuring the narrative flow of the page, and running the experiment for three to four weeks before you have a readable result. It is harder. It is also the work that moves the number by 30% rather than 3%.
I have run this exercise with clients across retail, financial services, and B2B SaaS. In almost every case, the first assumption audit surfaces at least two or three beliefs that the team holds with complete confidence and that turn out to be wrong when you put them in front of actual users. The most common one: that visitors arrive on the page with a clear understanding of the problem the product solves. They almost never do.
How to Find the Real Constraint in Your Funnel
Before you can test the right things, you need to know where the funnel is actually breaking. Analytics will tell you where people are leaving. It will not tell you why. Those are two completely different questions, and confusing them is how you end up spending six months optimising the wrong stage.
A structured analysis of your conversion funnel is the starting point. Look at drop-off rates at each stage, but treat the numbers as questions rather than answers. If 60% of users leave the product page without scrolling past the fold, that is a signal, not a diagnosis. The cause could be slow load time, a headline that does not match the ad they clicked, a price point that triggers immediate exit, or a layout that buries the information they came for.
Qualitative research is what turns signals into diagnoses. Session recordings, on-page surveys, and user interviews are not optional extras for teams with spare capacity. They are the primary source of hypotheses worth testing. When I was running performance programmes at iProspect, the most valuable hours any analyst spent were not in the data but on calls with users who had dropped out of the funnel. Ten conversations would consistently surface insights that months of A/B testing had missed.
Pay particular attention to bounce rate patterns across different traffic sources. A page that converts well from branded search but haemorrhages visitors from paid social is not a CRO problem in the traditional sense. It is a message-to-market mismatch. The page is fine. The audience expectation is wrong. No amount of button testing will fix that.
The Hierarchy of Conversion Problems Worth Solving
Not all conversion problems are equal, and one of the most useful frameworks I have used is a simple hierarchy of what to fix first. It runs from structural to cosmetic, and the commercial impact follows the same order.
At the top: offer and positioning. If the thing you are selling does not clearly solve a problem the visitor has, or if a competitor is solving it more credibly on the next tab, no amount of page optimisation will close that gap. I have seen teams spend a year on CRO when the real problem was a pricing model that made no sense to the buyer. The page was not the issue.
Below that: technical performance. Page speed has a direct and measurable relationship with conversion rates, particularly on mobile. A page that takes four seconds to load on a mid-range Android device is losing a meaningful share of its potential conversions before a single word is read. This is not a creative problem. It is an engineering problem, and it should be fixed before anything else.
Below that: clarity and trust. Does the user understand what they are being asked to do and why they should do it? Do they trust the business enough to hand over payment details or personal information? These are the questions that assumption testing is designed to answer.
At the bottom: cosmetic details. Button colour, font size, image selection. These are worth testing once everything above them is solid. Testing them first is the equivalent of polishing the windows on a building with structural problems.
The TOFU, MOFU, BOFU framework is a useful lens here too, not because the labels matter but because it forces you to think about where in the buying experience a user is when they hit your page. The conversion problem for a user in awareness mode is completely different from the problem for a user who is ready to buy and just needs a reason to choose you over the alternative.
Statistical Significance Is Not the Same as Commercial Relevance
This is the part of CRO that the industry consistently gets wrong, and it costs businesses real money.
Statistical significance tells you that the result you observed is unlikely to be due to chance. It does not tell you that the result is large enough to matter, that it will hold when you scale, or that it applies to your full audience rather than the segment that happened to be in the test window. These are different questions, and treating significance as a green light to ship is how you accumulate a portfolio of tests that technically won but commercially flatlined.
When I was judging the Effie Awards, I saw this pattern repeatedly in submissions. Teams would present statistically significant test results as evidence of a CRO success story, and the commercial numbers would tell a completely different story. The uplift was real in the test. It did not survive contact with the full funnel, the full audience, or the full measurement period.
The questions you should ask before calling a test a win: Is the effect size large enough to matter commercially, not just statistically? Does the result hold across different segments, or is it driven by one subset of users? What is the confidence interval, and does the lower bound still represent a meaningful improvement? What does the result look like downstream, not just at the conversion point being measured?
A 2% uplift in form completion rate is not a win if it comes at the cost of lead quality and your sales team is now working twice as hard to close the same number of deals. CRO that optimises one metric at the expense of another is not optimisation. It is displacement.
The AI and Personalisation Trap
A few years ago, I sat in a presentation from a major network agency selling an AI-driven personalised creative solution. The pitch was compelling: machine learning to serve the right message to the right user at the right moment, with claimed uplifts of 90% reduction in cost per acquisition and three times the conversion rate.
My first question was about the baseline. What were they comparing against? It turned out the control condition was a set of static display ads that had not been refreshed in eighteen months, were running on a broad audience with no segmentation, and had creative that was, to be generous, uninspired.
The AI had not achieved a breakthrough. It had replaced poor creative with slightly less poor creative and applied basic audience logic that any competent media planner would have applied manually. The uplift was real. The attribution was not. You do not get to claim AI success when the control was broken to begin with.
The same logic applies to personalisation in CRO. Dynamic content, behavioural triggers, and predictive segmentation can generate genuine lift when the fundamentals are already solid. When they are applied on top of a page that does not clearly explain the offer, does not load quickly enough, or does not give users a reason to trust the business, they are sophisticated wallpaper. The underlying problem remains.
Personalisation is a multiplier, not a fix. It amplifies what is already working. If what is already working is weak, personalisation amplifies the weakness.
Building a Test Roadmap That Generates Compounding Insight
The best CRO programmes I have seen share a common structural feature: they are designed to generate knowledge, not just results. Each test is scoped to answer a specific question about user behaviour, and the answer, whether the test wins or loses, feeds directly into the next hypothesis.
This is fundamentally different from a test roadmap that is just a list of things someone thought might improve conversion. A roadmap built on assumptions produces a series of disconnected experiments. A roadmap built on questions produces a body of knowledge about how your users think, what they need, and where the real friction is.
A useful CRO checklist helps ensure you are covering the right ground, but a checklist is a floor, not a ceiling. The ceiling is a programme that gets progressively smarter about your specific audience over time.
Practically, this means documenting not just test results but the reasoning behind each test. What assumption were you challenging? What did you expect to find? What did you actually find? What does that tell you about user behaviour that you did not know before? A well-maintained test log is one of the most valuable assets a CRO programme can produce, and it is almost universally neglected.
It also means being disciplined about test sequencing. Run the structural tests before the cosmetic ones. Test the offer before you test the headline. Test the headline before you test the button. The sequence matters because each layer of the problem depends on the one below it.
When to Bring in External CRO Expertise
There is a version of this conversation that turns into an agency pitch, and I want to avoid that. External CRO expertise is genuinely valuable in specific situations, and genuinely unnecessary in others.
It is valuable when your in-house team has the technical capability to run tests but lacks the experience to design hypotheses that go beyond surface-level optimisation. It is valuable when you need an outside perspective to challenge assumptions that have become invisible through familiarity. It is valuable when you are entering a new market or launching a new product and do not have historical data to anchor your hypothesis generation.
It is less valuable when the real problem is strategic, not executional. If your offer is wrong, your pricing is off, or your audience targeting is too broad, no CRO specialist will fix that. The decision to hire a CRO specialist should follow an honest assessment of where the constraint actually is.
The most effective external engagements I have seen are time-bounded, hypothesis-driven, and structured around knowledge transfer. The goal is not to create a dependency on an external team. It is to build internal capability and leave the organisation with a better understanding of its users than it had before.
The Metrics That Actually Tell You Something
Conversion rate is the obvious primary metric, but it is rarely sufficient on its own. The supporting metrics that tell you whether your CRO programme is generating genuine commercial value are the ones most teams underweight.
Revenue per visitor is more useful than conversion rate in isolation because it accounts for changes in average order value that a conversion rate change might obscure. A test that increases conversion rate by 8% but reduces average order value by 15% is not a win. Revenue per visitor catches that. Conversion rate alone does not.
Understanding the difference between click rate and click-through rate matters when you are measuring engagement across different stages of the funnel. These are not interchangeable terms, and conflating them produces misleading performance reads.
Downstream metrics, particularly lead quality for B2B and repeat purchase rate for e-commerce, are the ultimate test of whether a CRO win is real. I have seen conversion rate improvements that looked impressive in the dashboard and were quietly eroding business performance because the users converting were not the users who would become valuable customers. Optimising for conversion without understanding who is converting is a way of making the wrong number go up.
Reducing bounce rate is worth tracking as a secondary signal, particularly when you are making changes to page layout or load performance, but treat it as a leading indicator rather than an outcome metric. Bounce rate tells you something about initial engagement. It does not tell you whether the people who stayed converted, or converted at the value you needed.
For a broader view of how CRO connects to commercial performance across the full marketing mix, the CRO and Testing hub at The Marketing Juice covers the strategic and measurement dimensions in more depth.
The Compounding Logic of Getting This Right
The reason CRO deserves serious attention is not that any individual test will transform your business. It is that a well-run programme compounds. A 10% improvement in conversion rate on a channel spending £500,000 a year is worth £50,000 in equivalent media budget. Do that across three channels, sustain it over two years, and you have created material commercial value without spending an additional pound on acquisition.
But that compounding only happens if the programme is testing the right things. Twelve months of button colour tests will not get you there. Twelve months of assumption testing, built on rigorous qualitative research, structured hypothesis generation, and honest measurement of downstream outcomes, will.
The discipline required is not technical. It is intellectual. It means being willing to challenge what the team believes to be true about its users, being willing to call a test inconclusive when the data does not support a clean conclusion, and being willing to follow the evidence even when it leads somewhere uncomfortable, like the conclusion that the page is fine and the problem is the offer.
That is harder than running another A/B test. It is also the only version of CRO that actually works.
About the Author
Keith Lacy is a marketing strategist and former agency CEO with 20+ years of experience across agency leadership, performance marketing, and commercial strategy. He writes The Marketing Juice to cut through the noise and share what works.
