A/B testing on a small website: how many visitors you really need, and what to do without enough traffic
A/B testing on a small website: how many visitors one test really needs, what to do without enough traffic, how to test without hurting SEO, and the tools.
A/B testing sounds like exactly the right thing to do: two versions of the same page, and the data decides. In practice, most small business websites in Israel do not have enough visitors for the test to give a real answer, and the conclusion that comes out of it is noise. In this article we show how much traffic you really need, using a sample size calculator rather than a gut feeling. Then we set out what to do when you do not have that volume, how to test without damaging your organic rankings, and which tools are left after Google shut Google Optimize down.
What A/B tests do, and what they cannot tell you
An A/B test splits visitors randomly between two versions over the same period of time, and compares one metric between them. The random split and the simultaneity are the whole point. Comparing January with February is not an A/B test, because between those two periods the campaigns changed, and so did the weather and the holidays.
What a test like that will not give you: it does not explain why a version won, it does not pick up small changes when traffic is thin, and it does not measure what happens after the lead. A version that brings in more enquiries can bring in less serious ones. That is why we always test against the business metric, not just against the conversion rate. What counts as a reasonable conversion rate, and what to compare it with, we set out in our article on conversion rates on a website.
How many visitors one test needs, and the number that surprises small businesses
This is the part most Hebrew guides skip. Take an example: a site with a conversion rate of 3% that wants to detect an improvement to 4.5% — an improvement of 1.5 percentage points.
According to Evan Miller's sample size calculator, with 95% significance and 80% statistical power, you need around 2,170 visitors for each version. That is more than 4,300 visitors for a single test. A site getting 1,000 visitors a month would need four months, and in the meantime the business will have changed.
And that is still the comfortable scenario. Sample size grows in inverse proportion to the square of the improvement you are looking for, so an improvement half that size needs roughly four times as many visitors. From that come two rules of thumb we work by:
- Test big changes, not nuances. Swapping the offer, the headline or the structure of a form can move things by whole points. A shade of a button usually will not.
- Count conversions, not just visitors. A test that ends with ten enquiries in each version has proved nothing, however large the gap between them looks.
Two more rules that save you from wrong conclusions: run whole weeks, because visitor behaviour on a Sunday is different from a Friday, and do not stop the test the moment it looks good. Peeking daily and stopping at the flattering moment is the fastest way to discover a "win" that does not exist.
Not enough traffic? Here is how you test anyway
Most of the sites we come across will not reach the numbers above, so the honest answer is not "run the test anyway". There are four alternatives that do work.
- Test where the traffic is. Usually that is your paid advertising rather than your website. In Google you can run an experiment between two versions of an ad or a landing page, and in Meta you can test creative against creative. Thousands of impressions build up there in a week. How that is managed in practice we explained in our guide to managing Google campaigns.
- Test subject lines in email. A mailing list of a few thousand recipients gives you an indication within a day, because open rates are far higher than conversion rates on a website. Every email platform has this kind of test built in.
- Compare two periods, with a diary. Change one thing, compare two weeks against two weeks, and write down every other change that happened: budget, a holiday, a press mention, the season. This is not a statistical test, and the result has to be treated as an indication only.
- Test qualitatively. Session recordings, heatmaps and a direct question to your five most recent customers ("what almost stopped you getting in touch?") reveal obstacles that a quantitative test would not have found in six months.
Our rule: when there is no traffic, you first fix what is known to work. The list of changes that almost always pay off is collected in our conversion optimisation checklist, and building the whole page in the right order appears in our guide to building a landing page.
What to test first, and how to keep a testing diary
Our order of priority, from the biggest to the smallest: the offer itself, the main headline, the length of the form and the fields in it, the wording of the button, and after that images and design. Button wordings by type of business are collected in our article on the call to action button.
In every test you change one thing. If you changed both the headline and the form and the result improved, you will not know which one worked, and you will not be able to repeat the success on the next page. Keep a simple diary, a spreadsheet will do:
| Field | What you write |
|---|---|
| Dates | When the test started and when it ended |
| The hypothesis | "If we shorten the form to a name and a phone number, the number of enquiries will rise" |
| The metric | Enquiries, not clicks |
| What else changed | Budget, a holiday, a new campaign |
| The decision | Implement, reject or test again |
That diary is worth more than any tool. After a year you have a list of what worked and what did not in your own specific business, and that is exactly the information no general guide has.
Testing without hurting your rankings: what Google asks for
When you test two different addresses, there are consequences for organic search. Google Search Central sets out what to do, and the three central points are:
- No cloaking. You must not show Googlebot one piece of content and visitors another. Google defines that as cloaking and a breach of its spam policies.
- A temporary redirect, 302 and not 301. When the test redirects visitors to an alternative address, the redirect should be temporary, so that the original address stays in the index.
- A canonical tag on the variations. Google recommends marking the original address as the preferred one on the alternative versions, rather than blocking them from indexing.
On top of that, Google asks you to run an experiment only for as long as it takes, and to remove it when it is done. An experiment that runs for a long time with no reason may be read as an attempt to mislead the search engines. For most small sites none of this is relevant at all, because they are testing one version inside the same page, with no extra address.
The tools in 2026, after Google Optimize shut down
Plenty of Hebrew guides still recommend Google Optimize. That tool no longer exists: according to Google, Google Optimize and Optimize 360 are no longer available as of 30 September 2023, and every experiment and personalisation that was live on that day was stopped. On the same page Google notes that it worked on integrations with A/B testing providers, and lists AB Tasty, Optimizely and VWO in alphabetical order, and that the API was opened so that any tool could connect to Google Analytics.
What that means in practice for a small business:
- Dedicated tools like the ones Google lists are excellent, but they are built for high-traffic sites, and their pricing changes. Check the pricing on the vendor's site on the day you choose.
- What you already have. Your advertising platform, your email system and your website builder usually include a built-in testing capability at no extra cost. That is the right place to start.
- Plugins for WordPress and Elementor exist and suit simple tests, as long as they do not slow the page down. A slower page can lower conversions on its own, and then you are measuring the plugin instead of the idea.
What Simple Web does about it
Simple Web is an AI-first marketing agency from Bnei Brak, certified Meta partners and Google advertising experts, working with 200+ businesses and holding 28 five-star Google reviews. We do not sell A/B testing to a website that has no traffic. Instead we start from the things that are known to work, and move the testing to where there is enough data: the campaigns.
A landing page we build comes with conversion tracking, a heatmap and the groundwork for A/B testing, so that when traffic grows you can test without rebuilding. A page like that is usually ready within 5–7 working days, and the price is quoted according to scope. In paid media management, which starts at ₪3,500 a month and does not include the advertising budget, experiments in ads and landing pages are part of the ongoing work, with a monthly report on cost per lead. It all starts with a free diagnostic call.
Summary
A/B testing is an excellent tool, as long as you have enough visitors for the result to be real. Before you start, put your own numbers into a sample size calculator and see how long the test would take. If the answer is months, test where the traffic is, compare periods with a properly kept diary, and first fix what is known to work. And when you do test, change one thing, run whole weeks, and respect Google's rules on cloaking and redirects. Want to know whether your site has enough traffic to test? Book a free diagnostic call.
Sources
- Evan Miller: Sample Size Calculator: the sample size required: around 2,170 visitors per version for an improvement from 3% to 4.5%, at 95% significance and 80% power
- Google Search Central: Website testing and Google Search: the ban on cloaking, the temporary 302 redirect, the canonical tag, and a reasonable experiment length
- Google Analytics Help: Google Optimize (Sunset September 2023): the shutdown of Google Optimize on 30 September 2023 and the providers Google lists
