Q&A: Can I test with only 5000 visitors a month?

Five thousand visitors a month is not too little traffic to effectively test.

Here's a nice writeup of small sample sizes and simple statistical testing on conversion rates.

If you site does not have transactional components (low conversion, high gain), then it may be too little traffic to measure more highly variable dependent variables like bounce rate, depth of visit, or more refined engagement metrics.

As long as you have a contact, purchase, or download outcome, the techniques illustrated in the AdWords stats post will work for you.

Multivariate testing actually eeks more statistical power out of less traffic than simple split testing; it's just that the conclusion of a single "test" takes longer, but you get more done in a year than doing iterative splits.

I've worked in situations of immense scale -- you still have to test for a full week to deal with day of week effects. For the low traffic scenario, if you focus on key points along the line to conversion, you *can* do data driven design iteration.

The Luna Metrics Blog covers this same topic, with details of the mailing list post that triggered it. For those with the opposite scenario, lots of traffic, check a video excerpt from my web 2.0 expo talk on analytics at scale.