AB Testing - tools

Statistics 202: Statistical Aspects of Data Mining

Thu, 08 Oct 2009 07:42:00 -0700

Highly recommended - a Standford class delivered at Google with the lectures available on Google Video / Youtube. The course uses both the open source stats tool R as well as Excel for the heavy and lightweight ways to get things done. The slides are at stats202.com. Navigating the YouTube UI to work through these vids is painful, so I created a playlist with the videos. Click through for the larger format video: Happy data-mining!

Bing! You've got SERP Position

Tue, 30 Jun 2009 21:26:00 -0700

The SEO Position Plus script has been updated to pull page offset from Bing.com -- thanks to Stomper Jim MacKay. Get it at the normal place and keep an eye on http://gist.github.com/138555.

Google Analytics API Roundup

Mon, 27 Apr 2009 19:07:00 -0700

Man, you gotta love the internet. Within a week of the general availability of the Google Analytics API there's an explosion of interesting new works and open source creations. While some folks may have had access before the announcement, the checkin streams on github show there's no time like the present. There are two Ruby wrappers: Gattica and Garb as well as one in Python. (Update: See this offc. Google Analytics blog post for more) I did a bit of hacking over the weekend and adapted one of the code samples to present a word cloud of search referral keywords. Bounce rate is mapped to opacity and frequency to sum of traffic from all phrases including the word. Mad props to the Juice Analytics folks for their Google API explorer tool for making it easy to get a grasp on the capabilities and restrictions. There are some hard to predict limitations on the factor and metric combinations allowed. Also notable in limitations is no support for eventing data. There's sure to be lots of interesting things to come. I'm not especially interested in tools that mimic the functionality of the web site in desktop application form, but rather in tools that go beyond the analyses that GA enables. Check out Google's gallery for a mix of the two. The desktop reporting.com folks have a basic replication widget available with the promise of fancier stuff to come. The Juice Analytics folks have a keyword research tool called Concentrate. We're pondering keyword applications as well, along with traffic source, automated diagnostics, and more advanced path analyses. What would you like to see available from your Google Analytics data? Finally, if you'd like to try the keyword cloud view on your site, here's the link for the exceptionally quick keyword cloud hack above. Works best in Safari / Chrome at the moment strangely.

Taking Open Office 3.0 Through the (Web Analyst) Paces

Fri, 09 May 2008 06:39:00 -0700

The beta of Open Office 3 is a big step for Mac users, removing the need for X11 or the aging NeoOffice port. I tried out the spreadsheet app for some basic data crunching tasks. I was pleased with the design of the charting workflow. Click through to Flickr for the 4 image sequence. On casual inspection, it seemed a bit easier to work with than the multi-path 2003 Excel or the new 07 Excel.

There are also new charting features for regression plotting and custom error bars. Check out a quick Jing screencast of running a "data pilot" (akin to pivot tables) on some recent eyetracking data. Alternatively, check out this howto blog post on the Open Office blog. While I had to read docs to figure out how to do a pivot table, check out this neat menu explorer help function! While I opened a simple .xlsx file succesfully, Infoworld tried more complex files and found support in need of more work. I did notice that OO won't save to .xslx, though it opened several simple files fine. Will OO v3 be a credible alternative to MS Office? Depends on the task -- but it looks to do a decent job an analyst's basic requirements.

Got Leopard? Login to Google Analytics compulsively?

Wed, 31 Oct 2007 23:38:00 -0700

The new version of Mac OS X, Leopard, adds a new type of Dashboard widget, created from within the Safari browser. You just pick a part of a webpage you want on your dash, and it'll be there updated whenever you look. Turns out, this seems to work just fine with Google Analytics. Just click the webclip icon in safari while you're looking at a GA report. Here's a demo... I actually create a clip on wired.com but if you look carefully you'll see several GA widgets already in place.

Get the Flash Player to see this player.

There is one caveat, the dashboard widget won't remember any clicks that don't trigger a new url -- so it will forget if that you clicked the e-commerce tab. Still, very handy.

Q&A: Can I test with only 5000 visitors a month?

Wed, 24 Oct 2007 07:28:00 -0700

Five thousand visitors a month is not too little traffic to effectively test. Here's a nice writeup of small sample sizes and simple statistical testing on conversion rates. If you site does not have transactional components (low conversion, high gain), then it may be too little traffic to measure more highly variable dependent variables like bounce rate, depth of visit, or more refined engagement metrics. As long as you have a contact, purchase, or download outcome, the techniques illustrated in the AdWords stats post will work for you. Multivariate testing actually eeks more statistical power out of less traffic than simple split testing; it's just that the conclusion of a single "test" takes longer, but you get more done in a year than doing iterative splits. I've worked in situations of immense scale -- you still have to test for a full week to deal with day of week effects. For the low traffic scenario, if you focus on key points along the line to conversion, you *can* do data driven design iteration. The Luna Metrics Blog covers this same topic, with details of the mailing list post that triggered it. For those with the opposite scenario, lots of traffic, check a video excerpt from my web 2.0 expo talk on analytics at scale.

Heuristic Evaluation: Down to Earth Lessons from MIT

Sun, 26 Aug 2007 19:01:00 -0700

One of my favorite techniques in usability is simple "heuristic review". This is a sort of the "what would do" style evaluation guided by a set of questions. Debuted by Jakob Nielsen, the idea of a set of principles to evaluate as one of the most efficient ways to conduct a expertise oriented (as opposed to observation oriented) evalution has spread with lot of variants from the original ten heuristics.

This came to mind today as I was browsing the MIT Open Courseware site -- a really neat online repository of materials from the last few years of MIT classes. Yep, you can learn from the same materials as MIT students!

To get a quick feel for heuristics, take a look at this usability checklist from web.mit. For a heavy-weight engagement with heuristic review, check out the lecture notes (lesson 14) from User Interface Design and Implementation, Fall 2004. Foundational principles are discussed in Lesson 8

Trying Out Compete.com

Mon, 28 May 2007 08:00:00 -0700

With the questionable business behavior around Alexa, and the bubbling buzz on engagement metrics, and the announcement of a new API, this morning seemed the time to give Compete.com a thorough evaluation, checking for confirmation of Alexa's recognition of the meteoric growth of Free IQ over the last month.

Compete captures the rise of Free IQ to a top 10k site:

Compete also does the basic Alexa-style competitive user reach tracking:

Compare this to Alexa's similar view.

Compete's interesting deviation from the standard practice is computing the percent of attention a site receives across all the internet focused attention in their participant pool -- essentially the "temporal share" of a site:

The Compete site isn't very upfront with their usage stats, but I did see a reference to 2M users. The ComScore panel is only 200k users, but carefully balanced across demographics. Compete, and Alexa, which rely on installation of toolbars, likely over represent the geek population. None the less, they provide real insights, and it's generally been true that the geeks lead the way for the masses... though some of the new 2.0 tagging, presence sharing, etc may tap the larger marketplace's technology adoption capacity.

On the measurement side, these new applications are challenging traditional page view metrics. A great post over at WAA paralleling this change to the history of science in physics. Certainly, the attention based focus of Compete offers a compelling alternative to pageviews for business success, but the challenges of assessing clickstreams and usability in web 2.0 apps is a slightly thornier problem. More to come on that front with an upcoming scientific publication...

Eye Tracking vs Mouse Tracking

Sun, 29 Apr 2007 06:52:00 -0700

With more and more tools available to monitor mouse movement, like RobotReplay, the question of whether mouse tracking can be substituted, or at least partially replicate, eye tracking is active. This eye tracking video of Squidoo by ETRE demonstrates both times of synch between eye and mouse and times where they diverge. If you watch carefully, you'll see that given an intent / opportunity to click in the current user activity, the mouse is much more likely to be close to the eye. This is a basic rationality that says "If I might click, I might as well keep the mouse close to my eyes." Where there's no potential to click, either because the user is in an evaluative mode or the content of interest is devoid of links, the mouse and eye diverge. The scientific literature is closing in on this interpretation with results which show that mouse position is predictive of gaze, sometimes. How this partial mapping can best be turned into value is to be determined, but take a peek at robot replay's demos as watching the user's mouse interact with the page does create a powerful interpretation of the experience. Update: Summary of research on eye - mouse synch available from: Edmonds, A., White, R., Morris, D., Drucker, S. Instrumenting the Dynamic Web. Journal of Web Engineering (JWE), Vol. 6, No. 3 (2007), 243-260.

Understanding Day of Week Effects with Google Analytics

Tue, 06 Mar 2007 05:51:24 -0700

The amount of variance in your user behavior across days of the week can easily exceed the size of differences that matter for your business. In addition, understanding your day of week effects can offer insight into when to launch promotions, etc.

In general, the beginning of the week is the point of the most dramatic activity. The weekends bring a different type of user mindset with typically a little bit more time than during the week. Note, this availability of time may work against you if the user chooses to try a different site instead of working harder on your site.

You're likely familiar with the core date widget (to the right) if you spend much time in Google Analytics. The widget circled in red is the date selector and is the bread and butter of data manipulations. The blue circled widget is the date comparer and splits a selected data range into two segments, altering the data grids to show whether the metrics of the current report are trending up or down.

Clicking the blue widget in the core in the default Date Range view brings up the view shown at left.

There's an undocumented feature of the date compare widget that faciliates day of week analyses. Simply select a Day of Week header to view only data from that day across the date range.

If Tuesday is the day you make the most money, it may be that this is the day that is most sensitive to your split testing for certain features. On the other hand, a split test that's not showing significant results may be working for you on the weekend but failing on the weekdays. Try it out, at worst, you'll learn about the strong day of week effects present on every site.

SEO Radio Segment 3: Browsers

Tue, 13 Feb 2007 10:08:00 -0700

oy-oy Firefox plugin
to seo firefox addons ... saved by 4 other people ... on feb 05
Google pagerank extension for firefox and mozilla

to firefox addons seo search ... saved by 139 other people ... on jan 25
SearchStatus | Firefox SEO Toolbar Extension | Download

to seo firefox extensions search ... on jan 25
SeoQuake - extention for Mozilla Firefox browser

Even uses LinkDomain syntax for better insights into inbound linkage.

to search firefox addons seo ... saved by 40 other people ... on oct 14
SEO for Firefox

to seo firefox search ... saved by 870 other people ... on july 05
SEOpen: The SEO Firefox Extension

to search seo firefox ... saved by 74 other people ... on may 16

Listen here...Segment 3 - Firefox browser and suggested plug-ins for SEO