AB Testing - analytics

Statistics 202: Statistical Aspects of Data Mining

Thu, 08 Oct 2009 07:42:00 -0700

Highly recommended - a Standford class delivered at Google with the lectures available on Google Video / Youtube. The course uses both the open source stats tool R as well as Excel for the heavy and lightweight ways to get things done. The slides are at stats202.com. Navigating the YouTube UI to work through these vids is painful, so I created a playlist with the videos. Click through for the larger format video: Happy data-mining!

Monitoring Branded Traffic with Google Analytics

Mon, 07 Sep 2009 22:15:00 -0700

Tracking branded traffic is a useful activity for any business. Branded traffic refers to users who seek out your business by name, either through historical familiarity or perhaps the translation of offline marketing into online activity. Thus, it can measure the effectiveness of offline advertising and the growth of a loyal custom user base, with the help of return rate metrics. In some cases, these visits show up as "direct traffic" in analytics. This means the visit came in without a referring URL in the CGI headers. Clicking a link from a site to visit another site will pass the preceding site's URL in this referral section of the HTTP headers, that part of web "clicks" that's invisible to the average user. When a user types in a URL directly, or clicks a bookmark, this field is empty. Alas, the utility of an empty referrer at indicating branded traffic has degraded over the years as firewall products begin to block the passing of the referrer due to potential privacy concerns. Another phenomena has worked to decrease the comprehensiveness of direct traffic as a measure of branded search. The pervasiveness and ease of user of internet search engine in modern browsers has led users to simplify the task of reaching a destination down to one tool: the search box. While it's theoretically more efficient to enter the URL in the url bar, many users simply use their default search engine for keywords and URL entry (est. 5% of search engine queries from Teevan, Adar, Jones, & Potts. SIGIR 2007) because the simplicity of one operation overrides the efficiency gain of a dedicate search box. The large number of domain squatters with mis-spelling and typos of popular destinations producing generally useless experiences make the search engines popularity skew to error correction on URLs an especially nice change to a cybersquatter's shop of trinkets. So I recommend creating a Google Analytics advanced segment that captures both direct traffic and inbound search referrals with query terms that uniquely identify your business. For some names, this may not be viable, but for most you can capture a large percent of the direct navigation. For instance, myweddingfavors.com gets branded searches for "my wedding favors" and "myweddingfavors.com". The my term is the key signal that the query is branded. Here's an example of the config:

Be sure to name your segment for the site you set it up on as segments are shared across sites in an account. And here's a report validating the conclusions:

iBlipper

As it should be, no referring sites are included in iblipper branded

Over a third of search engine traffic is branded.

Bing! You've got SERP Position

Tue, 30 Jun 2009 21:26:00 -0700

The SEO Position Plus script has been updated to pull page offset from Bing.com -- thanks to Stomper Jim MacKay. Get it at the normal place and keep an eye on http://gist.github.com/138555.

Using Open Source Tools to Understand Your Data: R & GGobi

Sun, 14 Jun 2009 05:12:00 -0700

I've been a long time, occasional, user of an open source alternative to high-end statistic packages like SAS & SPSS called "R". I recently spent some time learning an associated data exploration tool called "GGobi" and the integration with R (rggobi). While R is a worthy tool for data summarization, stats, plotting, and cleaning, GGobi is especially useful for exploration. It features "linked plots" in which mousing over a point in one plot highlights it in all other plot windows. I'm a big fan of the scatterplot matrix (shown to right) for understanding relationships between variables and distribution shapes. This is probably overkill for basic website stats from Google Analytics, but is very useful for more complex data. I've crafted a screencast that walks through the basics of reading a CSV and launching GGobi through R. It then goes on create a scatterplot matrix and apply a custom color scheme. The dataset in use here is data volunteered from Firefox users about their bookmarks and history data-stores. Details of the analysis are described here. Getting a handle on R is pretty much a pre-requisite for getting value from GGobi. There are lots of great resources online including Statmethods.net Quick-R but the essential reference action is to add R-Seek.org to your browser search box.

Google Analytics API Roundup

Mon, 27 Apr 2009 19:07:00 -0700

Man, you gotta love the internet. Within a week of the general availability of the Google Analytics API there's an explosion of interesting new works and open source creations. While some folks may have had access before the announcement, the checkin streams on github show there's no time like the present. There are two Ruby wrappers: Gattica and Garb as well as one in Python. (Update: See this offc. Google Analytics blog post for more) I did a bit of hacking over the weekend and adapted one of the code samples to present a word cloud of search referral keywords. Bounce rate is mapped to opacity and frequency to sum of traffic from all phrases including the word. Mad props to the Juice Analytics folks for their Google API explorer tool for making it easy to get a grasp on the capabilities and restrictions. There are some hard to predict limitations on the factor and metric combinations allowed. Also notable in limitations is no support for eventing data. There's sure to be lots of interesting things to come. I'm not especially interested in tools that mimic the functionality of the web site in desktop application form, but rather in tools that go beyond the analyses that GA enables. Check out Google's gallery for a mix of the two. The desktop reporting.com folks have a basic replication widget available with the promise of fancier stuff to come. The Juice Analytics folks have a keyword research tool called Concentrate. We're pondering keyword applications as well, along with traffic source, automated diagnostics, and more advanced path analyses. What would you like to see available from your Google Analytics data? Finally, if you'd like to try the keyword cloud view on your site, here's the link for the exceptionally quick keyword cloud hack above. Works best in Safari / Chrome at the moment strangely.

SEO Position Plus: Log Your Exact Google Rank with Google Analytics

Wed, 22 Apr 2009 10:27:00 -0700

We've been working hard on SEO Position over the last few months and the latest version has been in use by Stomper members for a while now. With the announcement of updates to the Google referral string it's now important to update the script and there's some serious new features. For those unfamiliar with the SEO Position script, in the picture above/right, the query "analytics motion charts" generated a click from Page 1 of the Google results while "motion charts" generated a click from Page 2.
SEO Position Plus tracks more types of referrals than the original script. All the categories logged to Google Analytics events are prefixed with SEO.

(Google | Yahoo | AOL | Live | MSN) Page: tracks page number of referrals from different engines
Google Site: Distinguishes subdomains & verticals like news, reader, corp, etc.
Google TLD: Find you average ranking by page number by Country
Google Images: Pull out keywords, traffic flow, etc. from Google image referrals

Mark at MivaMerchant wrote up a great tutorial (Stomper Members: See my video in the portal). So that's all cool and useful, but the last item "SEO Google Position" captures the exact rank of the result clicked on Google to generate this visit. The announcement says the referral change is being rolled out and we're currently only seeing it for 1/10 to 1/40 of traffic. I expect the data volume to increase and for this to provide much better data on ranking, ranking changes, and the ROI on ranking changes. Here's what you'll get when a referral comes through with the new cd parameter:

The script is available at /abtest/includes/seopositionplus.js and is released as open source under the Mozilla Public License. Use it for free for whatever you like, but if you make it better, you have to share!

Place the script following your Google Analytics code/ Copy the SEOpositionPlus file to your server and add the following line beneath your call to pageTracker:

SEO Position & UI Links Google Analytics Scripts Updated

Wed, 28 Jan 2009 08:41:00 -0700

The SEO Position script now supports Yahoo, MSN & Live, in addition to Google. The name of the event has changed from "Google SEO" to Google. Thanks to Jim M. from Bunk Beds Now for the help expanding the script. Alas, it was not originally apparent that logging a source event will flip the bit on "user bounce" and deflate your bouce rate for a page. Rapid detection of near-page-1 rankings may be worth the trade-off. I've also updated the UI Region Logger to use eventing instead of synthetic page views. Just flip the ui_useEventing boolean to true in the script source. Read more in the orginal post, but to recap, this script requires that you tag key areas of your interface with a UI attribute. For example, you'd add ui="sidebar" to the div that contains your sidebar. With every click, the script walks up the DOM and checks if there's a UI label above it. I'm looking forward to providing a tool to do a heatmap style visualization of click regions once I build up a good data set with this one. The data is much easier to isolate than in the prior exit link mode. The image shows the first data from this site with one click on the tag cloud in the right sidebar and 2 clicks in the tools menu.

Using the new Google Eventing for SEO Reporting

Fri, 02 Jan 2009 15:01:00 -0700

I'm a huge fan of metrics about activities within a page like scroll depth or form abandonment analyses. Google Analytic's new eventing facilities (almost out of beta it seems!) enable this kinds of data logging and reporting.

Intro to GA Eventing

You get 4 slots of data to log to:

Action
Category
Label
Value

The official eventing docs describe an example for a Video action with categories for play/pause/stop and labels of the video name.

Ranking Insights

I've developed a script that enables ranking information to be logged to GA events. I pull the page offset (e.g. the start parameter) from the Google search referral string.

The reporting options are somewhat limited for events -- you can't pivot any report outside of the Event section by this data as of yet. It took a couple iterations to get the logging design to a useful point.

Rankings by Page

Using the Label report, we can see event data by Page with average rank (labeled "average value" in the data grid). The usage tab on this report will let you assess engagement by bounce rate and page views.

If you're logging multiple categories of events, you'll want to drill down to labels through the Google SEO category.

Rankings by Keyword

Using the drop down pivot to select keyword will get your average ranking per keyword, across all page.

Uh, Google...

Why did I have to write code to get this data? Hard to say, webmaster tools gives you some data on ranking position. This, along with accurate reporting of image search referrals should be built into google analytics.

A Word from Our Sponsor

This script is brought to you, in full open source fashion, by StomperNet LLC. The latest offering from StomperNet is Formula 5 -- a training program designed to help you amplify your business success. The program works with multiplicative effects, beyond those demonstrated in my conversion funnel modeler to encompass your entire business. Check it out now.

Get the Script

The script is a dozen or so lines of designed to be placed after your call to GA's pageTracker function. You do have to be running the new ga.js scripts, not the legacy urchin.js.

Copy the SEOposition file to your server and add the following line beneath your call to pageTracker:

The script is seoPosition.js and is released under the Mozilla Public License (MPL). The MPL is a friendly open source license allowing any type of use but requiring that enhancemnts to the existing file be contributed back to open source. This work was inspired by a filter hack from andrescholten.nl -- I didn't want to go through that trouble on every site. Thanks to DaveL @ E-Tail.be for calling my attention to this.

Hands on with Google Analytics Motion Charts

Thu, 06 Nov 2008 22:37:00 -0700

I was very excited earlier this week to discover I had access to the new Google Analytics beta features. Custom reports are certainly a useful tool; they allow you to construct both large scale exploratory views as well as concise views in which the viewer doesn't have to ask "which metrics should I look at?" The big payoff is in the new Motion Chart visualizations which attempts to capture 5 dimensions through the mapping of attributes to x, y, size, color animated over time. While you could check out the official videos, here's a look at my recent foray into creating an iPhone (web) application. Pictured are referrals from the Apple web application directory, where iBlipper landed Sept. 10th. On the x-axis are unique searches, or phrases typed into the iBlipper application. The y-axis is a correlated metric, time on site, and size is mapped to % new users. We can watch as page 1-5 deliver less and less traffic as the app drops off the category independent list and falls down the entertainment app list in the default recency ordering. Pay close attention to the axis values -- there are some subtle interpretations available from mixing engagement, volume, and loyalty (% new visits in this case). For instance, iBlipper briefly landed on the top 10 entertainment apps list, url of /webapps/entertainment/index_top.html. These users seemed to spend more time on the site w/o entering their own search phrases, suggesting a less directed choice in visiting iBlipper and more passive usage of the application. This is shown by the green dot highlighted to the right higher in time on site than average for the unique searches compared to most other referral paths.

I'll leave you with some power user tricks for using motion charts:

Filters applied in a report view control the data shown in the visualization. In the video case, I've filtered by referrals including '/webapp', a unique signature for the Apple directory
There's a subtle option on the x & y axis to code by lin(ear) or log(arithmic). Adjusting both axes to logarithmic can greatly inform on the underlying mechanisms.

Tracking UI Level Links: An Open Source Script

Tue, 22 Apr 2008 21:51:00 -0700

One of the challenges with the current complex site designs is that multiple links to the same destination tend to appear on the same page.

This does not allow you to understand how your allocation of real estate is being used without resorting to really fancy analytics packages.

To solve this problem I developed a script that upon every click, walks up the document object model (DOM) looking for a an attribute on an HTML tag of ui.

If it finds one before it hits the BODY tag, then it adds a parameter to the link called ui with the value of that attribute, allowing you to understand which links on a page are being used. Shown below is a report from Google Analytics for this site showing how people arrive at my bio page:

If you visit my bio from the About menu at the top of my blog, it adds ui=nav.

I've licensed the script under the MPL and you're welcome to use it on your site, providing you share any enhancements. Grab it at http://alwaysbetesting.com/abtest/includes/logger.js.

All you have to do is include the script on your page and add ui=element_name attributes to key HTML elements that make up your site structure.

Because this happens with Javascript, and only when the user clicks, there's no danger of creating duplicate content for the search engine spiders. Alas, that means it doesn't fix typical analytics overlay views either for duplicate links, but you can typically see the source element in your path reports.

Google Analytic Gems #2: Quantifying Deliberative Conversions

Tue, 08 Apr 2008 06:42:00 -0700

Another little known gem in Google Analytics is the Time to Purchase and Visits to Purchase.

For MyWeddingFavors, where the purchase is an exceptionally meaningful one for our customers, some 40% of our sales happen on subsequent visits:
. Careful though! Looking into Days to Purchase, we see only 25% of sales happen on a different day than the introduction to the customer.

So many (15%) of our shopping experiences are stretched out over a day, while only 25% happen on a subsequent day visit. Understanding this pattern has some serious implications for design, business strategy, and e-commerce feature set.

Google Analytic Gems #1: Split Test Evaluation with Only New Users

Thu, 03 Apr 2008 09:26:00 -0700

A question on LinkedIn, now closed, asked "what are some hard to find but useful reports in GA?" While clicks to task completion is far from the ultimate metric, Google Analytics does suffer from some slightly onerous depth issues for specific data points. We do a lot of split testing with Google Analytics by defining custom segments and serving each segment a different UI. One issue with understanding the impact of new features, particularly for sites with lots of repeat visitors (e.g. content / blog sites vs ecommerce), is the novelty effect. New features, or even simple changes in layout, can have a short term halo as users notice and engage with the changed content. Google Analytics does allow you to look at your user defined segments for new and repeat visitors, but it does require a few clicks. Follow along with the picture:
Starting in the visitors submenu (1), the New & Returning report allows you to drill into New users (2). The segment drop down has lots of useful pivots, including "user defined" (3). Picture #4 shows the results for new users of a split test that moved an mailing list subscribe box from left to right. The magnitude of the effect diminished over time as we tested this. However, by drilling into only new users, we see the original effect size. Looking at all users, the effect is smaller, and looking at returning users, the effect is smaller still. Dealing with the halo effect is one of the reservations that was expressed during "AB Testing: Designer Friend or Foe" at SXSW. Splitting users into new and returning is one of the easiest strategies for seeing through this confound.

Getting Serious About Testing: Learn from the Pros

Sun, 16 Mar 2008 10:24:00 -0700

Last week's SXSW panel on AB Testing: Designer Friend or Foe left me wishing for a more robust treatment of the experimental design issues around online testing. It was a great panel, and I appreciated the real world experience of the panelists, but aside from Micah, the approach was very much from a design world. This is fine, but issues came up that stats exist to solve, and the distinction between multivariate and AB testing was glossed over. In particular, designed well, multivariate testing can be used to test hypotheses about user models, not just a way to play roulette with font colors and sizes. There is a robust body of knowledge that lives between statistics, traditional experimental psychology, cognitive modeling, and resting on the shoulders of giants in practical business success through experimentation. The Exp Platform, led by Ronny Kohavi, at MSFT publishes from this position of strength. Their latest, 7 pitfalls to controlled experiments on the web, is a solid read for those aspiring to live in this space. AB testing might indeed be a foe to the designer when done without appropriate expert support -- at least for more aggressive evaluations. Here's a recap of the Seven Pitfalls:

Avoiding experiments because computing the success metric is hard.
Attempting to run experiments without the pre-requisites: representative & sufficient traffic, appropriate instrumentation, agreed upon metrics.
Hubris: Over-optimization without collecting data along the way.
Bad math: inappropriately deployed confidence intervals, % change, and interaction effects.
Use of composite metrics when power is insufficent. An example, not in the paper, is the use of checkout completion for a product page change, when add-to-cart % would be more sensitive.
Un-even sampling: bad balance between control and test distributions.
Lack of robot detection.

I've blogged the guide to practical web experiments and it's also highly recommended. It provides an overview of the key issues to deal with in setting things up including sampling, failure versus success evaluation, and common pitfalls like day of the week effects. More from the historical '05 SXSW Design perspective with How to Inform Design: How to Set Your Pants on Fire March 14th, 2005 presented by Nick Finck, Kit Seeborg, and Jeffrey Veen

Design Metrics Wrapup

Mon, 10 Mar 2008 12:51:00 -0700

What fun! The SXSW conversation format is quite cool, though it really needs a dedicated space as our group size was limited by how far voices carried. Drop a line in the comments if we promised to follow up on something I haven't posted. This is a work in progress, so check back if there are some empty items when you visit.

References during the chat

Common Industry Format for usability testing, from NIST, now an ISO standard.
Net Promoter - Would you recommend this product to others?
Subjective Duration Assessment: A nice metric sensitive to both good and bad experience.
Scrutinizer Browser: Simulation of vision in a web browser
TechSmith, maker of tools for usability testing, is in the tradeshow here at SXSW.

Blogs

Google Analytics Blogs
Functioning Form from LukeW.

Resource Lists

Analytics

Web Analytics Yahoo Group is a solid mailing list, moderate volume, with an awesome depth of industry practitioners.
A composite engagement metric from Eric Peterson
my Google Analytics resouces page
Exp-Platform Publications: From the testing team at MSFT.

Usability Training

User Interface Engineering does great stuff.
Nielsen Norman Usability Week conferences are great.

Books

... Thanks to everyone who participated, and to Micah for bringing me along.

SXSW: Driving Design From User Data

Sun, 09 Mar 2008 14:45:00 -0700

I wrote about the crucial conversation at SXSW with Micah Alpern a few weeks ago. The time has come! In talking through this with Micah, we came back to the crucial insight that the availability of artifacts of the usage of internet software creates an opportunity and challenge for designers. What follows is a reference for our conversation, which will include a short intro and mostly conversation. Subject to conversational flow, we'll be asking the participants to share stories:

What's your favorite HIPPO story? For those of you who haven't encountered the hippo meme, it's about decision making based upon something more than the highest paid individual's personal opinion.
What business or user goal would you like to be informed by metrics?

The talk precis:

Design Metrics: Better Than 'Because I Said So': Too often designers are put in a position of defending design decisions based on personal preference or an unarticulated sense of expertise. We'll discuss how to use metrics to understand user and business goals. Then how these metrics can be used to evaluate design decisions, make tradeoffs, and shape strategies.

Our goal is to better enable productive conversations with key stakeholders, using the tools of metrics to understand and advocate a position. In the most productive cases, this means designing with measurement toward end goals in mind. In less developed scenarios, there may be some foundations in need of construction. There are a lot of reasons to test designs with live users. The most pedestrian is business acceptance testing. We'll be more focused on using metrics to resolve internal debate, multivariate testing learn more about the motivations, mental models, and personas of users, as well as value estimation. We believe the "Role of Designer " is to drive hypotheses about the user and to internalize results and use to inform future design. Of course, testing is not the only tool in the toolbox. You have to choose the right tool for the job. Key dimensions:

Quantitative, Qualitative
Small vs. large scale
Advanced techniques: Sequence modeling, learnability metrics
Repeat vs. non-repeat visitor

That said, creative techniques with Greasemonkey or limited scale prototypes can make testing available in situations you might not think it's possible. We're scheduled for 11:30 AM in Ballroom E on Monday. Hope to see you there. If you can't make it, stay tuned for a follow-up.