Google Web Site Optimizer Flaw Found

If you have used Google's Web Site Optimizer you know what a useful tool it can be. If you have used it a number of times you may found some limitations with testing what you want, in the way you want to test it.

Well, I thought I found a flaw in how the Optimizer works, and Google has just confirmed that I am correct in my thinking. It's a little confusing to explain, so I will really try to make it clear what the problem is with a simple example:

Goal: To find which of two home page versions convert better.

Method: Create two variations of the home page and see what visitors that hit the home page fill out a form and then see a "thank you" page. Conversion tracking code is located on the "thank you" page and there are no direct links to this conversion page on the site.

Test: The test runs and after some period of time enough data is collected and we may or may not have a "winner" depending on how the conversions went.

Traffic sources: Organic traffic and some paid search traffic to the home page and other topical pages.


So, this sounds like just about every test you have run, right? Therein lies the flaw. Did you spot what the problem is?

I didn't either until I ran a test over the past few weeks, and the results for 9 variations were not only unclear, but also kept changing. This is also something I have seen before, but this time it got me thinking about the process and here is what I realized:

Because there is NO CONTROL over where the traffic lands or it's potential path to the conversion page, one cannot be sure that the conversion ( or lack of conversion ) had anything to do with the page being tested.

Visitors may enter the site at the home page, or they may enter someplace else. If they enter someplace else, then see the home page, they will be included in the test data. And if they do hit the home page, then visit another page which causes them to convert, it will be counted toward the home page version, even if it was really another page that resulted in them converting.

So, the answer to this is to set up your experiment so that you control what people are seeing and what options they have for converting. What I plan on doing in the future is setting up the test on a special page of the site that only gets PPC traffic so I know where it is coming from, and also a special conversion page as well. Visitors may really like our test page, and it may be behind a conversion on a later page, but there is no way to know that so the data really is not that useful and should be excluded.

To restate my solution, I will be setting up landing, contact, and conversion pages. If the conversion results from the test page, I will capture that info. If the visitor leaves the landing page without converting, but converts later, it will not be counted in the test. This will result is some test conversions being lost perhaps, but the data will not be inflated when it should not be. Those of you with one-page sites have been and will still be able to run perfectly accurate tests. :-)

PPC Conversion Fraud Documented

Almost anyone that does search marketing and pay-per-click advertising or PPC is aware of the concept of click fraud. They may think it's a small problem or a huge one, but we all know it exists. While advertisers may blow it out of proportion due to poor conversions and sales, the truth is that a poorly run campaign could be partly or completely to blame.

On the other hand we have the search engines that tell advertisers almost nothing about the problem other than that they have "advanced detection methods" in place. Some even provide large refunds from time to time but in most cases refunds are small and seem inappropriate to the lack of ad campaign results. But it can be hard to impossible to tell where the true numbers are.

Most advertisers rely on conversion numbers to tell them if a campaign is working or not, or if it needs to be adjusted. Lack of conversions can also indicate click fraud. Higher conversion number means it is working and should be generating revenue that justifies continued spending.

This post is to inform advertisers, and search engines that care to listen, about what seems to be a growing trend for those committing fraud to cover their tracks: Conversion Fraud.

Here is an example from a contact form:
===========================================
 Remote Address: 98.172.1.169

  Form Sent From: http ://googleads.g.doubleclick.net/aclk
?sa=l&ai=B4IPZS8jy13AP_7qCtG64YiHYAADM_f___8Yn
uboAo7ACZbbAEdLdYYgq6OAUQABUEAA&num=2
&adurl=http ://www.thisistheurltomyclientswebsite.org/%3F
type%3Dcontent%26keyword%3Dmachines%26adid
%3D2164885665%26placement%3Dbugabo.cn&client=
ca-afdo-pub-1913393681262590

HTTP User Agent: Mozilla/4.0 (compatible; MSIE 6.0; SV1)
===========================================

Here is what the info is and where it comes from:

- Remote Address:
The IP address of the user

-  Form Sent From:
This URL is captured in the form script when executed. It should ALWAYS be the page the form was submitted from. I have edited the first part of the URL so it is not valid, but the placement and publisher info is correct.

- HTTP User Agent:
Browser type

==========================================================
The only possible conclusion one can reach when looking at the evidence is that
someone connected to the parked domain owner has taken the click URLs from
some or all of the ads, and loaded them into a script that submits our form with
the listed data, using the parking page ad URL as the referrer.

This is click fraud that shows as conversions to the advertiser.
==========================================================

Here is the PHP code that is used for this section of the form:
------------------------------------------------------------------------
 Remote Address: $_SERVER[REMOTE_ADDR]
  Form Sent From: $_SERVER[HTTP_REFERER]
HTTP User Agent: $_SERVER[HTTP_USER_AGENT]
---------------------------------------------------------------------
Adding this code, or code like it for other scripting languages, can provide advertisers with more information about who is clicking on their ads and why.

And you notice that we are capturing the IP address of the person submitting the form.While this can indicate fraud as well, for months now we have seen fraudulent clicks and conversions from IPs on Comcast and other ISPs. While I cannot provide proof, my feeling is that most of these are from infected or "botnet" computers, designed to steal small amounts of money on a vast scale. If I was in the fraud business, that's what I would do!

I suggest that anyone concerned about click fraud also be aware of conversion fraud. In this case, it is done with a program and is easy to spot. If cheap labor is used and your forms filled out by a person, you will have no way to know what is going on except from a lack of sales and perhaps invalid from information. At this time there is no way to update your PPC statistics and change the number of recorded conversions to be more accurate.

My proposed solution to end click fraud is a simple one, but one that is still dismissed by those that hear it: Flat rate advertising. Google actually has something like this with their site targeted option, but since you pay for impressions rather than clicks, you can have what I call "Impression fraud".

Only by having ads displayed for a set period of time at a set price can we ever hope to be free of these kinds of fraud.