Actionable Analytics

Posts Tagged ‘Tools’

Recently I read Michael Notte’s excellent post on web analytics and the automotive industry in Europe, and then ended up getting drawn into a conversation about the tool Sophus3, which I used for a long time when I was working with a key automotive manufacturer. I ended up writing a bit of a review for it, so thought I would expand this out a bit on my own blog. I had a lot of problems with this tool and never really felt able to voice my issues about it due to client politics, but now I no longer work with this client I feel it is time my opinions were voiced. Hopefully this will assist with other client’s decision making processes.

Firstly, a disclaimer: these are my opinions alone and have nothing to do with any employer or client I have ever worked with – nor can I say that these are necessarily universal truths that others would have had as well; it’s simply a statement of my experiences working with the tool and supplier. Similarly, the client in question has an excellent relationship with Sophus3 and does get value from the tool; my beef is simply that they could do a hell of lot better and don’t realise it.

Furthermore, it really isn’t all bad – so just to avoid coming across too negative I will start with the positives:

Pros:

• They DO understand the automotive industry better than any other supplier. No other vendor targets vertical market segements in this way. It means that they have a lot of good insights on how the measurement framework for the sector should work and how reporting should be built.
• Their customer service is very good and their staff are very dedicated to client support (at least for their end clients, not so much if you’re an agency though).
• Their back end analytics interface, whilst extremely slow, is actually very flexible and feels more like querying a proper database than an OLAP set-up. Complex cross-tabs and tables can be built in a way which isn’t really possible in other tools.

Cons:

• Speed. The tool is very very slow to use in comparison to other tools, so much so that most of the time you just won’t bother. At times it is more like querying a huge SQL database than a web analytics tool.
• Accuracy of the competitive tool. The eData Exchange tool is supposed to provide benchmark data of all automotive suppliers. However, only key pages of the sites are tagged and non-standard stuff like microsites are ignored. This in my opinion makes the data too inaccurate to use. Some manufacturers rely heavily on campaign microsites and the customer never actually hits the main website. Other manufacturers do everything in the main site.
• Tagging. This does not work like any other web analytics tool. Tags are simply bits of code that are placed on the site – then Sophus3 themselves have to sort out all the meta-data and naming conventions at their end. This is a nightmare and removes vital control over how the tool is set up. It also creates a enormous possibilty for error that just doesn’t happen in other tools.
• ALL configuration has to be done by Sophus3. They have to set up all the dashboards, custom metrics etc etc. This wouldn’t be so bad, but personally I never really felt that this was done right and therefore wasn’t easy to use. No vendor, despite what they say, really has a proper handle on measurement strategy. This is something that needs to be handled by either the client or the agency; and they need the hands-on flexibilty to make the tool bend to this strategy.
• No proper page path analysis; no site overlay; no on-the-fly segmentation; and various other missing fundamental bits of functionality.

In summary, at this point in time I can honestly say that I would not recommend Sophus3 to any company, not even automotive manufacturers. The tool isn’t completely useless, but the point is that even free tools like GA are leaps and bounds ahead, not to mention the giants like Omniture and Webtrends. Sophus3 have a good organisational foundation, they just need to seriously update their tool to bring it in line with other players.

The issue of data quality and accuracy in web analytics is something that most web analysts have no option but to learn and internalise very quickly, especially when people start asking why numbers don’t match. However, it is often easy for us to forget that our clients, business users and marketing teams don’t live and breath this data as we do. This post is therefore a reminder of the essential (by no means definitive) facts about why web analytics data can’t necessarily be taken as fact.

Why are the numbers different?

Most people first recognise a problem with web analytics data because they are trying to reconcile absolute numbers between two different systems, for example when comparing visits in Google Analytics with clicks as reported by Atlas (or some other ad tracking tool). The following are the key reasons why these numbers don’t match:

• The terminology used to calculate metrics usually differs slightly. For example, unique visitors must always be unique visitors within [a certain time frame]. Different vendors may use different time frames. Neither is right or wrong; they are just different. This same principle can also apply to lots of other metrics, and sometimes on a much more subtle level.
• Whilst advances are constantly being made, there are currently no agreed standards to these definitions. Analytics vendors often try to name-drop ABCe standards (at least in the UK), but these are generally considered to be outdated and were created for reporting on visits that derive from banner advertising and search; not for web analysis. Here is a good synopsis of the current state of standards.
• Tracking methodologies, such as cookies, packet sniffers and IP addresses all collect data in different ways and all have pros and cons to the way in which they do this. See example below for further info on this one.
• The Internet is composed of a huge array of different technologies, which are all constantly evolving and changing. These technologies play a big part in the accuracy of data collection.
• New browser versions invariably feature new types of technology that allow increasingly savvy web users to hide their on-line behaviour, or even block this behaviour by default.
• Robots and spiders crawl Internet pages in order to e.g. index what is in them for search engines. Data quality in web analytics is a race to keep up with these creatures!

Cookies and Unique Visitors – An Example

The issue of cookies is generally the biggest area of confusion. A client of mine was recently comparing Google Analytics to their incumbent provider, Sophus3. They noticed large differences in unique visitors and wanted to understand why. Whilst this issue is in some respect the product of all the points raised above, the main cause is the type of cookie used:

With Google Analytics, visitors are tracked using 1st party cookies. Estimates suggest that around 1% of users block these cookies and a further 4% block JavaScript. GA is therefore unable to track these users, so real visitors may be under-counted by about 5%.

Sophus3, on the other hand, uses 3rd party cookies. Many browsers block these by default, so estimates suggest that around 65% of traffic is lost due to the combination of this and JavaScript blocking.

Sophus3 then use IP address to track visitors who have blocked cookies. However, most broadband providers use dynamic IP addresses, which change periodically. In some cases, the IP address could change every time the person switches on their computer. Therefore, Sophus3 will register individual people as multiple visitors, and overall numbers will therefore be inflated.

The following chart illustrates this issue in a more visual way (numbers are rough estimates to illustrate a point, and are not meant to be accurate):

How cookies can affect data accuracy in web analytics

Whilst 1st party cookies are generally considered in the industry to be best practice, in truth neither is perfect. For more information, here is a more detailed overview of how cookies affect web analytics data.

Get over it!

The issue of data accuracy can cripple companies and cause vast amounts of wasted time. In truth there is no solution, it is much better to:

• Understand the limitations in as much detail as possible and ensure that all recipients of web reporting and analysis are familiar with what the numbers do and don’t tell them.
• Focus on trends and segments, and not on absolute numbers. This is easy to do when the focus is on analysis and not pure reporting; insight never comes from pure numbers.
• Where numbers such as unique visitors are required for decision making, confidence levels should be used to make reasonable judgements about those numbers.
• If we set a consistent base-line of data at the most accurate that we can get it, then we can use this data to make accurate trend assumptions and draw conclusions about time-series analyses.

Analytics direct traffic reports are often viewed as both a highly insightful metric and, in itself, as a particularly valuable stream of visitors. These are people that typed your URL directly into their browser, right? They must have seen your TV ad or just been really engaged with your brand because they remembered your address and didn’t need to use search. Who could ask for better visitors? They are motivated and focused and really intended to come here.

This kind of language continues to dominate all kinds of discussions about web analytics, including blogs, forums, and articles – and even reaches into the field of the experts; just look at the way Google Analytics defines direct traffic. It’s even more worrying when I hear the way my clients talk about it!

The fact is, this definition of direct traffic in web analysis is extremely misleading. It’s true that the direct traffic bucket does include bookmark traffic and typed URLs, but these days (unless you are very strict about your campaign tracking parameters) it can and does include all kinds of other stuff. All it really means is that the session started without a referrer being passed by the user’s browser, and this can happen for lots of reasons as defined in this rather neat list. I have done some tests on some of my clients’ sites and estimate that in some cases up to 90% of ‘direct’ traffic is infact banner ad or PPC traffic!

Here’s an exercise you can perform that will demonstrate exactly how prolific this problem is: as you’re browsing the Internet and following links from one site to the next, you can check the referrer that is passed by typing the following snippet of code into the address bar of your browser:

javascript:alert(document.referrer)

For example, if you visit a site like AOL and click on one of the advertising banners, when you arrive at the destination page replace the URL with the code – a pop-up will appear showing you the referrer (or nothing if one wasn’t passed). Try this with different types of sites, banners and links. Also try it with different browsers. As you will see, quite a lot of the time the referrer is blank. This means that your visit would have been counted as direct traffic in the analytics reports of that site!

So, it’s time stop thinking of direct traffic as people typing in your URL, this isn’t necessarily the case. ‘Other’ or ‘unknown’ would be a more accurate description.

It’s also time to realise the importance of campaign tracking on your inbound links, as Avinash Kaushik points out in his definition of analytics direct traffic. If you always ensure that your links are passing source and campaign info, then you are forcing the referrer field to be populated even if the browser doesn’t pass it. Here’s an easy way to build campaign tracking URLs in Google Analytics.

With recent and continued advances to Google’s excellent and free analytics tool, one of the key questions that I seem to get asked these days is whether there is any real value in paying companies like Omniture and Webtrends for the commercial (and expensive!) services they provide. 

It’s probably already obvious that I’m a fan of Google Analytics (be prepared for gratuitous bias); for lots of clients I really don’t see how spending the money on something like Omniture would benefit them. However, this isn’t always the case, and I think a more systematic way of making this decision is often called for. 

This post is therefore an attempt to help make decisions about whether or not you should put your hand in your pocket, and I have chosen Omniture Site Catalyst as an example. 

Monetizing the Incremental Value of Site Catalyst

Now, it is undeniable that a tool like Site Catalyst does some more stuff than Google Analytics, and certainly that it has more dedicated and human support. However, it is very easy for clients to get blinded by the way sales people position these extra features; they don’t stop to think what they might actually use them for. Conversely, GA extremists will flatly deny that there is any use in these additional features (or sometimes that they even exist), likewise failing to provide adequate reasoning. 

It seems to me that there is a more simple way of stating the true question: 

Site Catalyst does various things that Google Analytics doesn’t. What benefit do these things provide on their own (i.e. in isolation from any of the things that both GA and SC can do)? And – can the entire cost of Site Catalyst therefore be justified based on these incremental benefits?

So what does Site Catalyst do that Google Analytics doesn’t?

Following is a list of the key things that I believe SC does that GA doesn’t. It isn’t meant to be completely definitive, but [in my honest opinion] everything else is pretty much cosmetic:

Weighing up the cost benefits

Real-time data – this basically means your stats update more-or-less straight away rather than after about 24 hours or at mid-night. Personally I find it hard to think of companies that could truly benefit from this, but if you think you might then you need to work out exactly what financial benefit it gives you over and above waiting half a day. Also check out Avinash Kaushik’s blog on real-time data.

Importing external data – at first glance, this is a fairly major thing that GA doesn’t do. In Omniture you could import a lookup table of postal codes and then use this to carve up sessions into sales territories. This can be pretty valuable, but what you really need to ask yourself is: ‘how much benefit does this give us over and above exporting the data to excel and making the table ourselves?’ How much extra work is it really to just do this outside the tool? This also applies to a lot of other stuff, such as the functionality that lets you add targets to KPIs – and also to most of the Genesis integrations.

Custom variables – you actually get 2 of these in GA, but then you get loads in Omniture. Yes, for some companies this is valuable, but are you one of them? Again, I’m not denying that these things are important; I’m saying that you need to make an actual financial calculation about the benefit you get from using them over not using them. ‘Nice to have’, ‘convenient’ and ‘handy’ are not good enough reasons! Another function with similar ramifications is the ability to link metrics with dimensions that are not available in the out-of-the-box package.

Creating paths and funnels on the fly – very nice, and I wish GA did this, but I would have a hard time selling it to a client and I also couldn’t say that it is critical. I’ve certainly never seen it as a barrier in GA. Monetize it if you need it!

And, seriously, that’s pretty much it! Like I said, everything else is cosmetic or falls into a similar category. The main point is that you don’t get swayed by the sales spiel, and you calculate the return on your investment not by asking what analytics per-se can do for you, but asking instead exactly what can commercial analytics do for you that the free stuff can’t?

But wait…

Having said all that, a big word of caution  – GA can do a lot of stuff that Site Catalyst can, but a lot of the time it isn’t necessarily easy or straight forward to do, so much so that you might not even know or believe that it is possible in GA. What I’m getting at here is, you may need specialist expertise (a decent analyst) to be able to match GA with Site Catalyst on some levels of functionality. Again though, monetize this properly – you would have to pay someone to use Site Catalyst, so how much more would you have to pay someone to get the most from GA and how does this weigh up against the cost of SC?

Finally, it is worth also noting that I haven’t even touched on what GA can do that SC (on its own, i.e. without Discover etc) can’t, and believe me there is plenty of stuff!