Actionable Analytics

Posts Tagged ‘accuracy’

Recently I read Michael Notte’s excellent post on web analytics and the automotive industry in Europe, and then ended up getting drawn into a conversation about the tool Sophus3, which I used for a long time when I was working with a key automotive manufacturer. I ended up writing a bit of a review for it, so thought I would expand this out a bit on my own blog. I had a lot of problems with this tool and never really felt able to voice my issues about it due to client politics, but now I no longer work with this client I feel it is time my opinions were voiced. Hopefully this will assist with other client’s decision making processes.

Firstly, a disclaimer: these are my opinions alone and have nothing to do with any employer or client I have ever worked with – nor can I say that these are necessarily universal truths that others would have had as well; it’s simply a statement of my experiences working with the tool and supplier. Similarly, the client in question has an excellent relationship with Sophus3 and does get value from the tool; my beef is simply that they could do a hell of lot better and don’t realise it.

Furthermore, it really isn’t all bad – so just to avoid coming across too negative I will start with the positives:

Pros:

• They DO understand the automotive industry better than any other supplier. No other vendor targets vertical market segements in this way. It means that they have a lot of good insights on how the measurement framework for the sector should work and how reporting should be built.
• Their customer service is very good and their staff are very dedicated to client support (at least for their end clients, not so much if you’re an agency though).
• Their back end analytics interface, whilst extremely slow, is actually very flexible and feels more like querying a proper database than an OLAP set-up. Complex cross-tabs and tables can be built in a way which isn’t really possible in other tools.

Cons:

• Speed. The tool is very very slow to use in comparison to other tools, so much so that most of the time you just won’t bother. At times it is more like querying a huge SQL database than a web analytics tool.
• Accuracy of the competitive tool. The eData Exchange tool is supposed to provide benchmark data of all automotive suppliers. However, only key pages of the sites are tagged and non-standard stuff like microsites are ignored. This in my opinion makes the data too inaccurate to use. Some manufacturers rely heavily on campaign microsites and the customer never actually hits the main website. Other manufacturers do everything in the main site.
• Tagging. This does not work like any other web analytics tool. Tags are simply bits of code that are placed on the site – then Sophus3 themselves have to sort out all the meta-data and naming conventions at their end. This is a nightmare and removes vital control over how the tool is set up. It also creates a enormous possibilty for error that just doesn’t happen in other tools.
• ALL configuration has to be done by Sophus3. They have to set up all the dashboards, custom metrics etc etc. This wouldn’t be so bad, but personally I never really felt that this was done right and therefore wasn’t easy to use. No vendor, despite what they say, really has a proper handle on measurement strategy. This is something that needs to be handled by either the client or the agency; and they need the hands-on flexibilty to make the tool bend to this strategy.
• No proper page path analysis; no site overlay; no on-the-fly segmentation; and various other missing fundamental bits of functionality.

In summary, at this point in time I can honestly say that I would not recommend Sophus3 to any company, not even automotive manufacturers. The tool isn’t completely useless, but the point is that even free tools like GA are leaps and bounds ahead, not to mention the giants like Omniture and Webtrends. Sophus3 have a good organisational foundation, they just need to seriously update their tool to bring it in line with other players.

The issue of data quality and accuracy in web analytics is something that most web analysts have no option but to learn and internalise very quickly, especially when people start asking why numbers don’t match. However, it is often easy for us to forget that our clients, business users and marketing teams don’t live and breath this data as we do. This post is therefore a reminder of the essential (by no means definitive) facts about why web analytics data can’t necessarily be taken as fact.

Why are the numbers different?

Most people first recognise a problem with web analytics data because they are trying to reconcile absolute numbers between two different systems, for example when comparing visits in Google Analytics with clicks as reported by Atlas (or some other ad tracking tool). The following are the key reasons why these numbers don’t match:

• The terminology used to calculate metrics usually differs slightly. For example, unique visitors must always be unique visitors within [a certain time frame]. Different vendors may use different time frames. Neither is right or wrong; they are just different. This same principle can also apply to lots of other metrics, and sometimes on a much more subtle level.
• Whilst advances are constantly being made, there are currently no agreed standards to these definitions. Analytics vendors often try to name-drop ABCe standards (at least in the UK), but these are generally considered to be outdated and were created for reporting on visits that derive from banner advertising and search; not for web analysis. Here is a good synopsis of the current state of standards.
• Tracking methodologies, such as cookies, packet sniffers and IP addresses all collect data in different ways and all have pros and cons to the way in which they do this. See example below for further info on this one.
• The Internet is composed of a huge array of different technologies, which are all constantly evolving and changing. These technologies play a big part in the accuracy of data collection.
• New browser versions invariably feature new types of technology that allow increasingly savvy web users to hide their on-line behaviour, or even block this behaviour by default.
• Robots and spiders crawl Internet pages in order to e.g. index what is in them for search engines. Data quality in web analytics is a race to keep up with these creatures!

Cookies and Unique Visitors – An Example

The issue of cookies is generally the biggest area of confusion. A client of mine was recently comparing Google Analytics to their incumbent provider, Sophus3. They noticed large differences in unique visitors and wanted to understand why. Whilst this issue is in some respect the product of all the points raised above, the main cause is the type of cookie used:

With Google Analytics, visitors are tracked using 1st party cookies. Estimates suggest that around 1% of users block these cookies and a further 4% block JavaScript. GA is therefore unable to track these users, so real visitors may be under-counted by about 5%.

Sophus3, on the other hand, uses 3rd party cookies. Many browsers block these by default, so estimates suggest that around 65% of traffic is lost due to the combination of this and JavaScript blocking.

Sophus3 then use IP address to track visitors who have blocked cookies. However, most broadband providers use dynamic IP addresses, which change periodically. In some cases, the IP address could change every time the person switches on their computer. Therefore, Sophus3 will register individual people as multiple visitors, and overall numbers will therefore be inflated.

The following chart illustrates this issue in a more visual way (numbers are rough estimates to illustrate a point, and are not meant to be accurate):

How cookies can affect data accuracy in web analytics

Whilst 1st party cookies are generally considered in the industry to be best practice, in truth neither is perfect. For more information, here is a more detailed overview of how cookies affect web analytics data.

Get over it!

The issue of data accuracy can cripple companies and cause vast amounts of wasted time. In truth there is no solution, it is much better to:

• Understand the limitations in as much detail as possible and ensure that all recipients of web reporting and analysis are familiar with what the numbers do and don’t tell them.
• Focus on trends and segments, and not on absolute numbers. This is easy to do when the focus is on analysis and not pure reporting; insight never comes from pure numbers.
• Where numbers such as unique visitors are required for decision making, confidence levels should be used to make reasonable judgements about those numbers.
• If we set a consistent base-line of data at the most accurate that we can get it, then we can use this data to make accurate trend assumptions and draw conclusions about time-series analyses.