Briefing

Last year was a year of migrations for anyone working in CRO and using Google products. 

Beyond the analytics migration to Google Analytics 4 (GA4), the procurement and onboarding of a new A/B testing tool was required by those using Google Optimize. 

Integrating this new A/B tool with GA4 and ensuring data accuracy for additional reporting and segmentation became a key challenge.

This is because no two data platforms match perfectly and, in this case, new GA4 features require you to reconsider how you use its data in your A/B test reporting.

GA4 data and A/B testing

So, why exactly is this extra consideration required?

The lag time – Google state that it’s a 24-48 hour lag time for GA4 to process data but many experience a 72 hour delay – during that time data can change in front of your eyes, making reporting inconsistent.

HyperLogLog – This is a clever probabilistic approach for estimating user and session counts, HyperLogLog had always existed in Universal Analytics but with the option of turning it off. With GA4, you can no longer do this and a different user or session count will, of course, impact and change any conversion calculations that you perform against the data, which is not ideal for A/B test reporting.

Thresholding – Depending on your settings,  the GA4 interface will hide rows of data  when your website’s visitors fall below a minimum threshold, this is for privacy purposes but means you can’t see the whole picture and if information is missing, you simply lose trust in the data presented.

Modelled data – Google uses modelling to estimate online conversions that can’t be observed directly e.g. for users that declined consenting to cookies, which again creates a trust issue in the data presented in the GA4 interface.

Using raw data from BigQuery

Whilst all of this is hard to get your head around, all is not lost. Clean unsampled, unmodelled data is still stored in the database, it’s just not displayed in GA4.

To get your hands on that data you need to set up access to Google BigQuery, Google’s cloud data warehouse solution, which is an integral part of the Google Cloud Platform. 

Google BigQuery houses all your GA4 data including any experiment data from your integrated A/B test tool in real-time, and with native integrations with Looker Studio and Google’s Connected Sheets, it provides an easy, cost effective and automated way of powering your A/B test analysis without any limitations.

With a streaming option meaning data will arrive within minutes, Google BigQuery gives experimentation teams the opportunity to query clean data to ensure statistical validity for any A/B test analysis performed against GA4 data, reducing business risk and improving confidence in insight.

How to get started

If you aren’t already, to start using BigQuery for your A/B test reporting, first ensure your website and/or app KPIs and behavioural touchpoints are robustly tracked and your A/B testing tool is integrated with GA4. Then link BigQuery to GA4, following our guide to getting set up and exporting data.

If you need support, talk to our CRO team about creating automated reporting dashboards for your experimentation analysis to ensure statistically sound insights into your CRO test performance.