Looking back on three decades of dunnhumby’s data science

Data science doesn’t stand still. Just as computers are exponentially more powerful than they were three decades ago, the models and techniques that we apply today have changed significantly since dunnhumby was founded back in 1989.

While some of that change is down to natural evolution, much of it comes from a conscious desire to stay at the forefront of the industry. We’re strong believers in the need to keep moving forwards and innovating, not only for the good of our clients but for our people too.

As much as it’s important to look forwards, looking back can also help us understand more about how we got to where we are today. And while it isn’t possible to do justice to 33 years of progress in a single blog post, we can at least look at some of the milestone moments along the way and how they’ve helped shape both dunnhumby and retail data science as a whole.

With that in mind, let’s explore the six ages of dunnhumby’s data science so far.

1. Setting out to define an industry

While dunnhumby was around for a few years before working with Tesco on the Clubcard, it’s that pioneering programme that represents the first step on the journey towards the company as we know it today.

They may seem like basic concepts now, but our work with the Clubcard helped to define some of the foundational elements of Customer Data Science (CDS). Here, for the first time, was the longitudinal view – the ability to understand how Customers buy grocery products over time. So too was a deeper, richer understanding of shoppers that allowed us to segment them based on their purchase preferences.

From this came the opportunity to start to define Customer loyalty, something that we could take an informed view on based on trip frequency and purchasing behaviour. During this period, we also started to develop what then became some of our best-known proprietary algorithms– like the rolling ball technique, which explores item-by-item purchases to build a complete view of a shopper.

2. Broadening out, honing in

With the foundations for our approach to CDS firmly in place, we started to look at how we could refine and improve the way that we were using some key analytical techniques. During this time, we focused heavily on areas like clustering, factor analysis, decision trees and logistical regression.

The progress we made here took us much further in the direction of personalisation. We developed the ability to create true one-to-one communications, tailoring mailings based on what Customers had already purchased. Attitudinal data from smaller shopper groups added an extra dimension to our longitudinal frameworks.

We also expanded our capabilities within category management, particularly around substitutability. This enabled us to start predicting what shoppers would buy when they couldn’t find what they initially wanted – and with a very high degree of accuracy.

3. Bringing the outside in, one partnership at a time

This third “age” saw a true step change in the way that we developed our data science. By this point, we were already working with a number of leading Retailers around the world – something that enabled us to shift towards a more collaborative approach to innovation.

Two things complemented this transition. The first was the creation of our academic partnerships programme, in which we built relationships with world leading institutions. This enabled us to invest in both the people and the science shaping the future of the industry.

The second was the completion of a number of key acquisitions, all designed to help bolster our abilities in key areas like econometric modelling and continuous optimisation. The amazing work done by many of these companies still exists in core parts of the dunnhumby product portfolio to this day.

4. Data science goes global as tech drives change

A renewed focus on personalisation was one of the defining factors here. The entire business began to centralise around true “segment of one” Customer insights, and the delivery of hyper-relevant, deeply tailored communications. Increased investment in the science to support that, as well as the formation of a global client team brimming with subject matter experts, helped to solidify our efforts.

Across the wider industry at this point, technology was starting to transform the nature of data science, with new languages and applications like Python and PySpark helping to redefine what was possible. Ever keen to remain at the cutting edge, many of our team started to engage in the (then relatively nascent) field of machine learning by entering, setting, and winning multiple Kaggle competitions.

5. Machine learning hits the mainstream

While our focus on machine learning up to this point had been relatively self-contained, we soon began to build up a global community around it. This helped to ensure that the core techniques and advanced methods that we’d started to develop could be employed more broadly by our data science colleagues around the world.

This mass of skills and talent helped to drive some phenomenal innovations, including ModelLab which automates the tuning and deployment of machine learning algorithms, the superfast accelerator oneFour and dunnhumby Beyond, a machine learning-powered insight tool that analyses data from more than 2.5m UK shopping trips.

Our media, category management, and price & promotion science also saw considerable evolution at this time, with greater automation helping to remove the manual burden from users.

6. Into tomorrow

All of which takes us full circle to today. In 2022 and beyond, we’ll be focused on the inherent potential of new technologies like quantum computing and the huge opportunities it offers in terms of speed and power over the next decade.

We’re not just fixated on the future though. We’re also looking at the day to day, and what the next trends and evolutions in data science will be. As we actively look into new sources of data like social, video and voice, we are excited about how they might transform our ability to understand Customers, keeping in mind that as we get to grips with the differing levels of quality that they present, they need to be appropriately structured, respecting data governance, privacy and ethics by design.

Wherever tomorrow takes us, it will be with the same lens we’ve always had: Customer First. The data science may change, but the belief in using it to do what’s right for shoppers never will.