Speed is everything in technology. The performance of enterprise tools effects everything from crucial business decisions and growth strategies to new product development and customer acquisition.
That’s why in the Fall of 2016, the Resonate engineering team started exploring how we could make our consumer insights platform much faster and more robust. We knew the platform at that time ran slow and limited the number of users at any given moment. Thankfully those days are over.
The engineering team spent the better part of a year revamping our analytics engine using Apache Geode, a distributed in-memory cache and query engine used in data management. It was a unique way of using Apache Geode’s technology, so my colleague, Sharif Ghazzawi, and I were chosen to present our approach at the recent SpringOne Conference in Washington, D.C. We’ll spare you the gritty tech details, but here is a quick summary of how we had transformed the Resonate Platform to make it even more robust and unique in the market. It was truly the most complicated and exciting engineering project of our careers.
HERE’S HOW IT HAPPENED
The Resonate Platform computes 10,000+ attribute values for over 200 million cookies. That means we’re looking at 1.7 trillion individual records of someone hitting one of our client’s digital touchpoints. To provide the analytics on these interactions, we examine every single data point. Yes, it’s as daunting as it sounds.
To understand the old way we processed analytics for clients, picture a large spreadsheet with over 200 million records. Every answer would need to be found in that spreadsheet. For example, say you want to know how many men drive Subarus. In that monster spreadsheet, you’re having to go down 200 million rows to filter out all the males. And then you’re going down the row of car owners to filter out who drives a Subaru.
We went from that very manual and slow process to probabilistic counting, where you have a piece of data that resembles the parts of that spreadsheet. It doesn’t take up as much space and allows you to do the same operations but much faster. With the new approach, we’re able to have a data structure that represents the set of people who drive a Subaru. We also have another data structure that represents the set of people who are men. We then take the two data structures to create a third one with our answer. As you can imagine, it’s a lot more efficient than having to go through 200 million rows of data. With this probabilistic approach, we’re able to estimate these counts and control the amount of accuracy on our platform.
LIFE IN THE FAST LANE
Before the move to Geode, the Resonate infrastructure was expensive to operate, with 40 nodes running to power the solution. Not to mention, it took far too long for a client to receive an answer to their query. Going from 40 machines to 10 means we can now provide the same precise answers but much faster to our clients. We can also support a higher number of customers—the user experience slowed down considerably in the old infrastructure when a certain number of people were using the platform at the same time.
Now our clients can run queries faster, which means they can run more queries than ever before. They’re also getting the most updated consumer insights now that we’re able to update the data in our platform nightly. One of the biggest challenges in the market is that data is stale by the time it gets to the marketer.
Using Apache Geode also lets us add more insights to the platform. We now have a system of bringing in new data, modeling it and representing it in our solution. So it’s much more scalable and allows us to provide more data points to our customers.
HOW WE WERE ABLE TO DO THIS
When embarking on a project of this scale, there was the early question of whether we buy or build this solution. We did explore the buy option but realized there was nothing out there to serve our needs. What we developed is a one of a kind solution to a one of a kind problem.
One of the most satisfying aspects of this project was how Resonate had all of the skills it required in-house. It involved engineers of different expertise, everything from critical thinking and database structures to design, data science and Java skills. No one person has all of these skills, so the completion of this project speaks to the willingness of the team to collaborate, bring in new perspectives and talk through the best solutions. Throughout the process, we were challenged with mysterious outages, multiple stress tests, working through weekends and exercises where we stayed in a room until all the problems were fixed. In the end, we have a platform that provides better numbers, faster than anyone else. Nobody has the data we have at the scale and scope that we have it. It was a real feat of engineering.
You should really see it in action. Reach out for a demo of what the Resonate Platform can do, specific to your industry.
