After a period of relative quiet, the industry appears to be interested again in multi-armed bandits. To find the wheat from the chaff within the marketing hype, it is important to at least understand bandit basics: what they are; how they compare to AB Tests; when to use them; and how they work.

While we don't often call it out explicitly, the driving force behind much of what and how much data we collect is driven by a "just in case" mentality: we don't know exactly HOW that next piece of data will be put to use, but we better collect it to minimize the potential for future regret about NOT collecting it. Data collection is an optionality play—we strive to capture "all the data" so that we have as many potential options as possible for how it gets crunched somewhere down the road. On this episode, we explored the many ways this deeply ingrained and longstanding mindset is problematic, and we were joined by the inimitable Matt Gershoff from Conductrics for the discussion! For complete show notes, including links to items mentioned in this episode and a transcript of the show, visit the show page.

Often in analytics and data science we have the 'big table' mental picture of data where we are continuously trying to append and link new bits of data back to each customer. The issue is that using approaches that follow this model often don't really follow a privacy by default design - rather this is more of an identify by default approach.

As analysts, it can be easy for us to lose the forest from the trees. By focusing primarily on technology, or analytical methods, we can lose sight of our ultimate objective which is to satisfy the wants and needs of our customers.

Matt will be your quiz master for a fun filled night of quizzical conundrums, bragging rights, and prizes.

We will discuss why AB Testing, and analytics more generally, always involves uncertainty. We will then briefly discuss the causal inference problem, and how AB Tests are one of the main ways to help solve these problems. In the course of the talk we will briefly touch on types of reasoning, the importance of assignment, and the logic of p-values.

We will review the basics of Multi Armed Bandits and contrast it with AB Testing.

Have you ever looked at different data sets and thought,’wow this data set is really informative but this other data set, not so much’’? Or though ‘this data is really surprising’. As analyst, wouldn’t it be amazing, almost magical, if rather than just using ‘information’ and ‘surprise’ as qualitative terms, we could actually quantify both how surprised we are, and how much information one data set has vs another?

Machine learning can be thought of as sitting at the intersection of computer science and statistics. "Computer Science has focused primarily on how to manually program computers, Machine Learning focuses on the question of how to get computers to program themselves ..." – Tom Mitchel

Let's pretend your goal as an analyst is to eloquently and accurately explain reinforcement learning. Now, let's pretend that you get to try that explanation again and again, and we'll give you an electric shock every time you state something inaccurately and a cookie every time you say something right. Well, you're an analyst, so you're now wondering if this is some clever play on words about cookies. As it happened, we didn't give Matt Gershoff from Conductrics any cookies of any kind in his return to the show. Instead, we gave him a lifetime's supply of opportunities to say, "Well, no, it's not really like that," which is a special kind of nourishment for the exceptionally smart and patient! In other words, the gang walked through a range of analogies and examples about machine learning, reinforcement learning, and AI with Matt, and no electric sheep were harmed in the process.  For complete show notes, including links to items mentioned in this episode and a transcript of the show, visit the show page.

Matt will argue that Optimization can be, and perhaps should be, framed as a sequential decision problem. Viewing optimization this way will let us unify Testing, Targeting, and even Attribution, as interrelated subtasks. Matt will then introduce Reinforcement Learning, a method from the field of Artificial Intelligence, as a general approach to find optimal policies over multi-touch problems. Note: This talk will be mostly conceptual. I will however provide a list of resource for those interested in learning more. If you are looking for a 'three practical skills to bring back to the office on Monday' talk, you may want to skip.

We've got the technology. We've got the behavioral data. We've got the content (or at least tell ourselves we do). We're all set to develop personalized experiences that knock consumers socks off and leave them begging us to take their money. Is it really that simple? If it is, why aren't more companies realizing the dream of 1-to-1 marketing? Matt Gershoff joins us to discuss how the pieces of the personalization puzzle often don't quite fall into place like we wish they would. Matt's also written a post that overlaps with our discussion: http://conductrics.com/complexity.

Personalization, one to one, predictive targeting, whatever you call it, serving the optimal digital experience for each customer is often touted as the pinnacle of digital marketing efficacy. But if predictive targeting is so great, why isn’t everyone doing it? In this session Matt will give an overview of predictive targeting methodologies as well as a general framework for thinking about the trade offs you will face when embarking on embedding predictive methods into your marketing systems/process. A warning: While this talk assumes only basic statistical knowledge, it will introduce some relatively advanced/technical concepts. If you are looking for a ‘Top Five Practical Predictive Analytics Tips’ type of talk, you may want skip.

In all the excitement around Big Data and Analytics, even savvy users of business intelligence can get a bit confused about how and when to use A/B Testing, Predictive Analytics, and Personalization to optimize. But optimizing isn’t about choosing which tool to use: Optimizing is about making decisions. The digital environment gives us an opportunity to make these marketing decisions at scale. In this session we’ll discuss how to bring these tools together to make better decisions, we’ll also touch on how machine learning can help us automate the process to free up analytics teams to focus on the higher value problems.