talk-data.com

One of the most pressing challenges in our increasingly data-driven world is the Data Deluge—the overwhelming flood of information that we generate and record every single day. With us are three experts from the University of Virginia’s School of Data Science. Phil Bourne, a professor of biomedical engineering and the founding dean of the school of data science, is joined by Terence Johnson and Alex Gates, both assistant professors of data science. Together, they have been exploring innovative methods to make sense of the vast oceans of data we’re all swimming in.

This episode unpacks the challenges of the data deluge—what it means for businesses, researchers, and society at large—and explore the strategies we can use to navigate it. How do we make sense of so much information? How do we ensure the ethical use of this data? And what opportunities does this overwhelming flood of data open up for the future?

Summary

Every business with a website needs some way to keep track of how much traffic they are getting, where it is coming from, and which actions are being taken. The default in most cases is Google Analytics, but this can be limiting when you wish to perform detailed analysis of the captured data. To address this problem, Alex Dean co-founded Snowplow Analytics to build an open source platform that gives you total control of your website traffic data. In this episode he explains how the project and company got started, how the platform is architected, and how you can start using it today to get a clearer view of how your customers are interacting with your web and mobile applications.

Preamble

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to dataengineeringpodcast.com/linode to get a $20 credit and launch a new server in under a minute. You work hard to make sure that your data is reliable and accurate, but can you say the same about the deployment of your machine learning models? The Skafos platform from Metis Machine was built to give your data scientists the end-to-end support that they need throughout the machine learning lifecycle. Skafos maximizes interoperability with your existing tools and platforms, and offers real-time insights and the ability to be up and running with cloud-based production scale infrastructure instantaneously. Request a demo at dataengineeringpodcast.com/metis-machine to learn more about how Metis Machine is operationalizing data science. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch. Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat This is your host Tobias Macey and today I’m interviewing Alexander Dean about Snowplow Analytics

Interview

Introductions How did you get involved in the area of data engineering and data management? What is Snowplow Analytics and what problem were you trying to solve when you started the company? What is unique about customer event data from an ingestion and processing perspective? Challenges with properly matching up data between sources Data collection is one of the more difficult aspects of an analytics pipeline because of the potential for inconsistency or incorrect information. How is the collection portion of the Snowplow stack designed and how do you validate the correctness of the data?

Cleanliness/accuracy

What kinds of metrics should be tracked in an ingestion pipeline and how do you monitor them to ensure that everything is operating properly? Can you describe the overall architecture of the ingest pipeline that Snowplow provides?

How has that architecture evolved from when you first started? What would you do differently if you were to start over today?

Ensuring appropriate use of enrichment sources What have been some of the biggest challenges encountered while building and evolving Snowplow? What are some of the most interesting uses of your platform that you are aware of?

Keep In Touch

Alex

@alexcrdean on Twitter LinkedIn

Snowplow

@snowplowdata on Twitter

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Links

Snowplow

GitHub

Deloitte Consulting OpenX Hadoop AWS EMR (Elastic Map-Reduce) Business Intelligence Data Warehousing Google Analytics CRM (Customer Relationship Management) S3 GDPR (General Data Protection Regulation) Kinesis Kafka Google Cloud Pub-Sub JSON-Schema Iglu IAB Bots And Spiders List Heap Analytics

Podcast Interview

Redshift SnowflakeDB Snowplow Insights Googl

Have you ever seen a one-man show in the theater? It's awesome. Unless it's terrible. The same can be said for one-person digital analytics teams. It can be awesome, in that you get to, literally, do EVERY aspect of analytics. It can be terrible because, well, you've got to do EVERYTHING, and it's easy for the fun stuff to get squeezed out of the day. On this episode, we head back Down Under for a chat with Moe Kiss, product (and digital) analyst at THE ICONIC. Whether you pronounce "data" as DAY-tuh or DAH-tuh, Moe's perspective will almost certainly motivate you find new ways to push yourself and your organization forward. People, places, things, sites, and doodads mentioned in this episode were many, and they include: R, Tableau, Snowplow, adjust, Datalicious, Moe's post on Analysis of Competing Hypotheses, Moe's post on getting started in digital analytics, Jeffalytics.com, RSiteCatalyst, The Millenial Whoop, Kabaddi, Michael Yates, ABC (the Australian Broadcasting Corporation), an Event Tracking Naming Strategy from Chris Le, Simo Ahava, Nico Miceli, and Towards Universal Event Analytics - Building an Event Grammar by Snowplow co-founder Alex Dean.