What's more sexy: analytics or innovation? What about combining them! That sounds great, and Thomas Davenport would be so proud if you pulled it off, but the reality is that the idea of innovation through analytics is one thing, while the reality of making it happen is another thing entirely. Dr. Tiffany Perkins-Munn, Head of Marketing Data & Analytics at JPMorgan Chase & Co., joined us for a discussion on the subject! For complete show notes, including links to items mentioned in this episode and a transcript of the show, visit the show page.

Summary

This podcast started almost exactly six years ago, and the technology landscape was much different than it is now. In that time there have been a number of generational shifts in how data engineering is done. In this episode I reflect on some of the major themes and take a brief look forward at some of the upcoming changes.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management Your host is Tobias Macey and today I'm reflecting on the major trends in data engineering over the past 6 years

Interview

Introduction 6 years of running the Data Engineering Podcast Around the first time that data engineering was discussed as a role

Followed on from hype about "data science"

Hadoop era Streaming Lambda and Kappa architectures

Not really referenced anymore

"Big Data" era of capture everything has shifted to focusing on data that presents value

Regulatory environment increases risk, better tools introduce more capability to understand what data is useful

Data catalogs

Amundsen and Alation

Orchestration engine

Oozie, etc. -> Airflow and Luigi -> Dagster, Prefect, Lyft, etc. Orchestration is now a part of most vertical tools

Cloud data warehouses Data lakes DataOps and MLOps Data quality to data observability Metadata for everything

Data catalog -> data discovery -> active metadata

Business intelligence

Read only reports to metric/semantic layers Embedded analytics and data APIs

Rise of ELT

dbt Corresponding introduction of reverse ETL

What are the most interesting, unexpected, or challenging lessons that you have learned while working on running the podcast? What do you have planned for the future of the podcast?

Parting Question

From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

Thank you for listening! Don't forget to check out our other shows. Podcast.init covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story. To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Sponsored By: Materialize:

Looking for the simplest way to get the freshest data possible to your teams? Because let's face it: if real-time were easy, everyone would be using it. Look no further than Materialize, the streaming database you already know how to use.

Materialize’s PostgreSQL-compatible interface lets users leverage the tools they already use, with unsurpassed simplicity enabled by full ANSI SQL support. Delivered as a single platform with the separation of storage and compute, strict-serializability, active replication, horizontal scalability and workload isolation — Materialize is now the fastest way to build products with streaming data, drastically reducing the time, expertise, cost and maintenance traditionally associated with implementation of real-time features.

Sign up now for early access to Materialize and get started with the power of streaming data with the same simplicity and low implementation cost as batch cloud data warehouses.

Go to materialize.comSupport Data Engineering Podcast

Colleague Dante DeAntonio joins the podcast for another round of Job's Friday. The group dissects the January report, which included a shockingly large increase in jobs and a 53-year low unemployment rate. Everyone's probability of recession in the next 12-18 months appear to be trending downward. Including Cris! Full episode transcript Follow Mark Zandi @MarkZandi, Cris deRitis @MiddleWayEcon, and Marisa DiNatale on LinkedIn for additional insight

Questions or Comments, please email us at [email protected]. We would love to hear from you.    To stay informed and follow the insights of Moody's Analytics economists, visit Economic View.

Do you need to be active on Linkedin to land a data job? What should you post or even do on the platform?

In this episode, Avery sits down with teacher turned Data Analyst Chris French to discuss how he landed his first data job leveraging Linkedin.

🌟 Join the data project club!

“25OFF” to get 25% off (first 50 members).

📊 Come to my next free “How to Land Your First Data Job” training

🏫 Check out my 10-week data analytics bootcamp

Chris’s Links:

Connect on LinkedIn

Timestamps:

(9:07) - How Chris grow Linkedin from 20 followers to 20,000 followers

(13:19) - His content types that gain attention

(18:02) - What Chris would do differently to do the job search

(23:29) - Technical skills vs soft skills

(26:53) - Before land data job, analyze your process first

Connect with Avery:

📺 Subscribe on YouTube: https://www.youtube.com/c/AverySmithDataCareerJumpstart/videos 🎙Listen to My Podcast: https://podcasts.apple.com/us/podcast/data-career-podcast/id1547386535 👔 Connect with me on LinkedIn: https://www.linkedin.com/in/averyjsmith/ 📸 Instagram: https://www.instagram.com/datacareerjumpstart/ 🎵 TikTok: https://www.tiktok.com/@verydata?

Mentioned in this episode: Join the last cohort of 2025! The LAST cohort of The Data Analytics Accelerator for 2025 kicks off on Monday, December 8th and enrollment is officially open!

To celebrate the end of the year, we’re running a special End-of-Year Sale, where you’ll get: ✅ A discount on your enrollment 🎁 6 bonus gifts, including job listings, interview prep, AI tools + more

If your goal is to land a data job in 2026, this is your chance to get ahead of the competition and start strong.

👉 Join the December Cohort & Claim Your Bonuses: https://DataCareerJumpstart.com/daa https://www.datacareerjumpstart.com/daa

On today’s episode, we’re joined by Ellie Fields. Ellie is the Chief Product and Engineering Officer at Salesloft, which helps sales teams drive more revenue with the only complete sales engagement platform available in the market. We talk about:

• Ellie’s background and what Salesloft does.
• The changing trends in how companies use data.
• Drawing valuable insights from unstructured data.
• Putting workflow at the center of what you do, and the challenges involved.
• Ellie’s experiences managing both product and engineering.
• Are more autonomous teams more scalable?
• Applying a metric- and data-oriented culture internally.
• The impact of remote work on how companies operate.

Ellie Fields - https://www.linkedin.com/in/elliefields/ Salesloft - https://www.linkedin.com/company/salesloft/

This episode is brought to you by Qrvey

The tools you need to take action with your data, on a platform built for maximum scalability, security, and cost efficiencies. If you’re ready to reduce complexity and dramatically lower costs, contact us today at qrvey.com.

Qrvey, the modern no-code analytics solution for SaaS companies on AWS.

saas #analytics #AWS #BI

Kamal will share his experiences in entering new competitive markets & global best practices.

Fireside Simolation is a safe space for all participants to have a friendly discussion about all the things that bother, inspire, or preoccupy you in the digital analytics space.

Even though the resources are always limited it might be tempting to focus on collecting data the right way instead of collecting the right data.

This session will cover a modern data stack approach that drives analytics, feature engineering, and dashboards across the enterprise. It will showcase real time streaming capabilities, data lakes, decentralized vs. centralized warehousing, and feature store architecture. The solution provides sub second latency to data and scales to billions of transactions a day.

As a digital analyst the core of our job is to translate what the data is saying about the business so we can use that knowledge to improve it. Pretty simple, awesomely valuable and radically focused on facts.

So you've got analytics working but you need to grow your data collection to other websites, other markets, other apps? In this presentation, Julien and the Media.Monks data team share best practices and solutions that were used (and often custom built!) to achieve at-scale deployment of data collection across thousands of websites and apps.

After a decade of digital marketing, I've decided to transition into digital analytics in 2022 because I wanted to work even more with data to solve complex problems.

In this talk, we will delve into the features and capabilities of Google Analytics 4's measurement protocol. We will discuss its strengths and limitations, and explore how the client-side protocol can be used to bypass some of these limitations. Finally, we will present a real-world use case for the client-side protocol and demonstrate its potential benefits.

Web bots have been around for decades, but the volume of bot traffic has increased massively. So has the complexity of bot filtering. While the bots have become way better, bot filtering solutions haven’t – especially in common analytics tools. Even IT security teams are usually not handling the problem properly.

I'll discuss why, and how data makes me tick. I've been doing work with data for decades but it's more exciting now than ever. Why do I get nerdy excitement from data? Because of my personal data, because data fuels change, because data is used for good. I'm immersed in data that has purpose on a daily basis.

Over the past 2 decades, Jim has watched the industry take on many different forms, maturing from the lawless wild west of the digital industry to an increasingly-regulated establishment. This departure from the lawless world of analytics has left some of us looking for new, exciting frontiers. Jim will share his perspective on where we might find it.

A delightful way to gather and continue socializing: competitive team challenges and prizes that will give you a short break from analytical topics.

Demonstrate any Digital Analytics solutions or method of your own that is way beyond the defaults. Who decides who's gonna win? The audience. Send your nomination to [email protected]!

Shifts in technology and the rise of privacy laws are reshaping the industry, bringing the uncertainty to analytics professionals' work. And the most significant changes are still ahead of us.

Kris Ewald will give you an overview of Innovations in Data Science you should be aware of. If data driven insights are key to competitiveness, you need to keep innovating on how you Collect, Manage and Challenge data. With plenty of other talks about very specific tools and data analytics frameworks, this talk will instead aim to inspire you to apply new approaches to your data science - it'll give you a list of topics you should care about and pay attention to. Expect to hear about Zero-knowledge proofs, Homomorphic encryption, DAGs, and Blockchain and data as value objects.