Summary In this episode of the Data Engineering Podcast Tulika Bhatt, a senior software engineer at Netflix, talks about her experiences with large-scale data processing and the future of data engineering technologies. Tulika shares her journey into the data engineering field, discussing her work at BlackRock and Verizon before joining Netflix, and explains the challenges and innovations involved in managing Netflix's impression data for personalization and user experience. She highlights the importance of balancing off-the-shelf solutions with custom-built systems using technologies like Spark, Flink, and Iceberg, and delves into the complexities of ensuring data quality and observability in high-speed environments, including robust alerting strategies and semantic data auditing.

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data managementData migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.Your host is Tobias Macey and today I'm interviewing Tulika Bhatt about her experiences working on large scale data processing and her insights on the future trajectory of the supporting technologiesInterview IntroductionHow did you get involved in the area of data management?Can you start by outlining the ways that operating at large scale change the ways that you need to think about the design of data systems?When dealing with small-scale data systems it can be feasible to have manual processes. What are the elements of large scal data systems that demand autopmation?How can those large-scale automation principles be down-scaled to the systems that the rest of the world are operating?A perennial problem in data engineering is that of data quality. The past 4 years has seen a significant growth in the number of tools and practices available for automating the validation and verification of data. In your experience working with high volume data flows, what are the elements of data validation that are still unsolved?Generative AI has taken the world by storm over the past couple years. How has that changed the ways that you approach your daily work?What do you see as the future realities of working with data across various axes of large scale, real-time, etc.?What are the most interesting, innovative, or unexpected ways that you have seen solutions to large-scale data management designed?What are the most interesting, unexpected, or challenging lessons that you have learned while working on data management across axes of scale?What are the ways that you are thinking about the future trajectory of your work??Contact Info LinkedInParting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.init covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.Links BlackRockSparkFlinkKafkaCassandraRocksDBNetflix Maestro workflow orchestratorPagerdutyIcebergThe intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Data teams are faced with increased demand for trusted data assets, but struggle to quickly and safely grant access to the most validated data. In this session, Schroders will share how they've enhanced discoverability and governance of data across the enterprise using Collibra's unified governance for data and AI to enable data consumers to easily find and access the trusted data they need. Learn how building a curated marketplace of data products has helped foster greater collaboration and increased the safe usage of data for greater advantage.

Join us for a session focused on unlocking the real potential of AI through product thinking. Dive into why AI projects aren’t taking off. Discover the most important metric for GenAI applications. We'll walk through Shape, Nail, Scale, Run – a practical framework for turning AI ideas into successful, scalable solutions. Be inspired by a real-world case study of what we did for the UK’s oldest private bank with product-led thinking. This session is perfect for product leaders, designers, and anyone looking to bring AI to life in meaningful ways.

Given the ease with which AI-based initiatives can be executed nowadays, it is no surprise that more and more enterprises are considering adopting AI-powered autonomous analytics to make well-informed decisions more economically, improve operational efficiencies, and delight customers. Achieving the true promise of AI, however, requires access to data and the execution of such analytics in real time. Join this session to learn about data processing challenges and architecture patterns fueling AI-driven businesses.

Responsible AI decisions are not black and white, they require trade-offs. Learn to make trade-offs and debate the alternatives to make AI governance and responsible AI decisions. Discuss controversial ideas in AI with your peers. Express your opinion and listen to what others are saying. Learn to ask the right questions and get the right answers to ensure responsible, trustworthy and ethical AI.

Asking your colleagues how analytics can help them often results in blank stares, defensiveness, or wildly incoherent suggestions involving AI. This session will show you how to work with your colleagues to pinpoint how you can help, identify the most helpful capabilities to build, and explain how to measure your impact.

Tech and service providers are rapidly providing new AI capabilities and features to customers and hoping for leadership. Unfortunately, customers don't really know what to do with these features and are balking at the cost. This session explores a value journey, from features to transformation using a "value accelerator" model designed to align product strategies with customer outcomes.

Data management resources face increasing challenges due to growing demand, including the need for AI-ready data. It is essential for data management leaders to make sure that data management is not misdirected and aligns with the business value. Proper alignment will maximize the value of data management initiative and support business growth.

A major shift is happening in the data space and it’s not just AI. Handling the explosive growth of data while managing costs - all without compromising performance is pushing leaders to reevaluate Data ROI. In this session, we explore how Exponential-e, a leading cloud, networking and services provider, adopted strategies to minimize TCO for data integration and to create additional value with data. Learn how they deliver intelligent systems that are better connected with automated data pipelines and how to put this data to use using managed AI agents.

Among organizations that have adopted generative AI, only 33% report implementing it into functional processes. To seize productivity gains from AI, enterprises need to get the most from their data, ensure governance, and make it easier to use. For many, that means better integration with the workforce tools already in use. Learn how companies are developing AI agents that act with greater autonomy and how they are scaling them across diverse environments to deliver impact. Discover how to manage your AI assistants and agents in a unified, adaptable experience that safeguards investments.

In the era of AI, harnessing data analytics is essential for innovation. Organizations are transforming their data strategies to harness the full potential of AI.  

Learn how organizations create powerful AI experiences by unifying their data and transforming their data strategies resulting in significant operational efficiencies.

AI is data hungry. The lack of data is the most frequently identified barrier to AI adoption. We’re not lacking in data, per se. We’re lacking data that is AI-ready. And traditional data governance practices often hinder AI project progress.

Learn how a European AI innovation leader more than halved the wait time for AI-ready data, saving hundreds of pounds per year by implementing both a data enablement culture and a data policy enforcement technology solution.

Looker’s AI-first analytics experience, with a conversational interface, enables all users in your organization to leverage trusted data and make better decisions. Discover how you can lay the foundations to deliver best-in-class conversational AI experiences. Join us, along with a cohort of your peers, to participate in discussions around foundational strategies for conversational AI and share existing use cases and experiences.

We are presently in the midst of an artificial intelligence (AI) revolution, where advances in deep learning technologies are rewriting the rules of entire industries – including how we forecast the weather and predict changes in our climate. In this session we will learn how the Met Office is successfully navigating the opportunities and challenges (both technological and cultural) of embedding AI across the organisation, and discuss how the complementary use of AI alongside traditional physics-based weather and climate models will further help people make better decisions to stay safe and thrive.

In an era where speed-to-insight defines competitive advantage, Dunelm is embracing a new paradigm: agentic analytics. This session explores how the UK’s leading homewares retailer is transforming decision-making by empowering business users with real-time, self-serve access to data—without relying solely on data teams. Learn how Dunelm is leveraging AI-powered analytics to surface insights proactively, drive operational efficiency, and reduce the time from question to answer. Whether you're modernising your data stack or looking to scale data literacy across the business, Dunelm’s journey offers a practical look at putting agentic analytics into action.

See through the hype to better understand the practical applications, risks, and aspirations of AI in the data governance industry. Find out how to position your data governance program as a leader for your organizations' AI initiatives.

AI explainability is crucial for fostering trust and creating responsible AI by providing transparency into decision-making processes, enabling accountability and mitigating biases. Join this session to know about some practices that D&A technical professionals must adopt to enhance AI explainability and trust, and ensure the responsible and accountable use of AI.

Lakehouse has become a cornerstone in managing large and heterogeneous data by providing capabilities that simplify organizational data architecture, unify data assets, and help streamline, harmonize, and enhance data processes, operations and governance. Lakehouse provides high value to the organization, reduces technical debt, and prepares the organization for new frontiers like AI.

Most metadata in 2024 will remain passive in approaches with stats, reports, schema and business-developed glossary terms. Most organizations must grow their maturity in metadata management. We start with traditional metadata techniques — passive. With AI undergoing confidence issues and the demand to reduce risk, grow AI confidence, and provide data assurance, active metadata becomes key.