The role of data teams and data engineers is evolving. No longer just pipeline builders or dashboard creators, today’s data teams must evolve to drive business strategy, enable automation, and scale with growing demands. Best practices seen in the software engineering world (Agile development, CI/CD, and Infrastructure-as-code) from the DevOps movement are gradually making their way into data engineering. We believe these changes have led to the rise of DataOps and a new wave of best practices that will transform the discipline of data engineering. But how do you transform a reactive team into a proactive force for innovation? We’ll explore the key principles for building a resilient, high-impact data team—from structuring for collaboration, testing, automation, to leveraging modern orchestration tools. Whether you’re leading a team or looking to future-proof your career, you’ll walk away with actionable insights on how to stay ahead in the rapidly changing data landscape.

This course provides a comprehensive review of DevOps principles and their application to Databricks projects. It begins with an overview of core DevOps, DataOps, continuous integration (CI), continuous deployment (CD), and testing, and explores how these principles can be applied to data engineering pipelines. The course then focuses on continuous deployment within the CI/CD process, examining tools like the Databricks REST API, SDK, and CLI for project deployment. You will learn about Databricks Asset Bundles (DABs) and how they fit into the CI/CD process. You’ll dive into their key components, folder structure, and how they streamline deployment across various target environments in Databricks. You will also learn how to add variables, modify, validate, deploy, and execute Databricks Asset Bundles for multiple environments with different configurations using the Databricks CLI. Finally, the course introduces Visual Studio Code as an Interactive Development Environment (IDE) for building, testing, and deploying Databricks Asset Bundles locally, optimizing your development process. The course concludes with an introduction to automating deployment pipelines using GitHub Actions to enhance the CI/CD workflow with Databricks Asset Bundles. By the end of this course, you will be equipped to automate Databricks project deployments with Databricks Asset Bundles, improving efficiency through DevOps practices. Pre-requisites: Strong knowledge of the Databricks platform, including experience with Databricks Workspaces, Apache Spark, Delta Lake, the Medallion Architecture, Unity Catalog, Delta Live Tables, and Workflows. In particular, knowledge of leveraging Expectations with Lakeflow Declarative Pipelines. Labs : Yes Certification Path: Databricks Certified Data Engineer Professional

This session is repeated.Managing data and AI workloads in Databricks can be complex. Databricks Asset Bundles (DABs) simplify this by enabling declarative, Git-driven deployment workflows for notebooks, jobs, Lakeflow Declarative Pipelines, dashboards, ML models and more.Join the DABs Team for a Deep Dive and learn about:The Basics: Understanding Databricks asset bundlesDeclare, define and deploy assets, follow best practices, use templates and manage dependenciesCI/CD & Governance: Automate deployments with GitHub Actions/Azure DevOps, manage Dev vs. Prod differences, and ensure reproducibilityWhat’s new and what's coming up! AI/BI Dashboard support, Databricks Apps support, a Pythonic interface and workspace-based deploymentIf you're a data engineer, ML practitioner or platform architect, this talk will provide practical insights to improve reliability, efficiency and compliance in your Databricks workflows.

Modern insurers require agile, integrated data systems to harness AI. This framework for a global insurer uses Azure Databricks to unify legacy systems into a governed lakehouse medallion architecture (bronze/silver/gold layers), eliminating silos and enabling real-time analytics. The solution employs: Medallion architecture for incremental data quality improvement. Unity Catalog for centralized governance, row/column security, and audit compliance. Azure encryption/confidential computing for data mesh security. Automated ingestion/semantic/DevOps pipelines for scalability. By combining Databricks’ distributed infrastructure with Azure’s security, the insurer achieves regulatory compliance while enabling AI-driven innovation (e.g., underwriting, claims). The framework establishes a future-proof foundation for mergers/acquisitions (M&A) and cross-functional data products, balancing governance with agility.

Moving AI projects from pilot to production requires substantial effort for most enterprises. AI Engineering provides the foundation for enterprise delivery of AI and generative AI solutions at scale unifying DataOps, MLOps and DevOps practices. This session will highlight AI engineering best practices across these dimensions covering people, processes and technology.

Summary In this episode of the Data Engineering Podcast Chakravarthy Kotaru talks about scaling data operations through standardized platform offerings. From his roots as an Oracle developer to leading the data platform at a major online travel company, Chakravarthy shares insights on managing diverse database technologies and providing databases as a service to streamline operations. He explains how his team has transitioned from DevOps to a platform engineering approach, centralizing expertise and automating repetitive tasks with AWS Service Catalog. Join them as they discuss the challenges of migrating legacy systems, integrating AI and ML for automation, and the importance of organizational buy-in in driving data platform success.

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data managementData migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.This is a pharmaceutical Ad for Soda Data Quality. Do you suffer from chronic dashboard distrust? Are broken pipelines and silent schema changes wreaking havoc on your analytics? You may be experiencing symptoms of Undiagnosed Data Quality Syndrome — also known as UDQS. Ask your data team about Soda. With Soda Metrics Observability, you can track the health of your KPIs and metrics across the business — automatically detecting anomalies before your CEO does. It’s 70% more accurate than industry benchmarks, and the fastest in the category, analyzing 1.1 billion rows in just 64 seconds. And with Collaborative Data Contracts, engineers and business can finally agree on what “done” looks like — so you can stop fighting over column names, and start trusting your data again.Whether you’re a data engineer, analytics lead, or just someone who cries when a dashboard flatlines, Soda may be right for you. Side effects of implementing Soda may include: Increased trust in your metrics, reduced late-night Slack emergencies, spontaneous high-fives across departments, fewer meetings and less back-and-forth with business stakeholders, and in rare cases, a newfound love of data. Sign up today to get a chance to win a $1000+ custom mechanical keyboard. Visit dataengineeringpodcast.com/soda to sign up and follow Soda’s launch week. It starts June 9th.Your host is Tobias Macey and today I'm interviewing Chakri Kotaru about scaling successful data operations through standardized platform offeringsInterview IntroductionHow did you get involved in the area of data management?Can you start by outlining the different ways that you have seen teams you work with fail due to lack of structure and opinionated design?Why NoSQL?Pairing different styles of NoSQL for different problemsUseful patterns for each NoSQL style (document, column family, graph, etc.)Challenges in platform automation and scaling edge casesWhat challenges do you anticipate as a result of the new pressures as a result of AI applications?What are the most interesting, innovative, or unexpected ways that you have seen platform engineering practices applied to data systems?What are the most interesting, unexpected, or challenging lessons that you have learned while working on data platform engineering?When is NoSQL the wrong choice?What do you have planned for the future of platform principles for enabling data teams/data applications?Contact Info LinkedInParting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.init covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.Links RiakDynamoDBSQL ServerCassandraScyllaDBCAP TheoremTerraformAWS Service CatalogBlog PostThe intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Today, we’re joined by Ted Elliott, Chief Executive Officer of Copado, the leader in AI-powered DevOps for business applications. We talk about:  Impacts of AI agents over the next 5 yearsTed’s AI-generated Dr. Seuss book based on walks with his dogThe power of small data with AI, despite many believing more data is the answerThe challenge of being disciplined to enter only good dataGaming out SaaS company ideas with AI, such as a virtual venture capitalist

Moving AI projects from pilot to production requires substantial effort for most enterprises. AI Engineering provides the foundation for enterprise delivery of AI and generative AI solutions at scale unifying DataOps, MLOps and DevOps practices. This session will highlight AI engineering best practices across these dimensions covering people, processes and technology.

Do you feel frustrated when you put in a lot of effort to complete a task, only for no one to be happy with the result? The tester says it's wrong, and the product manager insists it should be supported. What's wrong with me? Nothing — the challenge isn't just individual effort, but how development and QA work together. Developers and testers can influence these processes, not just managers. We just need to know what options we have. Alex will share communication strategies from his experience working with DevOps, QA, and developers, navigating informational chaos, and introducing new ideas.

Learn how to evaluate and optimize the impact of AI-assisted software development with Gemini Code Assist. This session covers processes for measuring AI-assistance effectiveness, exploring quantitative and qualitative measures available with Gemini Code Assist, and integrating with Cloud Monitoring and Cloud Logging. Discover how to leverage DevOps Research and Assessment (DORA) metrics to track productivity gains. Whether you’re a developer, team lead, architect, or IT manager, you’ll gain insights into measuring the impact of AI assistance.

Are you a DevOps engineer or site reliability engineer (SRE) tasked with keeping mission-critical applications running 24/7? What if AI could help you detect, diagnose, and resolve incidents faster than ever before? Join the session to learn how to use AI assistance to diagnose and troubleshoot incidences and improve the mean time to detect (MTTD) and mean time to repair (MTTR). Charles Schwab partnered with Google Cloud on exploring the capabilities of Gemini Cloud Assist. And in this session, they’ll share their firsthand experiences testing Gemini Cloud Assist to enhance the reliability of their login application that handles millions of logins daily. Get practical AI skills and tips that you can put into your job right away.

Platform Engineering promises to be the next evolutionary step beyond DevOps and SRE by reducing complexity and unlocking efficiency by shifting tasks down into the platform. Join us and connect with fellow experts in DevOps, SRE, and the Platform Engineering community to discuss essential new tools and principles needed for successful digital transformations in 2025. Gain valuable advice, guidance, and networking opportunities.

Presenting “one network,” a novel approach to simplify service integration and increase compliance across heterogeneous environments. Learn how using open source proxies provides a policy-oriented network architecture for consistent policy enforcement, service assembly, and secure-by-default deployments. One network also provides role separation across DevOps, SecOps, site reliability engineers, and app developers, streamlining operations and increasing efficiency. Attend this session to discover how one network can transform your approach to service delivery.

Supercharge your team’s developer experience (DevEx) with DevOps Research and Assessment (DORA) reports and AI. This session reveals how high-performing teams leverage DORA insights and AI to enhance their software development life cycle. Learn to boost code quality and delivery performance, with real-world examples of teams that have improved innovation and agility. Discover how to apply AI for actionable insights, and get practical strategies to build a high-performing team that gets better at getting better.

Gaining visibility and reducing risks across multi-cloud environments has historically meant sifting through an increasing number of disjointed tools across DevOps, Cloud Security, and the SOC. A scalable, unified approach is needed for visibility across multi-cloud environments, including where AI is being used, and which data AI can access. Join Wiz as we discuss how Google Cloud and Wiz are better together to meet these challenges through the adoption of a Cloud-Native Application Protection Platform (CNAPP).

This Session is hosted by a Google Cloud Next Sponsor.
Visit your registration profile at g.co/cloudnext to opt out of sharing your contact information with the sponsor hosting this session.

Ready to take your Day 2 operations to the next level? This session is your one-stop shop for mastering Day 2 ops in 2025. We’ll cover everything you need to know, such as leveraging observability and AI agents for DevOps use cases, building application-centric observability, and optimizing for cost and utilization on Google Cloud. This roundup session explores a variety of products and updates in the Day 2 space, giving you the tools and knowledge to supercharge your operational efficiency and drive innovation.

Calling all developers, DevOps, and SRE engineers: Join us to discuss observability best practices for Gen AI apps on Google Cloud. We'll explore tracing, monitoring, and logging for Gen AI. Share insights on TPU/GPU monitoring, log correlation, regulatory alignment, and data-driven insights to enhance your Gen AI deployments.

Data DevOps applies rigorous software development practices—such as version control, automated testing, and governance—to data workflows, empowering software engineers to proactively manage data changes and address data-related issues directly within application code. By adopting a "shift left" approach with Data DevOps, SWE teams become more aware of data requirements, dependencies, and expectations early in the software development lifecycle, significantly reducing risks, improving data quality, and enhancing collaboration.

This session will provide practical strategies for integrating Data DevOps into application development, enabling teams to build more robust data products and accelerate adoption of production AI systems.

CockroachDB is the distributed SQL database that handles the demands of today's data-driven applications. The second edition of this popular hands-on guide shows software developers, architects, and DevOps/SRE teams how to use CockroachDB for applications that scale elastically and provide seamless delivery for end users while remaining indestructible. Data professionals will learn how to migrate existing applications to CockroachDB's performant, cloud-native data architecture. You'll also quickly discover the benefits of strong data correctness and consistency guarantees, plus optimizations for delivering ultra-low latencies to globally distributed end users. Uncover the power of distributed SQL Learn how to start, manage, and optimize projects in CockroachDB Explore best practices for data modeling, schema design, and distributed infrastructure Discover strategies for migrating data into CockroachDB See how to read, write, and run ACID transactions across distributed systems Maximize resiliency in multiregion clusters Secure, monitor, and fine-tune your CockroachDB deployment for peak performance