You know AKS—but environment variables, secrets, and ingress controllers still feel like dev-speak? This demo-heavy session unpacks cloud-native development from an IT/Ops lens. Learn how app design impacts operations, what Day 2 AKS management really looks like, and how to confidently architect alongside your developer peers.

Managing modern Kubernetes environments requires you to act as soon as events take place. In this session, see how open source projects Drasi, Inspektor Gadget, and Headlamp simplify monitoring and management of Azure Kubernetes Service (AKS) clusters. Learn how to surface live state changes, deep runtime insights and leverage an intuitive management interface. Demonstrated on Azure Kubernetes Service (AKS) and applicable to any Kubernetes setup, this session shows how to bring a universal, cloud-agnostic approach to your cluster monitoring and management.

As Kubernetes and AI adoption grow, managing your clusters at scale becomes a strategic imperative. We'll share practical lessons from operating large clusters with AKS, including how to keep things reliable, efficient & secure. Learn about tools like Azure Kubernetes Fleet Manager, smart scheduling for AI, and ways to simplify multi-cluster ops. Whether you are scaling to thousands of nodes or fine-tuning for performance, you'll leave with practical tips to improve cluster ops.

Learn practical strategies from Sam’s Club for modernizing mission-critical apps to Azure. Dive into technical strategies, get hands-on guidance to implement key resilience patterns and how to operationalize them in Azure Kubernetes Service (AKS), Azure SQL, Cosmos DB, and other critical Azure services. By the end of this session, developers and cloud practitioners will walk away with tools to modernize legacy monolithic apps for resilience, performance, and scale in Azure.

Trusted by nearly half of Fortune 100 companies, Nasdaq Boardvantage powers secure, intelligent board operations. In this deep dive session, explore how Azure Database for PostgreSQL and MySQL, Microsoft Foundry, Azure Kubernetes Service (AKS), and API Management create a resilient architecture that safeguards confidential data while unlocking new agentic AI capabilities.

Azure Linux Container Host is an operating system image that's optimized for running container workloads on Azure Kubernetes Service (AKS). Learn how it improves performance, simplifies node lifecycle management, and enables innovations like pod sandboxing and OS Guard. See how customers use Azure Linux for workload isolation and hardened security with features like Integrity Policy Enforcement, SELinux, and dm-verity—helping enterprises modernize and scale with confidence.

Kubernetes unlocks powerful capabilities for modern apps—but can be complex. AKS Automatic changes this. In this session, we’ll introduce AKS Automatic and how it helps accelerate cloud-native adoption without the operational overhead. Whether you're modernizing or building new apps, AKS Automatic offers a streamlined path forward, delivering production-ready clusters out of the box, automating infrastructure operations, and embedding best practices for security, scalability, and performance.

Cloud-native architectures are transforming app development, and Azure’s infrastructure drives this evolution. This session dives into services that help develop and deploy resilient, scalable cloud-native solutions—with real customer insights and field-tested guidance. Learn how VMSS handles stateless workloads, Azure Storage and Container Networking optimize performance and cost, and how Kubernetes and other OSS thrive on Azure with enterprise-grade reliability.

Delivered in a silent stage breakout.

In the world of AI voice agents, especially in sensitive contexts like healthcare, audio clarity is everything. Background noise—a barking dog, a TV, street sounds—degrades transcription accuracy, leading to slower, clunkier, and less reliable AI responses. But how do you solve this in real-time without breaking the bank?

This talk chronicles our journey at a health-tech startup to ship background noise filtration at scale. We'll start with the core principles of noise reduction and our initial experiments with open-source models, then dive deep into the engineering architecture required to scale a compute-hungry ML service using Python and Kubernetes. You'll learn about the practical, operational considerations of deploying third-party models and, most importantly, how to measure their true impact on the product.

Kubeflow is a platform for building and deploying portable and scalable machine learning (ML) workflows using containers on Kubernetes-based systems.

We will code together a simple Kubeflow pipeline, show how to test it locally. As a bonus, we will explore one solution to avoid dependency hell using the modern dependency management tool uv.

If we want to run data science workloads (e.g. using Tensorflow, PyTorch, and others) in containers (for local development or production on Kubernetes), we need to build container images. Doing that with a Dockerfile is fairly straightforward, but is it the best method? In this talk, we'll take a well-known speech-to-text model (Whisper) and show various ways to run it in containers, comparing the outcomes in terms of image size and build time.

Building and deploying scalable, reproducible machine learning pipelines can be challenging, especially when working with orchestration tools like Slurm or Kubernetes. In this talk, we demonstrate how to create an end-to-end ML pipeline for anomaly detection in International Space Station (ISS) telemetry data using only Python code.

We show how Kubeflow Pipelines, MLFlow, and other open-source tools enable the seamless orchestration of critical steps: distributed preprocessing with Dask, hyperparameter optimization with Katib, distributed training with PyTorch Operator, experiment tracking and monitoring with MLFlow, and scalable model serving with KServe. All these steps are integrated into a holistic Kubeflow pipeline.

By leveraging Kubeflow's Python SDK, we simplify the complexities of Kubernetes configurations while achieving scalable, maintainable, and reproducible pipelines. This session provides practical insights, real-world challenges, and best practices, demonstrating how Python-first workflows empower data scientists to focus on machine learning development rather than infrastructure.

In modern data infrastructure, efficient and scalable data governance is essential for ensuring security, compliance, and accessibility. This session explores how to deploy Unity Catalog OSS on Kubernetes, leveraging its cloud-agnostic nature and efficient resource management. Helm makes Unity Catalog deployment simple and easy by providing a simplified installation process, easy configuration and credentials management.The session will cover why Kubernetes is the ideal platform, provide a technical breakdown of Unity Catalog on Kubernetes, and include a live showcase of its seamless deployment process. By the end, participants will confidently configure and deploy Unity Catalog OSS in their preferred Kubernetes environment and integrate it into their existing infrastructure.

Big Data Europe Onsite and online on 22-25 November in 2022 Learn more about the conference: https://bit.ly/3BlUk9q

Join our next Big Data Europe conference on 22-25 November in 2022 where you will be able to learn from global experts giving technical talks and hand-on workshops in the fields of Big Data, High Load, Data Science, Machine Learning and AI. This time, the conference will be held in a hybrid setting allowing you to attend workshops and listen to expert talks on-site or online.

The future of app development is at the intersection of AI and cloud-native technologies like Kubernetes. Whether you’re a Dev team using generative AI or an Ops teams balancing innovation with security, compliance, and cost, Azure has the tools to help you succeed. Discover how: 1. Cutting-edge features in Azure Kubernetes Service, Azure Functions, & Azure Container Apps help seamlessly bring your intelligent apps to production. 2. AI assistance built into Azure empowers Dev and Ops to scale.

To learn more, please check out these resources: * https://aka.ms/Ignite23CollectionsBRK208H * https://info.microsoft.com/ww-landing-contact-me-for-events-m365-in-person-events.html?LCID=en-us&ls=407628-contactme-formfill * https://aka.ms/azure-ignite2023-dataaiblog

𝗦𝗽𝗲𝗮𝗸𝗲𝗿𝘀: * Devanshi Joshi * Jorge Palma * Kamala Dasika * Daria Grigoriu * Tara E Walker * Ed Donahue * Nate Ceres * Simon Jakesch * Thiago Almeida

𝗦𝗲𝘀𝘀𝗶𝗼𝗻 𝗜𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻: This video is one of many sessions delivered for the Microsoft Ignite 2023 event. View sessions on-demand and learn more about Microsoft Ignite at https://ignite.microsoft.com

BRK208H | English (US) | AI & Apps

MSIgnite

ABOUT THE TALK: In this talk, Erik Bernhardsson will share how Modal starts 1000s of large containers in seconds, and what they had to do under the surface to build this. This includes a custom file system written in Rust, their own container runtime, and their own container image builder. This talk will give you an idea of how containers work along with some of the low-level Linux details underneath. We'll also talk about many infrastructure tools hold data teams back, and why they deserve faster and better tools.

ABOUT THE SPEAKER: Erik Bernhardsson is the founder and CEO of Modal, which is an infrastructure provider for data teams. Before Modal, Erik was the CTO at Better for six years, and previously spent seven years at Spotify, building the music recommendation system and running data teams.

ABOUT DATA COUNCIL: Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers.

Make sure to subscribe to our channel for the most up-to-date talks from technical professionals on data related topics including data infrastructure, data engineering, ML systems, analytics and AI from top startups and tech companies.

FOLLOW DATA COUNCIL: Twitter: https://twitter.com/DataCouncilAI LinkedIn: https://www.linkedin.com/company/datacouncil-ai/

Building data pipelines is difficult—and adding a data governance and observability framework doesn’t make it any easier. But that was the task ahead for Deepak Konidena during his early days at Zillow. In this session, he’ll share how the platform they build on top of dbt, Airflow, Spark, and Kubernetes—ZSQL—eliminated the need for internal data teams to build their own DAGs, models, schemas and lineage from scratch, while also providing an easy way to enforce data quality, monitor changes, and alert on disruptions.

Check the slides here: https://docs.google.com/presentation/d/18HEil3_nXD8nYBhcg4m-Kpy8I8Na6MXI/edit?usp=sharing&ouid=110293204340061069659&rtpof=true&sd=true

Coalesce 2023 is coming! Register for free at https://coalesce.getdbt.com/.

How do you train and enable 20 data analysts to use dbt Core in a short amount of time?

At M1, engineering and analytics are far apart on the org chart, but work hand-in-hand every day. M1 engineering has a culture that celebrates open source, where every data engineer is trained and empowered to work all the way down the infrastructure stack, using tools like Terraform and Kubernetes. The analytics team is comprised of strong SQL writers who use Tableau to create visualizations used company wide. When M1 knew they needed a tool like dbt for change management and data documentation generation, they had to figure out how to bridge the gap between engineering and analytics to enable analysts to contribute with minimal engineering intervention. Join Kelly Wachtel, a senior data engineer at M1, explain how they trained about 20 analysts to use git and dbt Core over the past year, and strengthened their collaboration between their data engineering and analytics teams.

Check the slides here: https://docs.google.com/presentation/d/1CWI97EMyLIz6tptLPKt4VuMjJzV_X3oO/edit?usp=sharing&ouid=110293204340061069659&rtpof=true&sd=true

Coalesce 2023 is coming! Register for free at https://coalesce.getdbt.com/.