Ensuring the operational excellence of AI agents in production requires robust monitoring capabilities that span both performance metrics and quality evaluation. This session explores Databricks' comprehensive Mosaic Agent Monitoring solution, designed to provide visibility into deployed AI agents through an intuitive dashboard that tracks critical operational metrics and quality indicators. We'll demonstrate how to use the Agent Monitoring solution to iteratively improve a production agent that delivers a better customer support experience while decreasing the cost of delivering customer support. We will show how to: Identify and proactively fix a quality problem with the GenAI agent’s response before it becomes a major issue. Understand user’s usage patterns and implement/test an feature improvement to the GenAI agent Key session takeaways include: Techniques for monitoring essential operational metrics, including request volume, latency, errors, and cost efficiency across your AI agent deployments Strategies for implementing continuous quality evaluation using AI judges that assess correctness, guideline adherence, and safety without requiring ground truth labels Best practices for setting up effective monitoring dashboards that enable dimension-based analysis across time periods, user feedback, and topic categories Methods for collecting and integrating end-user feedback to create a closed-loop system that drives iterative improvement of your AI agents

As a leading personalized product retailer, Shutterfly needed a modern, secure, and performant data foundation to power GenAI-driven customer experiences. However, their existing stack was creating roadblocks in performance, governance, and machine learning scalability. In partnership with Impetus, Shutterfly embarked on a multi-phase migration to Databricks Unity Catalog. This transformation not only accelerated Shutterfly’s ability to provide AI-driven personalization at scale but also improved governance, reduced operational overhead, and laid a scalable foundation for GenAI innovation. Join experts from Databricks, Impetus, and Shutterfly to discover how this collaboration enabled faster data-driven decision-making, simplified compliance, and unlocked the agility needed to meet evolving customer demands in the GenAI era. Learn from their journey and take away best practices for your own modernization efforts.

SMBC, a major Japanese multinational financial services institution, has embarked on an initiative to build a GenAI-powered, modern and well-governed cloud data platform on Azure/Databricks. This initiative aims to build an enterprise data foundation encompassing loans, deposits, securities, derivatives, and other data domains. Its primary goals are: To decommission legacy data platforms and reduce data sprawl by migrating 20+ core banking systems to a multi-tenant Azure Databricks architecture To leverage Databrick’s delta-share capabilities to address SMBC’s unique global footprint and data sharing needs To govern data by design using Unity Catalog To achieve global adoption of the frameworks, accelerators, architecture and tool stack to support similar implementations across EMEA Deloitte and SMBC leveraged the Brickbuilder asset “Data as a Service for Banking” to accelerate this highly strategic transformation.

In this session you will learn how to leverage a wide set of GenAI models in Databricks, including external connections to cloud vendors and other model providers. We will cover establishing connection to externally served models, via Mosaic AI Gateway. This will showcase connection to Azure, AWS & Google Cloud models, as well as model vendors like Anthropic, Cohere, AI21 Labs and more. You will also discover best practices on model comparison, governance and cost control on those model deployments.

Discover how Tahoe Therapeutics (formerly Vevo) is generating gigascale single-cell data that map how drugs interact with cells from cancer patients. They are using that to find better therapeutics, and to build AI models that can predict drug-patient interactions on Databricks. Their technology enabled the landmark Tahoe-100M atlas, the world’s largest dataset of drug responses-profiling 100 million cells across 60,000 conditions. Learn how we use Databricks to process this massive data, enabling AI models that predict drug efficacy and resistance at the cellular level. Recognized as the Grand Prize Winner of the Databricks Generative AI Startup Challenge, Tahoe sets a new standard for scalable, data-driven drug discovery.

Our project demonstrates building enterprise AI systems cost-effectively, focusing on matching merchant descriptors to known businesses. Using fine-tuned LLMs and advanced search, we created a solution rivaling alternatives at minimal cost. The system works in three steps: A fine-tuned Llama 3 8B model parses merchant descriptors into standardized components. A hybrid search system uses these components to find candidate matches in our database. A Llama 3 70B model then evaluates top candidates, with an AI judge reviewing results for hallucination. We achieved a 400% latency improvement while maintaining accuracy and keeping costs low and each fine-tuning round cost hundreds of dollars. Through careful optimization and simple architecture for a balance between cost, speed and accuracy, we show that small teams with modest budgets can tackle complex problems effectively using this technology. We share key insights on prompt engineering, fine-tuning and cost and latency management.

Cognizant developed a GenAI-driven market intelligence chatbot for RJR using Dash UI. This chatbot leverages Databricks Vector Search for vector embeddings and semantic search, along with the DBRX-Instruct LLM model to provide accurate and contextually relevant responses to user queries. The implementation involved loading prepared metadata into a Databricks vector database using the GTE model to create vector embeddings, indexing these embeddings for efficient semantic search, and integrating the DBRX-Instruct LLM into the chat system with prompts to guide the LLM in understanding and responding to user queries. The chatbot also generated responses containing URL links to dashboards with requested numerical values, enhancing user experience and productivity by reducing report navigation and discovery time by 30%. This project stands out due to its innovative AI application, advanced reasoning techniques, user-friendly interface, and seamless integration with MicroStrategy.

Crisis Text Line has been innovating for ten years in text-based mental health crisis intervention and is now leading the next wave of GenAI use cases in the space. With over 300 million messages exchanged since 2013 and a decade of expertise, Crisis Text Line is unlocking the potential of AI to amplify human connection at a global scale.We will discuss how we leveraged our bedrock application to co-navigate crisis care through a set of early AI agent workflows. First, a simulator that reproduces texter behavior to train responders in taking conversations ranging in difficulty where the texter is in imminent risk of suicide or self-harm. Second, a tool that automatically monitors clinical quality of conversations. Third, predicted summarization to capture key context before conversations are transferred. Through the power of suggestion, this compound system aims to reduce burden and drive efficiency, such that our responders can focus on what they do best — support people in need.

This talk dives into leveraging GenAI to scale sophisticated decision intelligence. Learn how an AI copilot interface simplifies running complex Bayesian probabilistic models, accelerating insight generation, and accurate decision making at the enterprise level. We talk through techniques for deploying AI agents at scale to simulate market dynamics or product feature impacts, providing robust, data-driven foresight for high-stakes innovation and strategy directly within your Databricks environment. For marketing teams, this approach will help you leverage autonomous AI agents to dynamically manage media channel allocation while simulating real-world consumer behavior through synthetic testing environments.

Join industry pioneers Boeing and CARIAD (Volkswagen Group) as they showcase how advanced data platforms are revolutionizing mobility across air and ground transportation. Boeing's Jeppesen Smart NOTAMs system demonstrates the power of compound AI in aviation safety, processing over 4.5M critical flight notices annually and serving 75% of commercial aviation through an innovative combination of MLflow, GenAI, and Delta Sharing technologies. CARIAD follows with insights into their groundbreaking Unified Data Ecosystem (UDE), the singular data platform powering Volkswagen Group's global mobility transformation across all brands and markets. Together, these leaders illustrate how smart data architecture is building the foundation for the future of transportation, from the skies to the streets.

In the complex world of logistics, efficiency and accuracy are paramount. At Pilot, the largest travel center network in North America, managing fuel delivery operations was a time-intensive and error-prone process. Tasks like processing delivery records and validating fuel transaction data posed significant challenges due to the diverse formats and handwritten elements involved. After several attempts to use robotic process automation failed, the team turned to Generative AI to automate and streamline this critical business process. In this session, discover how Pilot leverages GenAI, powered by advanced text and vision models, to revolutionize BOL processing. By implementing few-shot learning and vectorized examples, the data team at Pilot was able to increase document parsing accuracy from 70% to 95%, enabling real-time validation against truck driver inputs, which has resulted in millions of savings from accelerating credit reconciliation and improved financial operations.

DSPy is a framework for authoring GenAI applications with automatic prompt optimization, while MLflow provides powerful MLOps tooling to track, monitor, and productize machine learning workflows. In this lightning talk, we demonstrate how to integrate MLflow with DSPy to bring full observability to your DSPy development. We’ll walk through how to track DSPy module calls, evaluations, and optimizers using MLflow’s tracing and autologging capabilities. By the end, you'll see how combining these two tools makes it easier to debug, iterate, and understand your DSPy workflows, then deploy your DSPy program — end to end.

Are data teams ready for AI? Prophecy’s exclusive survey, “The Impact of GenAI on Data Teams”, gives the clearest picture yet of GenAI’s potential in data management, and what’s standing in the way. The top two obstacles? Poor governance and slow access to high-quality data. The message is clear: Modernizing your data platform with Databricks is essential. But it’s only the beginning. To unlock the power of AI and analytics, organizations must deliver governed, self-service access to clean, trusted data. Traditional data prep tools introduce risks around security, quality, and cost. It’s no wonder data leaders cited data transformation as the area where GenAI will make the biggest impact. To deliver what’s needed teams need a shift to governed self-service. Data analysts and scientists move fast while staying within IT’s guardrails. Join us to learn more details from the survey and how leading organizations are ahead of the curve, using GenAI to reshape how data gets done.

GenAI systems are evolving beyond basic information retrieval and question answering, becoming sophisticated agents capable of managing multi-turn dialogues and executing complex, multi-step tasks autonomously. However, reliably evaluating and systematically improving their performance remains challenging. In this session, we'll explore methods for assessing the behavior of LLM-driven agentic systems, highlighting techniques and showcasing actionable insights to identify performance bottlenecks and to creating better-aligned, more reliable agentic AI systems.

The ability for different AI systems to collaborate is more critical than ever. From traditional ML development to fine-tuning GenAI models, Databricks delivers the stability, cost-optimization and productivity Expedia Group (EG) needs. Learn how to unlock the full potential of AI interoperability with Databricks. AI acceleration: Discover how Databricks acts as a central hub, helping to scale AI model training and prediction generation to deliver high-quality insights for customers. Cross-platform synergy: Learn how EG seamlessly integrated Databricks' powerful features into its ecosystem, streamlining workflows and accelerating time to market. Scalable deployment: Understand how Databricks stability and reliability increased efficiency in prototyping and running scalable production workloads. Join Shiyi Pickrell to understand the future of AI interoperability, how it’s generating business value and driving the next generation of travel AI-powered experiences.

Databricks Financial Service customers in the GenAI space have a common use case of ingestion and processing of unstructured documents — PDF/images — then performing downstream GenAI tasks such as entity extraction and RAG based knowledge Q&A. The pain points for the customers for these types of use cases are: The quality of the PDF/image documents varies since many older physical documents were scanned into electronic form The complexity of the PDF/image documents varies and many contain tables — images with embedding information — which require slower Tesseract OCR They would like to streamline postprocess for downstream workloads In this talk we will present an optimized structured streaming workflow for complex PDF ingestion. The key techniques include Apache Spark™ optimization, multi-threading, PDF object extraction, skew handling and auto retry logics

Join this deep dive session for practitioners on Unity Catalog, Databricks’ unified data governance solution, to explore its capabilities for managing data and AI assets across workflows. Unity Catalog provides fine-grained access control, automated lineage tracking, quality monitoring and policy enforcement and observability at scale. Whether your focus is data pipelines, analytics or machine learning and generative AI workflows, this session offers actionable insights on leveraging Unity Catalog’s open interoperability across tools and platforms to boost productivity and drive innovation. Learn governance best practices, including catalog configurations, access strategies for collaboration and controls for securing sensitive data. Additionally, discover how to design effective multi-cloud and multi-region deployments to ensure global compliance.

Nondeterministic AI models, like large language models (LLMs), offer immense creative potential but require new approaches to testing and scalability. Drawing from her experience running New York Times-featured Generative AI comedy shows, Erin uncovers how traditional benchmarks may fall short and how embracing unpredictability can lead to innovative, laugh-inducing results. This talk will explore methods like multi-tiered feedback loops, chaos testing and exploratory user testing, where AI outputs are evaluated not by rigid accuracy standards but by their adaptability and resonance across different contexts — from comedy generation to functional applications. Erin will emphasize the importance of establishing a root source of truth — a reliable dataset or core principle — to manage consistency while embracing creativity. Whether you’re looking to generate a few laughs of your own or explore creative uses of Generative AI, this talk will inspire and delight enthusiasts of all levels.

Nestlé USA, a division of the world’s largest food and beverage company, Nestlé S.A., has embarked on a transformative journey to unlock GenAI capabilities on their data platform. Deloitte, Databricks, and Nestlé have collaborated on a data platform modernization program to address gaps associated with Nestlé’s existing data platform. This joint effort introduces new possibilities and capabilities, ranging from development of advanced machine learning models, implementing Unity Catalog, and adopting Lakehouse Federation, all while adhering to confidentiality protocols. With help from Deloitte and Databricks, Nestlé USA is now able to meet its advanced enterprise analytics and AI needs with the Databricks Data Intelligence Platform.

In the rapidly evolving landscape of pharmaceuticals, the integration of AI and GenAI is transforming how organizations operate and deliver value. We will explore the profound impact of the AI program at Takeda Pharmaceuticals and the central role of Databricks. We will delve into eight pivotal AI/GenAI use cases that enhance operational efficiency across commercial, R&D, manufacturing, and back-office functions, including these capabilities: Responsible AI Guardrails: Scanners that validate and enforce responsible AI controls on GenAI solutions Reusable Databricks Native Vectorization Pipeline: A scalable solution enhancing data processing with quality and governance One-Click Deployable RAG Pattern: Simplifying deployment for AI applications, enabling rapid experimentation and innovation AI Asset Registry: A repository for foundational models, vector stores, and APIs, promoting reuse and collaboration