talk-data.com talk-data.com

Andrew Shieh

Speaker

Andrew Shieh

2

talks

Software Engineer Databricks

Andrew Shieh is a software engineer at Databricks on the AI Serving team. He joined Databricks in 2022 and currently works on developing Foundation Model API products. He is interested in scalable AI, distributed systems, and the outdoors.

Bio from: Data + AI Summit 2025

Filter by Event / Source

Talks & appearances

2 activities · Newest first

Search activities →
Scaling Generative AI: Batch Inference Strategies for Foundation Models

Curious how to apply resource-intensive generative AI models across massive datasets without breaking the bank? This session reveals efficient batch inference strategies for foundation models on Databricks. Learn how to architect scalable pipelines that process large volumes of data through LLMs, text-to-image models and other generative AI systems while optimizing for throughput, cost and quality. Key takeaways: Implementing efficient batch processing patterns for foundation models using AI functions Optimizing token usage and prompt engineering for high-volume inference Balancing compute resources between CPU preprocessing and GPU inference Techniques for parallel processing and chunking large datasets through generative models Managing model weights and memory requirements across distributed inference tasks You'll discover how to process any scale of data through your generative AI models efficiently.

talk
with Jonathan Hsieh (LanceDB) , Cathy Yin (Databricks) , Andrew Shieh (Databricks) , Ziyi Yang (Databricks) , Andy Konwinski (Databricks) , Denny Lee (Databricks) , Asfandyar Qureshi (Databricks) , Yuki Watanabe (Databricks) , Brandon Cui (Databricks) , Andrew Drozdov (Databricks) , Anand Kannappan (Patronus AI) , Harsh Panchal (Databricks) , Tomu Hirata (Databricks) , Daya Khudia (Databricks) , Jose Javier Gonzalez (Databricks) , Jasmine Collins (Databricks) , MAHESWARAN SATHIAMOORTHY (Bespoke Labs) , Jonathan Chang (Databricks) , Matei Zaharia (Databricks) , Alexander Trott (Databricks) , Tejas Sundaresan (Databricks) , Pallavi Koppol (Databricks) , Jonathan Frankle (Databricks) , Erich Elsen (Databricks) , Ivan Zhou (Databricks) , Davis Blalock , Gayathri Murali (META)

https://bit.ly/devconnectdais