talk-data.com

All of the product innovation and open source project updates Databricks announced at Data + AI Summit–in less than 5 min.

At this year's event, over 250 customers shared their data and AI journies. They showcased a wide variety of use cases, best practices and lessons from their leadership and innovation with the latest data and AI technologies.

See how enterprises are leveraging generative AI in their data operations and how innovative data management and data governance are fueling organizations as they race to develop GenAI applications. https://www.databricks.com/blog/how-real-world-enterprises-are-leveraging-generative-ai

To see more real-world use cases and customer success stories, visit: https://www.databricks.com/customers

Reynold Xin, Co-founder and Chief Architect, Databricks shares the latest innovation coming out of the Apache Spark™ open source project including a preview of the anticipated release of Spark 4.0

Speakers: Reynold Xin, Co-founder and Chief Architect, Databricks Tareef Kawaf, President, Posit Sofware, PBC

Shant Hovsepian, Chief Technology Officer of Data Warehousing at Databricks explains why Delta Lake is the most adopted open lakehouse format.

Includes: - Delta Lake UniForm GA (support for and compatibility with Hudi, Apache Iceberg, Delta) - Delta Lake Liquid Clustering - Delta Lake production-ready catalog (Iceberg REST API) - The growth and strength of the Delta ecosystem - Delta Kernel - DuckDB integration with Delta - Delta 4.0

Shant Hovsepian, CTO of Data Warehousing at Databricks announced the biggest Delta Lake release to date, Delta 4.0, during the Data + AI Summit 2024 in San Francisco.

Speaker: Shant Hovsepian, Chief Technology Officer of Data Warehousing, Databricks

Speaker: Matei Zaharia, Original Creator of Apache Spark™ and MLflow; Chief Technologist, Databricks

Matei Zaharia, Original Creator of Apache Spark™ and MLflow and Chief Technologist at Databricks open sourced Unity Catalog live onstage at the Data + AI Summit 2024 in San Francisco.

Hannes Mühleisen of DuckDB Labs addressed an audience of thousands during his keynote address at Data + AI Summit 2024 in San Francisco. Mühleisen announced DuckDB support for Delta Lake, a new DuckDB Extension to Unity Catalog, and Community Extensions.

Speaker: Hannes Mühleisen, Creator of DuckDB, DuckDB Labs @duckdb @duckdb3282

Speaker: Bilal Aslam, Sr. Director of Product Management, Databricks

Bilal explains that everything starts with good data and outlines the three steps to good data including, ingesting, transforming and orchestrating your data. Then Bilal announces Databricks LakeFlow - a unified solution for data engineering. With LakeFlow you can ingest data from databases, enterprise apps and cloud sources, transform it in batch and real-time streaming, and confidently deploy and operate in production. Includes a live demo of Databricks LakeFlow.

To learn more about Databricks LakeFlow, see the announcement blog post: https://www.databricks.com/blog/introducing-databricks-lakeflow

Ali Ghodsi, Co-founder and CEO of Databricks closes the 2024 Data + AI Summit with a recap of Databricks and open source innovation announced during the 4-day conference in San Francisco.

Speaker: Ali Ghodsi, Co-founder and CEO, Databricks @Databricks

Speakers: Matei Zaharia, Original Creator of Apache Spark™ and MLflow; Chief Technologist, Databricks Darshana Sivakumar, Staff Product Manager, Databricks

Organizations are looking for ways to securely exchange their data and collaborate with external partners to foster data-driven innovations. In the past, organizations had limited data sharing solutions, relinquishing control over how their sensitive data was shared with partners and little to no visibility into how their data was consumed. This created the risk for potential data misuse and data privacy breaches. Customers who tried using other clean room solutions have told us these solutions are limited and do not meet their needs, as they often require all parties to copy their data into the same platform, do not allow sophisticated analysis beyond basic SQL queries, and have limited visibility or control over their data.

Organizations need an open, flexible, and privacy-safe way to collaborate on data, and Databricks Clean Rooms meets these critical needs.

See a demo of Databricks Clean Rooms, now in Public Preview on AWS + Azure

Speaker: Matei Zaharia, Original Creator of Apache Spark™ and MLflow; Chief Technologist, Databricks

Summary: Data sharing and collaboration are important aspects of the data space. Matei Zaharia explains the evolution of the Databricks data platform to facilitate data sharing and collaboration for customers and their partners.

Delta Sharing allows you to share parts of your table with third parties authorized to view them. Over 16,000 data recipients use Delta Sharing, and 40% are not on Databricks—a testament to the open nature.

Databricks Marketplace has been growing rapidly and now has over 2,000 data listings, making it one of the largest data marketplaces available. New Marketplace partners include T-Mobile, Tableau, Atlassian, Epsilon, Shutterstock and more.

To learn more about Delta Sharing features and the expansion of partner sharing ecosystem, see the recent blog: https://www.databricks.com/blog/whats-new-data-sharing-and-collaboration

Speaker: Matei Zaharia, Original Creator of Apache Spark™ and MLflow; Chief Technologist, Databricks

Reynold Xin explains the evolution of Apache Spark™, outlining several historical challenges and how the Spark community worked to make improvements, including the addition of PySpark.

Speaker: Reynold Xin, Co-founder and Chief Architect at Databricks

Speaker: Tareef Kawaf, President, Posit Sofware, PBC

Get open access to your data no matter where it resides while applying unified governance - see the latest features of Unity Catalog.

Speaker: Zeashan Pappa, Staff Product Manager, Databricks

Databricks Co-founder and Chief Architect, Reynold Xin, on the evolution of Apache Spark™ and what's next, including Spark Connect and a preview of Apache Spark™ 4.0

Speaker: Reynold Xin, Co-founder and Chief Architect, Databricks

Speaker: Alexander Booth, Assistant Director of R&D, Texas Rangers Baseball Club

Shant Hovsepian, Chief Technology Officer of Data Warehousing at Databricks, discusses the UniForm data format and its interoperability with other data formats. Shant explains that Delta Lake is the most adopted open lakehouse format.

Speaker: Shant Hovsepian, Chief Technology Officer of Data Warehousing, Databricks

Speakers: Ali Ghodsi, Co-founder and CEO, Databricks Ryan Blue, Creator of Apache Iceberg and co-founder of Tabular