Amazon Aurora is a serverless relational database with unparalleled high performance and availability at global scale for PostgreSQL, MySQL, and DSQL. Aurora provides managed high availability (HA) and disaster recovery (DR) capabilities in and across AWS Regions. In this session, explore the Aurora HA and DR capabilities and discover design patterns that enable the development of resilient applications. Learn how to establish in-Region and cross-Region HA and DR using Aurora features including Multi-AZ deployments and Aurora Global Database, and how Aurora DSQL multi-Region clusters provides the highest level of availability and application resilience.

Learn more: More AWS events: https://go.aws/3kss9CP

Subscribe: More AWS videos: http://bit.ly/2O3zS75 More AWS events videos: http://bit.ly/316g9t4

ABOUT AWS: Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts. AWS is the world's most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

AWSreInvent #AWSreInvent2025 #AWS

With an innovative architecture that decouples compute from storage and advanced features like Amazon Aurora Global Database and low-latency read replicas, Aurora reimagines what it means to be a relational database. Aurora is a built-for-the-cloud, serverless relational database service that delivers unparalleled performance and availability at global scale for MySQL, PostgreSQL, and DSQL. In this session, dive deep into new features – and Aurora’s most popular offerings including serverless, I/O-Optimized, zero-ETL integrations, MCP integration, and generative AI support for vector search and storage. Also learn about the groundbreaking Aurora DSQL engine.

Learn more: More AWS events: https://go.aws/3kss9CP

Subscribe: More AWS videos: http://bit.ly/2O3zS75 More AWS events videos: http://bit.ly/316g9t4

ABOUT AWS: Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts. AWS is the world's most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

AWSreInvent #AWSreInvent2025 #AWS

Trusted by nearly half of Fortune 100 companies, Nasdaq Boardvantage powers secure, intelligent board operations. In this deep dive session, explore how Azure Database for PostgreSQL and MySQL, Microsoft Foundry, Azure Kubernetes Service (AKS), and API Management create a resilient architecture that safeguards confidential data while unlocking new agentic AI capabilities.

CHG Healthcare migrated 2000+ legacy MySQL jobs to dbt Cloud and Snowflake in record time. We'll share how Datafold used their AI-powered Migration Agent to migrate and refactor convoluted legacy code into dbt Cloud and Snowflake with full automatic validation, dramatically accelerating our modernization.

PySpark supports many data sources out of the box, such as Apache Kafka, JDBC, ODBC, Delta Lake, etc. However, some older systems, such as systems that use JMS protocol, are not supported by default and require considerable extra work for developers to read from them. One such example is ActiveMQ for streaming. Traditionally, users of ActiveMQ have to use a middle-man in order to read the stream with Spark (such as writing to a MySQL DB using Java code and reading that table with Spark JDBC). With PySpark 4.0’s custom data sources (supported in DBR 15.3+) we are able to cut out the middle-man processing using batch or Spark Streaming and consume the queues directly from PySpark, saving developers considerable time and complexity in getting source data into your Delta Lake and governed by Unity Catalog and orchestrated with Databricks Workflows.

Dropbox, a leading cloud storage platform, is on a mission to accelerate data insights to better understand customers’ needs and elevate the overall customer experience. By leveraging Fivetran’s data movement platform, Dropbox gained real-time visibility into customer sentiment, marketing ROI, and ad performance-empowering teams to optimize spend, improve operational efficiency, and deliver greater business outcomes.Join this session to learn how Dropbox:- Cut data pipeline time from 8 weeks to 30 minutes by automating ingestion and streamlining reporting workflows.- Enable real-time, reliable data movement across tools like Zendesk Chat, Google Ads, MySQL, and more — at global operations scale.- Unify fragmented data sources into the Databricks Data Intelligence Platform to reduce redundancy, improve accessibility, and support scalable analytics.

Too Good To Go, a Danish start-up focused on fighting food waste, discusses how it managed complexity as it faced challenges of rapid growth and global expansion, while helping save over 400 million meals from going to waste. This includes rapidly scaling its initial PHP application with a single MySQL database to serve its growing user base, and implementing solutions like Amazon Aurora.

Learn more about AWS events: https://go.aws/events

Subscribe: More AWS videos: http://bit.ly/2O3zS75 More AWS events videos: http://bit.ly/316g9t4

ABOUT AWS Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts. AWS is the world’s most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

AWSreInvent #AWSreInvent2024 #AWS

Amazon Aurora is a relational database service built for the cloud that is designed for unparalleled high performance and availability at global scale, with full MySQL and PostgreSQL compatibility. In this session, learn how Amazon Aurora PostgreSQL Limitless Database enables applications to scale to millions of transactions per second across petabytes of data. Explore the architecture, distributed transaction management, and serverless scaling capabilities of Aurora PostgreSQL Limitless Database. Also, discover application patterns that are a good fit for Aurora PostgreSQL Limitless Database and which patterns to avoid. Learn how Aurora PostgreSQL Limitless Database makes it easier than ever to scale Aurora.

Learn more: AWS re:Invent: https://go.aws/reinvent. More AWS events: https://go.aws/3kss9CP

Subscribe: More AWS videos: http://bit.ly/2O3zS75 More AWS events videos: http://bit.ly/316g9t4

About AWS: Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts. AWS is the world's most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

AWSreInvent #AWSreInvent2024

With an innovative architecture that decouples compute from storage and advanced features like Global Database and low-latency read replicas, Amazon Aurora reimagines what it means to be a relational database. Aurora is a modern database service offering unparalleled performance and high availability at scale with full open source MySQL and PostgreSQL compatibility. In this session, dive deep into the most exciting new features that Aurora offers, including Aurora Limitless Database, Aurora I/O-Optimized, Aurora zero-ETL integration with Amazon Redshift, and Aurora Serverless v2. Additionally, learn how the addition of the pgvector extension allows for the storage of vector embeddings and support of vector similarity searches for generative AI.

Learn more: AWS re:Invent: https://go.aws/reinvent. More AWS events: https://go.aws/3kss9CP

Subscribe: More AWS videos: http://bit.ly/2O3zS75 More AWS events videos: http://bit.ly/316g9t4

About AWS: Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts. AWS is the world's most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

AWSreInvent #AWSreInvent2024

Amazon Aurora is a fully managed relational database designed for unparalleled high performance and availability at global scale with full MySQL and PostgreSQL compatibility. Aurora provides managed high availability (HA) and disaster recovery (DR) capabilities in and across AWS Regions. In this session, explore the Aurora HA and DR capabilities and discover design patterns that enable the development of resilient applications. Learn how to establish in-Region and cross-Region HA and DR using Aurora features, including Multi-AZ deployments, Aurora Global Database, and Amazon RDS Proxy, and discover how to reduce failover times with a JDBC driver.

Learn more: AWS re:Invent: https://go.aws/reinvent. More AWS events: https://go.aws/3kss9CP

Subscribe: More AWS videos: http://bit.ly/2O3zS75 More AWS events videos: http://bit.ly/316g9t4

About AWS: Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts. AWS is the world's most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

AWSreInvent #AWSreInvent2024

Microsoft is making significant investments in relational and NoSQL open-source databases. Learn about Azure Database for PostgreSQL, Azure Database for MySQL, Azure Cosmos DB for MongoDB, with new enterprise-ready features to support daily business operations. See new migration capabilities as well as Microsoft's new contributions to the open-source community, including DiskANN, a PostgreSQL extension for Azure OpenAI Service, and much more.

𝗦𝗽𝗲𝗮𝗸𝗲𝗿𝘀: * Aditi Gupta * AVIJIT GUPTA * Dingding Lu * Prasanth Tammiraju

𝗦𝗲𝘀𝘀𝗶𝗼𝗻 𝗜𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻: This is one of many sessions from the Microsoft Ignite 2024 event. View even more sessions on-demand and learn about Microsoft Ignite at https://ignite.microsoft.com

BRK207 | English (US) | Data

MSIgnite

Improve performance with the latest capabilities for Azure SQL Databases, Azure Database for PostgreSQL, and SQL Server enabled by Azure Arc for hybrid and multi-cloud. You’ll learn how customers enabled ongoing innovation by migrating to Azure Database for MySQL. This session will cover tactical ways to get the most from your applications with the databases that are easy to use, deliver unmatched price/performance, support open-source and enable transformative AI technologies.

To learn more, please check out these resources: * https://aka.ms/Ignite23CollectionsBRK224H * https://info.microsoft.com/ww-landing-contact-me-for-events-m365-in-person-events.html?LCID=en-us&ls=407628-contactme-formfill * https://aka.ms/ArcSQL * https://aka.ms/azure-ignite2023-dataaiblog

𝗦𝗽𝗲𝗮𝗸𝗲𝗿𝘀: * Chandra Gavaravarapu * Maximilian Conrad * Shireesh Thota * Simon Faber * Vlad Rabenok * Xiaoxuan Guo * Ed Donahue * Aditya Badramraju * Bob Ward * Denzil Ribeiro * Parikshit Savjani

𝗦𝗲𝘀𝘀𝗶𝗼𝗻 𝗜𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻: This video is one of many sessions delivered for the Microsoft Ignite 2023 event. View sessions on-demand and learn more about Microsoft Ignite at https://ignite.microsoft.com

BRK224H | English (US) | Data

MSIgnite

Many companies are trying to solve the challenges of ingesting transactional data in Data lake and dealing with late-arriving updates and deletes.

To address this at Swiggy, we have built CDC(Change Data Capture) system, an incremental processing framework to power all business-critical data pipelines at low latency and high efficiency.

It offers: Freshness: It operates in near real-time with configurable latency requirements. Performance: Optimized read and write performance with tuned compaction parameters and partitions and delta table optimization. Consistency: It supports reconciliation based on transaction types. Basically applying insert, update, and delete on existing data.

To implement this system, AWS DMS helped us with initial bootstrapping and CDC replication for Mysql sources. AWS Lambda and DynamoDB streams helped us to solve the bootstrapping and CDC replication for DynamoDB source.

After setting up the bootstrap and cdc replication process we have used Databricks delta merge to reconcile the data based on the transaction types.

To support the merge we have implemented supporting features - * Deduplicating multiple mutations of the same record using log offset and time stamp. * Adding optimal partition of the data set. * Infer schema and apply proper schema evolutions(Backward compatible schema) * We have extended the delta table snapshot generation technique to create a consistent partition for partitioned delta tables.

FInally to read the data we are using Spark sql with Hive metastore and Snowflake. Delta tables read with Spark sql have implicit support for hive metastore. We have built our own implementation of the snowflake sync process to create external, internal tables and materialized views on Snowflake.

Stats: 500m CDC logs/day 600+ tables

Connect with us: Website: https://databricks.com Facebook: https://www.facebook.com/databricksinc Twitter: https://twitter.com/databricks LinkedIn: https://www.linkedin.com/company/data... Instagram: https://www.instagram.com/databricksinc/