ABOUT THE TALK: In this talk, Dan Selans shows you how we developed a schema discovery process that is able to automatically evolve schemas in a complex distributed system that is processing upwards of a 100,000 messages per second.

He dives deep into the details of schema versioning, detecting schema conflicts, compatibility and normalization, all without the use of any batching processes.

He shows how they developed a schema discovery process that is able to automatically evolve schemas in a complex distributed system that is processing upwards of a 100,000 messages per second. He also details how to detect schema drift, determine compatibility and ultimately how to do all of this, without having to involve batching.

ABOUT THE SPEAKER: Daniel Selans is the co-founder and CTO of Streamdal.com, a streaming data performance monitoring company. Dan previously wrote software at companies such as InVisionApp, New Relic and DigitalOcean and before that, spent over 10 years doing integration and R&D work at data centers.

ABOUT DATA COUNCIL: Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers.

Make sure to subscribe to our channel for the most up-to-date talks from technical professionals on data related topics including data infrastructure, data engineering, ML systems, analytics and AI from top startups and tech companies.

FOLLOW DATA COUNCIL: Twitter: https://twitter.com/DataCouncilAI LinkedIn: https://www.linkedin.com/company/datacouncil

ABOUT THE TALK: What are the latest trends and buzzwords in Data?

Barry McCordel welcomes panelists from Hex, DBT Labs and West Marin Data to discuss their thoughts on the latest trends and buzzwords in Data.

Learn about the latest in the world of streaming, data teams doing more with less, data meshes, innovations in different kids of SQL plus more!

ABOUT THE SPEAKERS: Julia Schottenstein is the Product Manager at dbt labs. Prior to this, she worked in Venture Capital as a Principal at NEA.

Drew Banin is the co-founder of dbt labs. He has built event collection systems that scaled to billions of events per month, implemented Markov-based marketing attribution models on millions of dollars of marketing spend, and dreams in NetworkX graphs.

Barry McCardel is the CEO and co-founder of Hex. He previously worked at TrialSpark leading operation and Palantir Technologies where he led teams at the intersection of product development and real-world impact.

Pedram Navid is the Founder of West Marin Data. In his role he helps startups implement their data stack. He also supports them with product, marketing and community-building.

ABOUT DATA COUNCIL: Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers.

Make sure to subscribe to our channel for the most up-to-date talks from technical professionals on data related topics including data infrastructure, data engineering, ML systems, analytics and AI from top startups and tech companies.

FOLLOW DATA COUNCIL: Twitter: https://twitter.com/DataCouncilAI LinkedIn: https://www.linkedin.com/company/datacouncil

ABOUT THE TALK: The most innovative companies have learned to leverage their data platform as a competitive advantage in all facets of business. Teams are supplied with the data they need with high confidence to solve problems and meet their stakeholders' needs. Sound like the holy grail?

Learn what processes, platforms, and personnel are needed to transform your data culture from centralized ad-hoc analysis to self-service, proactive insights, and universally loved features.

ABOUT THE SPEAKER: DeVaris Brown is the CEO and co-founder of Meroxa, a data application platform as a service that enables engineers to solve problems that leverage streaming data with code. Before founding Meroxa, DeVaris was a product leader at Twitter, Heroku, VSCO, and Zendesk.

ABOUT DATA COUNCIL: Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers.

Make sure to subscribe to our channel for the most up-to-date talks from technical professionals on data related topics including data infrastructure, data engineering, ML systems, analytics and AI from top startups and tech companies.

FOLLOW DATA COUNCIL: Twitter: https://twitter.com/DataCouncilAI LinkedIn: https://www.linkedin.com/company/datacouncil-ai/

ABOUT THE TALK: A streaming database is a potentially intimidating product to build. Frank McSherry, Chief Scientist at Materialize, breaks down the manageable parts, through three foundational choices that fit together well. Frank also talks about the trade-offs, and how their simplifications lead to a much more manageable streaming database.

ABOUT THE SPEAKER: Frank McSherry is Chief Scientist at Materialize, where he (and others) convert SQL into scale-out, streaming, and interactive dataflows. Before this, he developed the timely and differential dataflow Rust libraries (with colleagues at ETHZ), and led the Naiad research project and co-invented differential privacy while at MSR Silicon Valley. He has a PhD in computer science from the University of Washington.

ABOUT DATA COUNCIL: Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers.

Make sure to subscribe to our channel for the most up-to-date talks from technical professionals on data related topics including data infrastructure, data engineering, ML systems, analytics and AI from top startups and tech companies.

FOLLOW DATA COUNCIL: Twitter: https://twitter.com/DataCouncilAI LinkedIn: https://www.linkedin.com/company/datacouncil-ai/

ABOUT THE TALK: Microservices are one of the big trends in software engineering of the last few years.

In this session we'll discuss and showcase how open-source change data capture (CDC) with Debezium can help developers with typical challenges they often face when working on microservices.

Learn how to:

• Employ the outbox pattern for reliable, eventually consistent data exchange between microservices, without incurring unsafe dual writes or tight coupling
• Gradually extract microservices from existing monolithic applications, using CDC, the strangler fig pattern and Apache Flink
• Coordinate long-running business transactions across multiple services using CDC-based saga orchestration, ensuring such activity gets consistently applied or aborted by all participating services.

ABOUT THE SPEAKER: Gunnar Morling is a software engineer and open-source enthusiast by heart, currently working at Decodable on stream processing based on Apache Flink. In his prior role as a software engineer at Red Hat, he led the Debezium project, a distributed platform for change data capture. He is a Java Champion and has founded multiple open source projects such as JfrUnit, kcctl, and MapStruct. Gunnar is an avid blogger (morling.dev) and has spoken at a wide range of conferences like QCon, Java One, and Devoxx. He lives in Hamburg, Germany.

ABOUT DATA COUNCIL: Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers.

Make sure to subscribe to our channel for the most up-to-date talks from technical professionals on data related topics including data infrastructure, data engineering, ML systems, analytics and AI from top startups and tech companies.

FOLLOW DATA COUNCIL: Twitter: https://twitter.com/DataCouncilAI LinkedIn: https://www.linkedin.com/company/datacouncil-ai/

ABOUT THE TALK: With more and more demand for data pipelines and applications to go real-time it can get overwhelming. This talk demystifies the when, why, and how of moving from batch processing to real-time/stream processing. We will look at arguments for and against stream processing, common architectures, common pitfalls, and open source tools used.

ABOUT THE SPEAKER: Zander Matheson is the founder and CEO of Bytewax, an open source software company focused on enabling more developers to work with streaming data. Before starting Bytewax he worked on data infrastructure and data science at GitHub and Heroku.

ABOUT DATA COUNCIL: Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers.

Make sure to subscribe to our channel for the most up-to-date talks from technical professionals on data related topics including data infrastructure, data engineering, ML systems, analytics and AI from top startups and tech companies.

FOLLOW DATA COUNCIL: Twitter: https://twitter.com/DataCouncilAI LinkedIn: https://www.linkedin.com/company/datacouncil-ai/

ABOUT THE TALK: This talk explains what the control plane of data looks like and how it fits into the reference architecture for the deconstructed data stack: a data stack that includes operational data stores, streaming systems, transformation engines, BI tools, warehouses, ML tools and orchestrators.

It dives into the fundamental characteristics for a control plane:

Breadth (completeness) Latency (freshness) Scale Source of Truth Auditability

ABOUT THE SPEAKER: Shirshanka Das is the Co-founder and CEO of Acryl Data, the company which is commercializing the open source DataHub project, a real-time metadata platform used by LinkedIn, Stripe, Pinterest, Optum, Expedia and many others. Prior to founding Acryl, he was the overall architect for Big Data at LinkedIn from 2010 to 2020, and responsible for creating the metadata and data management strategy at the company.

ABOUT DATA COUNCIL: Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers.

Make sure to subscribe to our channel for the most up-to-date talks from technical professionals on data related topics including data infrastructure, data engineering, ML systems, analytics and AI from top startups and tech companies.

FOLLOW DATA COUNCIL: Twitter: https://twitter.com/DataCouncilAI LinkedIn: https://www.linkedin.com/company/datacouncil-ai