This session explores advanced JSON Schema handing(inference and evolving), and event DemuxingTopics include: How from_json is currently used today and its challenges. How to use Variant for rapidly changing schema. How from_json in Lakeflow Declarative Pipelines with primed schema helps simplify schema handling. Demultiplexing patterns for scalable stream processing. Simply event Demuxing with Lakeflow Declarative Pipelines.

Surfline uses Segment to collect product analytics events to understand how surfers use their forecasts and live surf cameras across 9000+ surf spots worldwide. An open source tool was developed to define and manage product analytics event schemas using JSON schema which are used to build dbt staging models for all events.

With this solution, the data team has more time to build intermediate and mart models in dbt, knowing that our staging layer fully reflects Surfline’s product analytics events. This presentation is a real-life example on how schemas (or data contracts) can be used as a medium to build consensus, enforce standards, improve data quality, and speed up the dbt workflow for product analytics.

Speaker: Greg Clunies, Senior Analytics Engineer, Surfline

Register for Coalesce at https://coalesce.getdbt.com/

Is your SQL query the problem, or how you ask for the data you need, when you need it. In this deep dive, Jake Thomas shares his hypothesis for why the jsonschema is the ticket to contract-driven communication, system interoperability, and an overall improvement to data processing quality of life.

Check the slides here: https://docs.google.com/presentation/d/1kiGyQF7NUWfx-5RyIyeEwSUCwqtIdrXADeI2iixUgiI/edit?usp=sharing

Coalesce 2023 is coming! Register for free at https://coalesce.getdbt.com/.