Search – talk-data.com

Title & Speakers	Event
OpenLineage Meetup @ Google 2025-04-03 · 15:30 Note: this event is hybrid. Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift without a shared standard for lineage metadata. It requires duplication of effort across pipeline tooling, and deployment of new tools can break existing lineage workflows. Getting useful lineage can seem like a sisyphean task. Enter OpenLineage, an increasingly adopted open standard for lineage metadata collection. It defines a generic model of run, job, and dataset entities identified using consistent naming strategies. The core lineage model is extensible by defining specific facets to enrich those entities. Agenda: 17:30 Arrival and welcome 17:40 GCP Composer Migration to OpenLineage (Augusto Hidalgo, Google) 18:05 Adding OpenLineage support to Airflow operators (Kacper Muda, GetInData now Astronomer) 18:30 Are they compatible? Standardized compatibility testing for OpenLineage (Tomasz Nazarewicz, GetInData) 18:55 Break for Food 19:40 Connecting OpenLineage within Observability Ecosystems (Maciej Obuchowski, Datadog) 20:05 Exposing metrics through Spark OpenLineage connector (Paweł Leszczynski, GetInData) 20:30 Adjourn Please note: First name and last name will be checked against ID before entrance to Google premises. Registrations sent after March 31 will not be accepted	OpenLineage Meetup @ Google
Airflow Meetup @ G-Research 2023-08-24 · 17:00 Save the date!! Let's meet at the G-Research office for an evening of great talks with pizza and drinks! * Talk #1: OpenLineage in Airflow: A Comprehensive Guide Speaker: Maciej Obuchowski (Software engineer @ GetInData and OpenLineage committer) With native support for Openlineage in Airflow, users can now easily observe and manage their data pipelines. This talk will cover the benefits of using OpenLineage, its implementation in Airflow, practical examples of how to take advantage of it, and what’s in our roadmap. Whether you’re an Airflow user or provider maintainer, this session will give you the knowledge to make the most of this tool. * Talk #2: Running dbt pipelines in Airflow Speaker: Tatiana Al-Chueyr (Staff Software Engineer @ Astronomer) dbt is an open-source project that allows data teams to transform data by defining pipelines, mostly with SQL files. Using Airflow to orchestrate and execute dbt projects as DAGs gives users a reliable and scalable environment to run them. This talk introduces Cosmos, an open-source package from Astronomer that helps users run dbt pipelines as Airflow DAGs with few lines of code. Cosmos allows users to have fine-grained control over dbt resources while benefiting from various Airflow features, such as data-aware scheduling and retries. *** Thanks to G-Research for hosting the event! We are looking for speakers for future Meetup Sessions. Please fill out https://forms.gle/ES1YDE6wsHy95xKf8 if you are interested.	Airflow Meetup @ G-Research

Note: this event is hybrid.

Data engineers and pipeline managers know that producing data lineage – end-to-end pipeline metadata instrumented at runtime or parsed at design time – is a heavy lift without a shared standard for lineage metadata. It requires duplication of effort across pipeline tooling, and deployment of new tools can break existing lineage workflows. Getting useful lineage can seem like a sisyphean task.

Enter OpenLineage, an increasingly adopted open standard for lineage metadata collection. It defines a generic model of run, job, and dataset entities identified using consistent naming strategies. The core lineage model is extensible by defining specific facets to enrich those entities.

Agenda:

17:30 Arrival and welcome
17:40 GCP Composer Migration to OpenLineage (Augusto Hidalgo, Google)
18:05 Adding OpenLineage support to Airflow operators (Kacper Muda, GetInData now Astronomer)
18:30 Are they compatible? Standardized compatibility testing for OpenLineage (Tomasz Nazarewicz, GetInData)
18:55 Break for Food
19:40 Connecting OpenLineage within Observability Ecosystems (Maciej Obuchowski, Datadog)
20:05 Exposing metrics through Spark OpenLineage connector (Paweł Leszczynski, GetInData)
20:30 Adjourn

Please note:

First name and last name will be checked against ID before entrance to Google premises.
Registrations sent after March 31 will not be accepted

Save the date!! Let's meet at the G-Research office for an evening of great talks with pizza and drinks!

***

Talk #1: OpenLineage in Airflow: A Comprehensive Guide

Speaker: Maciej Obuchowski (Software engineer @ GetInData and OpenLineage committer)

With native support for Openlineage in Airflow, users can now easily observe and manage their data pipelines. This talk will cover the benefits of using OpenLineage, its implementation in Airflow, practical examples of how to take advantage of it, and what’s in our roadmap. Whether you’re an Airflow user or provider maintainer, this session will give you the knowledge to make the most of this tool.

***

Talk #2: Running dbt pipelines in Airflow

Speaker: Tatiana Al-Chueyr (Staff Software Engineer @ Astronomer)

dbt is an open-source project that allows data teams to transform data by defining pipelines, mostly with SQL files. Using Airflow to orchestrate and execute dbt projects as DAGs gives users a reliable and scalable environment to run them.

This talk introduces Cosmos, an open-source package from Astronomer that helps users run dbt pipelines as Airflow DAGs with few lines of code. Cosmos allows users to have fine-grained control over dbt resources while benefiting from various Airflow features, such as data-aware scheduling and retries.

***

Thanks to G-Research for hosting the event!

We are looking for speakers for future Meetup Sessions. Please fill out https://forms.gle/ES1YDE6wsHy95xKf8 if you are interested.

talk-data.com

People (5 results)

Activities & events