Productive cross-team collaboration between data engineers and analysts is the goal of all data teams, however, fulfilling on that mission can be challenging given the diverse set of skills that each group brings. In this talk we present an example of how one team tackled this topic by creating a flexible, dynamic and extensible framework using Airflow and cloud services that allowed engineers and analysts to jointly create data-centric micro-services to serve up projections and other robust analysis for use in the organization. The framework, which utilized dynamic DAG generation configured using yaml files, Kubernetes jobs and dbt transformations, abstracted away many of the details associated with workflow orchestration, allowing analysts to focus on their Python or R code and data processing logic while enabling data engineers to monitor the pipelines and ensure their scalability.
talk-data.com
Topic
YAML
Yet Another Markup Language (YAML)
data_serialization
configuration_file_format
human_readable
file_format
1
tagged
Activity Trend
9
peak/qtr
2020-Q1
2026-Q1
Top Events
Airflow Summit 2025
3
dbt Coalesce 2023
3
O'Reilly Data Engineering Books
2
Airflow Summit 2024
2
Data Engineering Podcast
1
dbt Coalesce 2022
1
PyData Rhein-Main I Security Risks in AI & Structured Automation with Agentic AI
1
DataTopics: All Things Data, AI & Tech
1
AI Camp NYC: GenAI, LLMs and Agent
1
Big Data LDN 2025
1
Data Expo NL 2025
1
AI and Deep Learning for Enterprise #11
1
Filtering by:
Stanisław Smyl
×