High-performance analytics can be a game-changer, but the infrastructure to support it often comes with a hefty price tag. In this talk, I'll share strategies to build a cost-effective sub-second analytics platform using open-source tools and lessons learned from cloud services. I'll cover the trade-offs between self-hosting and managed offerings and discuss ways to optimize infrastructure costs — from maximizing cloud 'nuggets' to ensuring your engineering team is right-sized and focused on solving important business challenges. Drawing from patterns, caveats, and success stories observed with our clients, the goal is to power lightning-fast analytics without breaking the bank.

With recent works in the direction of Executor Decoupling and interest in Hybrid Execution, we find it’s still quite common for Airflow users to use the old-time rule of thumbs like “Don’t Use Airflow with LocalExecutor in production”, “If your scheduler lags, split your DAGs over two separate Airflow Clusters”, and so on. In our talk, we will show a deep dive comparison between various Execution models Airflow support and hopefully update understanding of their efficiency and limitations.