In today’s data-driven world, whether you’re building your own data pipelines or relying on third-party vendors, understanding the fundamentals of great data movement systems is invaluable. It’s not just about making things work—it’s about ensuring your data operations are reliable, scalable, and cost-effective.

As an early employee and Airbyte’s Platform Architect, I’ve spent the last 3.5 years working through the challenges and intricacies of building a data movement platform. Along the way, I’ve learned some important lessons, often the hard way, that I believe could be helpful to others who are on a similar journey.

In this session, I’ll share these lessons in the hope that my experiences can offer some guidance, whether you’re just starting out or looking to refine what you’ve already built. I’ll also touch on how the rapid rise of generative AI is changing the landscape, and how we’re trying to adapt to these new challenges. My goal is to provide insights anyone can take back to their own projects, helping them avoid some of the pitfalls and navigate the complexities of modern data movement.

2 - 3 Main Actionable Takeaways:

• A general framework for designing a data movement system.

• Crucial fine print such as managing various destination memory types, the surprising need to re-import data and the shortcuts & pitfalls of artificial cursors.

• Adjusting data movement systems for an AI-first world.