talk-data.com

Paolo Platter, CTO and co-founder of Agile Lab and Witboost, joined Yuliia to share how his 10 years of building custom data solutions for clients led to creating Witboost - a platform that helps big companies manage their data products at scale. One of their customers used Witboost to build over 250 data products in just 18 months, showing how well the platform works at scale. Paolo explained why setting rules for data teams becomes harder as companies grow, and shared how he shifted from saying "yes" to every client request as a consultant to building a product that works for many companies.Paolo Platter - https://www.linkedin.com/in/paoloplatter/

Join us for our monthly Data Mesh Learning Roundtable. This is an interactive roundtable that encourages participant interaction. We will divide the group into break-out sessions moderated by our four hosts:

Andrew Sharp, Principal Consultant/Data Governance, Oakland Group Amy Raygada, Senior Data Product Manager, Swiss Marketplace Karin Håkansson, Data Governance/Data Mesh Lead Consultant Wannes Rosiers, Product Manager and Data Enthusiast, Conveyor

with special guest Paolo Platter, CTO - Co-founder at Agile Lab

Navigating the Data Mesh: Designing Data Products for Success

In today's data-driven landscape, organizations are increasingly adopting the data mesh paradigm to manage and leverage their vast data resources effectively. However, navigating the complexities of data product design within a data mesh framework requires a nuanced understanding of both technical and organizational dynamics.

Join us for a dynamic roundtable discussion where industry experts, thought leaders, and practitioners converge to explore the intricacies of designing data products within a data mesh architecture.

Whether you're a seasoned data architect, a data engineer grappling with complex infrastructure challenges, or a business leader seeking to harness the power of data for strategic advantage, this roundtable promises to offer valuable insights and actionable strategies for designing data products within a data mesh framework.

Summary Data mesh is a frequent topic of conversation in the data community, with many debates about how and when to employ this architectural pattern. The team at AgileLab have first-hand experience helping large enterprise organizations evaluate and implement their own data mesh strategies. In this episode Paolo Platter shares the lessons they have learned in that process, the Data Mesh Boost platform that they have built to reduce some of the boilerplate required to make it successful, and some of the considerations to make when deciding if a data mesh is the right choice for you.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show! Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value. Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos. Prefect is the modern Dataflow Automation platform for the modern data stack, empowering data practitioners to build, run and monitor robust pipelines at scale. Guided by the principle that the orchestrator shouldn’t get in your way, Prefect is the only tool of its kind to offer the flexibility to write code as workflows. Prefect specializes in glueing together the disparate pieces of a pipeline, and integrating with modern distributed compute libraries to bring power where you need it, when you need it. Trusted by thousands of organizations and supported by over 20,000 community members, Prefect powers over 100MM business critical tasks a month. For more information on Prefect, visit dataengineeringpodcast.com/prefect. The only thing worse than having bad data is not knowing that you have it. With Bigeye’s data observability platform, if there is an issue with your data or data pipelines you’ll know right away and can get it fixed before the business is impacted. Bigeye let’s data teams measure, improve, and communicate the quality of your data to company stakeholders. With complete API access, a user-friendly interface, and automated yet flexible alerting, you’ve got everything you need to establish and maintain trust in your data. Go to dataengineeringpodcast.com/bigeye today to sign up and start trusting your analyses. Your host is Tobias Macey and today I’m interviewing Paolo Platter about Agile Lab’s lessons learned through helping large enterprises establish their own data mesh

Interview

Introduction How did you get involved in the area of data management? Can you share your experiences working with data mesh implementations? What were the stated goals of project engagements that led to data mesh implementations? What are some examples of projects where you explored data mesh as an option and decided that it was a poor fit? What are some of the technical and process investments that are necessary to support a mesh str