talk-data.com

Register Here

Take the next step in your API platform journey by mastering APIOps automation with Kong Konnect. This hands-on workshop builds on the foundational concepts introduced in Kong Konnect Automations: Introduction to API Platform-as-a-Service Concepts and Delivery Methods, shifting the focus from platform setup to end-to-end API lifecycle automation.

Learn how to implement APIOps pipelines that seamlessly move APIs from design to deployment—without manual handoffs. We’ll walk through designing API contracts, automating gateway configurations, and enforcing governance and security policies as code. Explore how platform teams can define guardrails and reusable patterns, while API product teams gain autonomy to iterate and deploy APIs with speed and confidence.

Key Takeaways:

• Design a GitOps-driven APIOps pipeline for API publishing, versioning, and deployment to Kong Konnect.
• Apply automated governance: enforce linting, style guides, and approval gates across the API lifecycle.
• Separate responsibilities between platform teams (infrastructure and governance) and API teams (API design and product delivery).
• Integrate API specifications (e.g., OpenAPI) into CI/CD workflows for fully automated, policy-compliant API gateway provisioning.
• Build a reusable reference APIOps workflow to take back to your organization.

Through live demos and real-world patterns you’ll leave with a clear blueprint for running APIs as products, at scale, through automation.

Register Here

Take the next step in your API platform journey by mastering APIOps automation with Kong Konnect. This hands-on workshop builds on the foundational concepts introduced in Kong Konnect Automations: Introduction to API Platform-as-a-Service Concepts and Delivery Methods, shifting the focus from platform setup to end-to-end API lifecycle automation.

Learn how to implement APIOps pipelines that seamlessly move APIs from design to deployment—without manual handoffs. We’ll walk through designing API contracts, automating gateway configurations, and enforcing governance and security policies as code. Explore how platform teams can define guardrails and reusable patterns, while API product teams gain autonomy to iterate and deploy APIs with speed and confidence.

Key Takeaways:

• Design a GitOps-driven APIOps pipeline for API publishing, versioning, and deployment to Kong Konnect.
• Apply automated governance: enforce linting, style guides, and approval gates across the API lifecycle.
• Separate responsibilities between platform teams (infrastructure and governance) and API teams (API design and product delivery).
• Integrate API specifications (e.g., OpenAPI) into CI/CD workflows for fully automated, policy-compliant API gateway provisioning.
• Build a reusable reference APIOps workflow to take back to your organization.

Through live demos and real-world patterns you’ll leave with a clear blueprint for running APIs as products, at scale, through automation.

Register Here

Take the next step in your API platform journey by mastering APIOps automation with Kong Konnect. This hands-on workshop builds on the foundational concepts introduced in Kong Konnect Automations: Introduction to API Platform-as-a-Service Concepts and Delivery Methods, shifting the focus from platform setup to end-to-end API lifecycle automation.

Learn how to implement APIOps pipelines that seamlessly move APIs from design to deployment—without manual handoffs. We’ll walk through designing API contracts, automating gateway configurations, and enforcing governance and security policies as code. Explore how platform teams can define guardrails and reusable patterns, while API product teams gain autonomy to iterate and deploy APIs with speed and confidence.

Key Takeaways:

• Design a GitOps-driven APIOps pipeline for API publishing, versioning, and deployment to Kong Konnect.
• Apply automated governance: enforce linting, style guides, and approval gates across the API lifecycle.
• Separate responsibilities between platform teams (infrastructure and governance) and API teams (API design and product delivery).
• Integrate API specifications (e.g., OpenAPI) into CI/CD workflows for fully automated, policy-compliant API gateway provisioning.
• Build a reusable reference APIOps workflow to take back to your organization.

Through live demos and real-world patterns you’ll leave with a clear blueprint for running APIs as products, at scale, through automation.

Register Here

Take the next step in your API platform journey by mastering APIOps automation with Kong Konnect. This hands-on workshop builds on the foundational concepts introduced in Kong Konnect Automations: Introduction to API Platform-as-a-Service Concepts and Delivery Methods, shifting the focus from platform setup to end-to-end API lifecycle automation.

Learn how to implement APIOps pipelines that seamlessly move APIs from design to deployment—without manual handoffs. We’ll walk through designing API contracts, automating gateway configurations, and enforcing governance and security policies as code. Explore how platform teams can define guardrails and reusable patterns, while API product teams gain autonomy to iterate and deploy APIs with speed and confidence.

Key Takeaways:

• Design a GitOps-driven APIOps pipeline for API publishing, versioning, and deployment to Kong Konnect.
• Apply automated governance: enforce linting, style guides, and approval gates across the API lifecycle.
• Separate responsibilities between platform teams (infrastructure and governance) and API teams (API design and product delivery).
• Integrate API specifications (e.g., OpenAPI) into CI/CD workflows for fully automated, policy-compliant API gateway provisioning.
• Build a reusable reference APIOps workflow to take back to your organization.

Through live demos and real-world patterns you’ll leave with a clear blueprint for running APIs as products, at scale, through automation.

In this session, we’ll introduce the concept of Pulumi Components - packages that can be authored in one language and consumed in any other language. This enables platform engineering teams to create powerful patterns for reuse across their organization, such as sharing infrastructure libraries written in common programming languages that can easily be instantiated from a simple YAML file.

In this session, we’ll introduce the concept of Pulumi Components - packages that can be authored in one language and consumed in any other language. This enables platform engineering teams to create powerful patterns for reuse across their organization, such as sharing infrastructure libraries written in common programming languages that can easily be instantiated from a simple YAML file.

An important tenet of Platform Engineering is building infrastructure code that is DRY (don’t repeat yourself). This makes it easier to maintain core infrastructure components and ensures best practices are reflected in each new instance of your infrastructure.

In this session, we’ll introduce the concept of Pulumi Components - packages that can be authored in one language and consumed in any other language.

This enables platform engineering teams to create powerful patterns for reuse across their organization, such as sharing infrastructure libraries written in common programming languages that can easily be instantiated from a simple YAML file. Speakers:

• Rob Smith, Solutions Architect
• Josh Kodroff, Principal Solutions Architect

Join us to learn:

• How to author and share Pulumi Components.
• Tips for structuring YAML (or any other language!) to consume a Library with maximum reusability.
• How to integrate reusable components into your IDP.

Where: BigMarker at https://pulumip.us/DRY-IDP Please register using the link above to attend and receive the link to the example code and workshop recording. Join our other workshop here: https://pulumip.us/UpcomingWorkshops

An important tenet of Platform Engineering is building infrastructure code that is DRY (don’t repeat yourself). This makes it easier to maintain core infrastructure components and ensures best practices are reflected in each new instance of your infrastructure.

In this session, we’ll introduce the concept of Pulumi Components - packages that can be authored in one language and consumed in any other language.

This enables platform engineering teams to create powerful patterns for reuse across their organization, such as sharing infrastructure libraries written in common programming languages that can easily be instantiated from a simple YAML file.

Speakers:

• Rob Smith, Solutions Architect
• Josh Kodroff, Principal Solutions Architect

Join us to learn:

• How to author and share Pulumi Components.
• Tips for structuring YAML (or any other language!) to consume a Library with maximum reusability.
• How to integrate reusable components into your IDP.

Where: BigMarker at https://pulumip.us/DRY-IDP Please register using the link above to attend and receive the link to the example code and workshop recording. Join our other workshop here: https://pulumip.us/UpcomingWorkshops

Summary Data platforms are exemplified by a complex set of connections that are subject to a set of constantly evolving requirements. In order to make this a tractable problem it is necessary to define boundaries for communication between concerns, which brings with it the need to establish interface contracts for communicating across those boundaries. The recent move toward the data mesh as a formalized architecture that builds on this design provides the language that data teams need to make this a more organized effort. In this episode Abhi Sivasailam shares his experience designing and implementing a data mesh solution with his team at Flexport, and the importance of defining and enforcing data contracts that are implemented at those domain boundaries.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show! Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription StreamSets DataOps Platform is the world’s first single platform for building smart data pipelines across hybrid and multi-cloud architectures. Build, run, monitor and manage data pipelines confidently with an end-to-end data integration platform that’s built for constant change. Amp up your productivity with an easy-to-navigate interface and 100s of pre-built connectors. And, get pipelines and new hires up and running quickly with powerful, reusable components that work across batch and streaming. Once you’re up and running, your smart data pipelines are resilient to data drift. Those ongoing and unexpected changes in schema, semantics, and infrastructure. Finally, one single pane of glass for operating and monitoring all your data pipelines. The full transparency and control you desire for your data operations. Get started building pipelines in minutes for free at dataengineeringpodcast.com/streamsets. The first 10 listeners of the podcast that subscribe to StreamSets’ Professional Tier, receive 2 months free after their first month. Your host is Tobias Macey and today I’m interviewing Abhi Sivasailam about the different social and technical interfaces available for defining and enforcing data contracts

Interview

Introduction How did you get involved in the area of data management? Can you start by explaining what your working definition of a "data contract" is?

What are the goals and purpose of these contracts?

What are the locations and methods of defining a data contract?

What kind of information needs to be encoded in a contract definition?

How do you manage enforcement of contracts? manifestations of contracts in data mesh implementation ergonomics (technical and social) of data contracts and how to prevent them from prohibiting productivity What are the most interesting, innovative