This session presents practical patterns for architecting secure, AI-ready frontends on AWS with Cognito-powered scalable auth. Topics include OIDC (auth code + PKCE), token/session hygiene, least-privilege IAM, and safely brokering AI endpoints via API Gateway/Lambda with rate limits, redaction, and cost guardrails. Attendees leave with a reference architecture, a hardening checklist, and a repeatable test strategy using Cypress Cloud.

Most teams are being asked to do more with less. The fastest productivity wins now come from Agentic AI, but legacy ETL and brittle data estates hold you back. In this 15-minute session, Infinite Lambda and Pacific Life Re share a practical roadmap to make your Snowflake platform AI-ready in weeks, not years.

Un témoignage inspirant pour les entreprises souhaitant moderniser leur stack ETL et exploiter pleinement les avantages du cloud.

MACIF partage son retour d'expérience sur la migration de plus de 400 workflows Informatica vers BigQuery et la plateforme dbt, en utilisant les accélérateurs développés par Infinite Lambda, dans le cadre de son projet de modernisation data "Move to Cloud". 

Aux côtés de Laurent, découvrez comment la MACIF a tiré parti du cloud pour accélérer la livraison de ses data products, réduire les risques techniques et améliorer la gouvernance des données.

Summary In this episode of the AI Engineering Podcast Mark Brooker, VP and Distinguished Engineer at AWS, talks about how agentic workflows are transforming database usage and infrastructure design. He discusses the evolving role of data in AI systems, from traditional models to more modern approaches like vectors, RAG, and relational databases. Mark explains why agents require serverless, elastic, and operationally simple databases, and how AWS solutions like Aurora and DSQL address these needs with features such as rapid provisioning, automated patching, geodistribution, and spiky usage. The conversation covers topics including tool calling, improved model capabilities, state in agents versus stateless LLM calls, and the role of Lambda and AgentCore for long-running, session-isolated agents. Mark also touches on the shift from local MCP tools to secure, remote endpoints, the rise of object storage as a durable backplane, and the need for better identity and authorization models. The episode highlights real-world patterns like agent-driven SQL fuzzing and plan analysis, while identifying gaps in simplifying data access, hardening ops for autonomous systems, and evolving serverless database ergonomics to keep pace with agentic development.

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data managementData teams everywhere face the same problem: they're forcing ML models, streaming data, and real-time processing through orchestration tools built for simple ETL. The result? Inflexible infrastructure that can't adapt to different workloads. That's why Cash App and Cisco rely on Prefect. Cash App's fraud detection team got what they needed - flexible compute options, isolated environments for custom packages, and seamless data exchange between workflows. Each model runs on the right infrastructure, whether that's high-memory machines or distributed compute. Orchestration is the foundation that determines whether your data team ships or struggles. ETL, ML model training, AI Engineering, Streaming - Prefect runs it all from ingestion to activation in one platform. Whoop and 1Password also trust Prefect for their data operations. If these industry leaders use Prefect for critical workflows, see what it can do for you at dataengineeringpodcast.com/prefect.Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.Your host is Tobias Macey and today I'm interviewing Marc Brooker about the impact of agentic workflows on database usage patterns and how they change the architectural requirements for databasesInterview IntroductionHow did you get involved in the area of data management?Can you describe what the role of the database is in agentic workflows?There are numerous types of databases, with relational being the most prevalent. How does the type and purpose of an agent inform the type of database that should be used?Anecdotally I have heard about how agentic workloads have become the predominant "customers" of services like Neon and Fly.io. How would you characterize the different patterns of scale for agentic AI applications? (e.g. proliferation of agents, monolithic agents, multi-agent, etc.)What are some of the most significant impacts on workload and access patterns for data storage and retrieval that agents introduce?What are the categorical differences in that behavior as compared to programmatic/automated systems?You have spent a substantial amount of time on Lambda at AWS. Given that LLMs are effectively stateless, how does the added ephemerality of serverless functions impact design and performance considerations around having to "re-hydrate" context when interacting with agents?What are the most interesting, innovative, or unexpected ways that you have seen serverless and database systems used for agentic workloads?What are the most interesting, unexpected, or challenging lessons that you have learned while working on technologies that are supporting agentic applications?Contact Info BlogLinkedInParting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.init covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.Links AWS Aurora DSQLAWS LambdaThree Tier ArchitectureVector DatabaseGraph DatabaseRelational DatabaseVector EmbeddingRAG == Retrieval Augmented GenerationAI Engineering Podcast EpisodeGraphRAGAI Engineering Podcast EpisodeLLM Tool CallingMCP == Model Context ProtocolA2A == Agent 2 Agent ProtocolAWS Bedrock AgentCoreStrandsLangChainKiroThe intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Apache Airflow’s executor landscape has traditionally presented users with a clear trade-off: choose either the speed of local execution or the scalability, isolation and configurability of remote execution. The AWS Lambda Executor introduces a new paradigm that bridges this gap, offering near-local execution speeds with the benefits of remote containerization. This talk will begin with a brief overview of Airflow’s executors, how they work and what they are responsible for, highlighting the compromises between different executors. We will explore the emerging niche for fast, yet remote execution and demonstrate how the AWS Lambda Executor fills this space. We will also address practical considerations when using such an executor, such as working within Lambda’s 15 minute execution limit, and how to mitigate this using multi-executor configuration. Whether you’re new to Airflow or an experienced user, this session will provide valuable insights into task execution and how you can combine the best of both local and remote execution paradigms.

Discover how Apache Airflow powers scalable ELT pipelines, enabling seamless data ingestion, transformation, and machine learning-driven insights. This session will walk through: Automating Data Ingestion: Using Airflow to orchestrate raw data ingestion from third-party sources into your data lake (S3, GCP), ensuring a steady pipeline of high-quality training and prediction data. Optimizing Transformations with Serverless Computing: Offloading intensive transformations to serverless functions (GCP Cloud Run, AWS Lambda) and machine learning models (BigQuery ML, Sagemaker), integrating their outputs seamlessly into Airflow workflows. Real-World Impact: A case study on how INTRVL leveraged Airflow, BigQuery ML, and Cloud Run to analyze early voting data in near real-time, generating actionable insights on voter behavior across swing states. This talk not only provides a deep dive into the Political Tech space but also serves as a reference architecture for building robust, repeatable ELT pipelines. Attendees will gain insights into modern serverless technologies from AWS and GCP that enhance Airflow’s capabilities, helping data engineers design scalable, cloud-agnostic workflows.

It’s time for another episode of the Data Engineering Central Podcast. In this episode, we cover … * AWS Lambda + DuckDB and Delta Lake (Polars, Daft, etc). * IAC - Long Live Terraform. * Databricks Data Quality with DQX. * Unity Catalog releases for DuckDB and Polars * Bespoke vs Managed Data Platforms * Delta Lake vs. Iceberg and UinFORM for a single table. Thanks for b…

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit dataengineeringcentral.substack.com/subscribe

The Node.js ecosystem offers incredible options—from AWS Lambda and Cloudflare Workers to the maturity of Express and the speed of Bun.js and uWS servers. But navigating this diversity can be a challenge. Introducing Vramework: a TypeScript-first, function-based framework that seamlessly integrates with them all. Packed with essentials like authentication, permissions, documentation, HTTP, WebSockets, and more, Vramework simplifies backend development while embracing the ecosystem’s full potential.

Intro to Serverless with AWS Lambda Functions In today’s fast-evolving tech world, serverless is transforming how we build scalable architecture by allowing engineers to focus on the code logic while the infrastructure is handled by the cloud provider. In this session, we will discuss the fundamentals of serverless with Lambda functions, have a quick introduction to NodeJS runtime and how to get a working lambda function in AWS.

🌟 Session Overview 🌟

Session Name: An Evolution of Data Architectures - Lambda, Kappa, Delta, Mesh & Fabric Speaker: Paul Andrew Session Description: How have advancements in highly scalable cloud technology influenced the design principles we apply when building data platform solutions? Are we designing solely for speed and batch layers, or do we want more from our platforms? Who says these patterns must be delivered exclusively?

Let’s disrupt the theory and consider the practical application of everything Microsoft now has to offer, where concepts and patterns meet technology. Can we now utilize cloud technology to build architectures that cater to lambda, kappa, and Delta Lake concepts in a complete stack of services? Should we be considering a solution that offers all these principles in a nirvana of data insight perfection? How does the concept of Data Fabric align with Microsoft Fabric as a product?

In this session, we’ll explore the answers to these questions and more in a thought-provoking, argument-generating examination of the challenges every data platform engineer/architect faces.

🚀 About Big Data and RPA 2024 🚀

Unlock the future of innovation and automation at Big Data & RPA Conference Europe 2024! 🌟 This unique event brings together the brightest minds in big data, machine learning, AI, and robotic process automation to explore cutting-edge solutions and trends shaping the tech landscape. Perfect for data engineers, analysts, RPA developers, and business leaders, the conference offers dual insights into the power of data-driven strategies and intelligent automation. 🚀 Gain practical knowledge on topics like hyperautomation, AI integration, advanced analytics, and workflow optimization while networking with global experts. Don’t miss this exclusive opportunity to expand your expertise and revolutionize your processes—all from the comfort of your home! 📊🤖✨

📅 Yearly Conferences: Curious about the evolution of QA? Check out our archive of past Big Data & RPA sessions. Watch the strategies and technologies evolve in our videos! 🚀 🔗 Find Other Years' Videos: 2023 Big Data Conference Europe https://www.youtube.com/playlist?list=PLqYhGsQ9iSEpb_oyAsg67PhpbrkCC59_g 2022 Big Data Conference Europe Online https://www.youtube.com/playlist?list=PLqYhGsQ9iSEryAOjmvdiaXTfjCg5j3HhT 2021 Big Data Conference Europe Online https://www.youtube.com/playlist?list=PLqYhGsQ9iSEqHwbQoWEXEJALFLKVDRXiP

💡 Stay Connected & Updated 💡

Don’t miss out on any updates or upcoming event information from Big Data & RPA Conference Europe. Follow us on our social media channels and visit our website to stay in the loop!

🌐 Website: https://bigdataconference.eu/, https://rpaconference.eu/ 👤 Facebook: https://www.facebook.com/bigdataconf, https://www.facebook.com/rpaeurope/ 🐦 Twitter: @BigDataConfEU, @europe_rpa 🔗 LinkedIn: https://www.linkedin.com/company/73234449/admin/dashboard/, https://www.linkedin.com/company/75464753/admin/dashboard/ 🎥 YouTube: http://www.youtube.com/@DATAMINERLT

In this session, discover how AI services, including Amazon Q Business, can help to scale and improve community engagement, streamline events planning, and handle everyday tasks as an event organizer. Dive into technical insights with a demo as we share practical ideas and offer guidance on getting started. Learn how to use AI and services like Amazon S3 and AWS Lambda. Building and growing communities is crucial. Whether you’re organizing monthly meetups or full-scale events, learn how to use these tools to work smarter and more efficiently.

Learn more: AWS re:Invent: https://go.aws/reinvent. More AWS events: https://go.aws/3kss9CP

Subscribe: More AWS videos: http://bit.ly/2O3zS75 More AWS events videos: http://bit.ly/316g9t4

About AWS: Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts. AWS is the world's most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

AWSreInvent #AWSreInvent2024

In this episode, Conor interviews Andor, Stephen and an attendee from Lambda World 2024. Link to Episode 204 on WebsiteDiscuss this episode, leave a comment, or ask a question (on GitHub)Twitter ADSP: The PodcastConor HoekstraGuests Interviewed Andor PénzesStephen TaylorShow Notes Date Recorded: 2024-10-04 Date Released: 2024-10-18 Lambda WorldADSP Episode 133: 🇵🇱 Lambda Days Live 🇵🇱 José Valim, Alexis King & More!Lambda World 2024 - The Butcherbird Combinator - Chris FordLambda World 2024 - Scala Sampler for Functional Soundscapes - Johanna OderskyUnite 2024 Barcelone (Unity Conference)Examples of easy dependently typed programming (in Idris) by Andor Penzes | Lambda Days 2023Dependently-Typed Python by Andor Penzes | Lambda Days 2024DepPy (Dependently Typed Python)CORECURSIVE #065 From Competitive Programming to APL With Conor HoekstraY CombinatorCategory Theory for Programmers - Bartosz MilewskiDevWorld ConferenceQCon ConferenceScala Days ConferenceLambda World 2024 - Stephen Taylor TalkAbove Average in APLDon't Be Mean in APLAPL Wiki MerchCan Programming Be Liberated from the von Neumann Style? John Backus Turing Award PaperLambda World 2024 - The Power of Function Composition - Conor HoekstraLambda World 2024 - Kamila Szewczyk TalkIntro Song Info Miss You by Sarah Jansen https://soundcloud.com/sarahjansenmusic Creative Commons — Attribution 3.0 Unported — CC BY 3.0 Free Download / Stream: http://bit.ly/l-miss-you Music promoted by Audio Library https://youtu.be/iYYxnasvfx8

Generative AI brings exciting new innovations, but it also presents challenges regarding responsible usage and compliance with governance requirements. This session guides you through the journey of a generative AI application and how AWS can help you ensure that your use of Amazon Bedrock and other related services, such as Amazon S3, AWS Lambda, and Amazon VPC, follows best practices for compliance and governance. Explore compliance services that AWS offers, like AWS Audit Manager and AWS CloudTrail, that can assist you in continuously auditing your generative AI infrastructure. Learn how these services automate audit evidence collection and provide audit-ready reports to meet your compliance and audit needs.

Learn more about AWS re:Inforce at https://go.aws/reinforce.

Subscribe: More AWS videos: http://bit.ly/2O3zS75 More AWS events videos: http://bit.ly/316g9t4

ABOUT AWS Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts.

AWS is the world's most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

reInforce2024 #CloudSecurity #AWS #AmazonWebServices #CloudComputing

Tired of chasing security threats by looking in many different places? Imagine a chatbot that understands security findings, prioritizes risks, and suggests solutions all through natural language. This session unveils how to create a conversational AI to get faster answers about your security posture. Learn how to build this interactive ChatSecOps tool using Amazon Q, AWS Lambda, Amazon S3, and AWS Security Hub.

Learn more about AWS re:Inforce at https://go.aws/reinforce.

Subscribe: More AWS videos: http://bit.ly/2O3zS75 More AWS events videos: http://bit.ly/316g9t4

ABOUT AWS Amazon Web Services (AWS) hosts events, both online and in-person, bringing the cloud computing community together to connect, collaborate, and learn from AWS experts.

AWS is the world's most comprehensive and broadly adopted cloud platform, offering over 200 fully featured services from data centers globally. Millions of customers—including the fastest-growing startups, largest enterprises, and leading government agencies—are using AWS to lower costs, become more agile, and innovate faster.

reInforce2024 #CloudSecurity #AWS #AmazonWebServices #CloudComputing

Summary

Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. One of the core complexities that needs to be addressed is the fractal set of integrations that need to be managed across the individual components. In this episode Tobias Macey shares his thoughts on the challenges that he is facing as he prepares to build the next set of architectural layers for his data platform to enable a larger audience to start accessing the data being managed by his team.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. With Materialize, you can! It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. Whether it’s real-time dashboarding and analytics, personalization and segmentation or automation and alerting, Materialize gives you the ability to work with fresh, correct, and scalable results — all in a familiar SQL interface. Go to dataengineeringpodcast.com/materialize today to get 2 weeks free! Developing event-driven pipelines is going to be a lot easier - Meet Functions! Memphis functions enable developers and data engineers to build an organizational toolbox of functions to process, transform, and enrich ingested events “on the fly” in a serverless manner using AWS Lambda syntax, without boilerplate, orchestration, error handling, and infrastructure in almost any language, including Go, Python, JS, .NET, Java, SQL, and more. Go to dataengineeringpodcast.com/memphis today to get started! Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics. Trusted by teams of all sizes, including Comcast and Doordash, Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Want to see Starburst in action? Go to dataengineeringpodcast.com/starburst and get $500 in credits to try Starburst Galaxy today, the easiest and fastest way to get started using Trino. Your host is Tobias Macey and today I'll be sharing an update on my own journey of building a data platform, with a particular focus on the challenges of tool integration and maintaining a single source of truth

Interview

Introduction How did you get involved in the area of data management? data sharing weight of history

existing integrations with dbt switching cost for e.g. SQLMesh de facto standard of Airflow

Single source of truth

permissions management across application layers Database engine Storage layer in a lakehouse Presentation/access layer (BI) Data flows dbt -> table level lineage orchestration engine -> pipeline flows

task based vs. asset based

Metadata platform as the logical place for horizontal view

Contact Info

LinkedIn Website

Parting Questio

dbt packages are libraries for dbt. Packages can produce information about best practice for your dbt project (ex: dbt project evaluator) and cloud warehouse cost overviews. Unfortunately, all theses KPIs are stored in your data warehouse and it can be painful and expensive to create data visualization dashboards. This application build automatically dashboards from dbt packages that you are using. You just need to parameter your dbt Cloud API key - that's it! In this session, you'll learn how.

Speaker: Adrien Boutreau, Head of Analytics Engineers , Infinite Lambda

Register for Coalesce at https://coalesce.getdbt.com

Testing is often the foremost challenge that newcomers face when venturing into serverless technologies. A common narrative suggests we have to choose between slow remote tests or grappling with the near-impossible task of simulating an entire AWS environment locally. But what if this narrative isn't the whole story? Imagine having a fast feedback loop and the capability to debug your code locally without leaning on local simulators. The good news? You can have the best of both worlds! Join this session to discover a straightforward approach that's applicable across any framework (be it Serverless, CDK, SAM, or others) and any programming language.

Yan is an experienced engineer who has run production workload at scale on AWS since 2010. He has been an architect and principal engineer in a variety of industries ranging from banking, e-commerce, sports streaming to mobile gaming. He has worked extensively with AWS Lambda in production since 2015. Nowadays, he splits his time between advancing the state of serverless observability as a Developer Advocate at lumigo.io and helping companies around the world adopt serverless as an independent consultant. Yan is also an AWS Serverless Hero and a regular speaker at user groups and conferences internationally. He is the author of Production-Ready Serverless and co-author of Serverless Architectures on AWS, 2nd Edition. And he keeps an active blog at theburningmonk.com and hosts a serverless-focused podcast at realworldserverless.com.