Graham Hosking shares tips and tricks for building use-case specific agents and leveraging state-of-the-art AI features in the Microsoft Cloud.

Supported by Our Partners •⁠ Modal⁠ — The cloud platform for building AI applications •⁠ CodeRabbit⁠⁠ — Cut code review time and bugs in half. Use the code PRAGMATIC to get one month free. — What happens when LLMs meet real-world codebases? In this episode of The Pragmatic Engineer,  I am joined by Varun Mohan, CEO and Co-Founder of Windsurf. Varun talks me through the technical challenges of building an AI-native IDE (Windsurf) —and how these tools are changing the way software gets built.  We discuss:  • What building self-driving cars taught the Windsurf team about evaluating LLMs • How LLMs for text are missing capabilities for coding like “fill in the middle” • How Windsurf optimizes for latency • Windsurf’s culture of taking bets and learning from failure • Breakthroughs that led to Cascade (agentic capabilities) • Why the Windsurf teams build their LLMs • How non-dev employees at Windsurf build custom SaaS apps – with Windsurf! • How Windsurf empowers engineers to focus on more interesting problems • The skills that will remain valuable as AI takes over more of the codebase • And much more! — Timestamps (00:00) Intro (01:37) How Windsurf tests new models (08:25) Windsurf’s origin story  (13:03) The current size and scope of Windsurf (16:04) The missing capabilities Windsurf uncovered in LLMs when used for coding (20:40) Windsurf’s work with fine-tuning inside companies  (24:00) Challenges developers face with Windsurf and similar tools as codebases scale (27:06) Windsurf’s stack and an explanation of FedRAMP compliance (29:22) How Windsurf protects latency and the problems with local data that remain unsolved (33:40) Windsurf’s processes for indexing code  (37:50) How Windsurf manages data  (40:00) The pros and cons of embedding databases  (42:15) “The split brain situation”—how Windsurf balances present and long-term  (44:10) Why Windsurf embraces failure and the learnings that come from it (46:30) Breakthroughs that fueled Cascade (48:43) The insider’s developer mode that allows Windsurf to dogfood easily  (50:00) Windsurf’s non-developer power user who routinely builds apps in Windsurf (52:40) Which SaaS products won’t likely be replaced (56:20) How engineering processes have changed at Windsurf  (1:00:01) The fatigue that goes along with being a software engineer, and how AI tools can help (1:02:58) Why Windsurf chose to fork VS Code and built a plugin for JetBrains  (1:07:15) Windsurf’s language server  (1:08:30) The current use of MCP and its shortcomings  (1:12:50) How coding used to work in C#, and how MCP may evolve  (1:14:05) Varun’s thoughts on vibe coding and the problems non-developers encounter (1:19:10) The types of engineers who will remain in demand  (1:21:10) How AI will impact the future of software development jobs and the software industry (1:24:52) Rapid fire round — The Pragmatic Engineer deepdives relevant for this episode: • IDEs with GenAI features that Software Engineers love • AI tooling for Software Engineers in 2024: reality check • How AI-assisted coding will change software engineering: hard truths • AI tools for software engineers, but without the hype — See the transcript and other references from the episode at ⁠⁠https://newsletter.pragmaticengineer.com/podcast⁠⁠ — Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email [email protected].

Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

Apache Kafka, start to finish. Apache Kafka in Action: From basics to production guides you through the concepts and skills you’ll need to deploy and administer Kafka for data pipelines, event-driven applications, and other systems that process data streams from multiple sources. Authors Anatoly Zelenin and Alexander Kropp have spent years using Kafka in real-world production environments. In this guide, they reveal their hard-won expert insights to help you avoid common Kafka pitfalls and challenges. Inside Apache Kafka in Action you’ll discover: Apache Kafka from the ground up Achieving reliability and performance Troubleshooting Kafka systems Operations, governance, and monitoring Kafka use cases, patterns, and anti-patterns Clear, concise, and practical, Apache Kafka in Action is written for IT operators, software engineers, and IT architects working with Kafka every day. Chapter by chapter, it guides you through the skills you need to deliver and maintain reliable and fault-tolerant data-driven applications. About the Technology Apache Kafka is the gold standard streaming data platform for real-time analytics, event sourcing, and stream processing. Acting as a central hub for distributed data, it enables seamless flow between producers and consumers via a publish-subscribe model. Kafka easily handles millions of events per second, and its rock-solid design ensures high fault tolerance and smooth scalability. About the Book Apache Kafka in Action is a practical guide for IT professionals who are integrating Kafka into data-intensive applications and infrastructures. The book covers everything from Kafka fundamentals to advanced operations, with interesting visuals and real-world examples. Readers will learn to set up Kafka clusters, produce and consume messages, handle real-time streaming, and integrate Kafka into enterprise systems. This easy-to-follow book emphasizes building reliable Kafka applications and taking advantage of its distributed architecture for scalability and resilience. What's Inside Master Kafka’s distributed streaming capabilities Implement real-time data solutions Integrate Kafka into enterprise environments Build and manage Kafka applications Achieve fault tolerance and scalability About the Reader For IT operators, software architects and developers. No experience with Kafka required. About the Authors Anatoly Zelenin is a Kafka expert known for workshops across Europe, especially in banking and manufacturing. Alexander Kropp specializes in Kafka and Kubernetes, contributing to cloud platform design and monitoring. Quotes A great introduction. Even experienced users will go back to it again and again. - Jakub Scholz, Red Hat Approachable, practical, well-illustrated, and easy to follow. A must-read. - Olena Kutsenko, Confluent A zero to hero journey to understanding and using Kafka! - Anthony Nandaa, Microsoft Thoughtfully explores a wide range of topics. A wealth of valuable information seamlessly presented and easily accessible. - Olena Babenko, Aiven Oy

In this episode of Data Unchained, host Molly Presley sits down with Brett Ferrin, VP of Sales at Traffic Logix, to uncover how a company that once molded rubber products evolved into a cloud-based, data-driven traffic safety powerhouse. From speed cameras and pedestrian tracking to AI-enabled traffic insights, Brett shares how Traffic Logix is leveraging distributed data systems to make roads safer while navigating privacy regulations and digital transformation.

DataUnchained #TrafficLogix #SmartCities #DataDrivenDecisions #CloudComputing #AIAnalytics #EdgeToCloud #IoTInfrastructure #CIO #CTO #DigitalTransformation #MobilityTech #Hammerspace #DataStrategy #PublicSafetyTech

Cyberpunk by jiglr | https://soundcloud.com/jiglrmusic Music promoted by https://www.free-stock-music.com Creative Commons Attribution 3.0 Unported License https://creativecommons.org/licenses/by/3.0/deed.en_US Hosted on Acast. See acast.com/privacy for more information.

This blog offers five guiding principles to help CIOs, CDOs, and team leaders optimize hybrid data environments. Published at: https://www.eckerson.com/articles/balancing-act-five-principles-to-optimize-hybrid-cloud-environments

Summary In this episode of the Data Engineering Podcast Derek Collison, creator of NATS and CEO of Synadia, talks about the evolution and capabilities of NATS as a multi-paradigm connectivity layer for distributed applications. Derek discusses the challenges and solutions in building distributed systems, and highlights the unique features of NATS that differentiate it from other messaging systems. He delves into the architectural decisions behind NATS, including its ability to handle high-speed global microservices, support for edge computing, and integration with Jetstream for data persistence, and explores the role of NATS in modern data management and its use cases in industries like manufacturing and connected vehicles.

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data managementData migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.Your host is Tobias Macey and today I'm interviewing Derek Collison about NATS, a multi-paradigm connectivity layer for distributed applications.Interview IntroductionHow did you get involved in the area of data management?Can you describe what NATS is and the story behind it?How have your experiences in past roles (cloud foundry, TIBCO messaging systems) informed the core principles of NATS?What other sources of inspiration have you drawn on in the design and evolution of NATS? (e.g. Kafka, RabbitMQ, etc.)There are several patterns and abstractions that NATS can support, many of which overlap with other well-regarded technologies. When designing a system or service, what are the heuristics that should be used to determine whether NATS should act as a replacement or addition to those capabilities? (e.g. considerations of scale, speed, ecosystem compatibility, etc.)There is often a divide in the technologies and architecture used between operational/user-facing applications and data systems. How does the unification of multiple messaging patterns in NATS shift the ways that teams think about the relationship between these use cases?How does the shared communication layer of NATS with multiple protocol and pattern adaptaters reduce the need to replicate data and logic across application and data layers?Can you describe how the core NATS system is architected?How have the design and goals of NATS evolved since you first started working on it?In the time since you first began writing NATS (~2012) there have been several evolutionary stages in both application and data implementation patterns. How have those shifts influenced the direction of the NATS project and its ecosystem?For teams who have an existing architecture, what are some of the patterns for adoption of NATS that allow them to augment or migrate their capabilities?What are some of the ecosystem investments that you and your team have made to ease the adoption and integration of NATS?What are the most interesting, innovative, or unexpected ways that you have seen NATS used?What are the most interesting, unexpected, or challenging lessons that you have learned while working on NATS?When is NATS the wrong choice?What do you have planned for the future of NATS?Contact Info GitHubLinkedInParting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.init covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.Links NATSNATS JetStreamSynadiaCloud FoundryTIBCOApplied Physics Lab - Johns Hopkins UniversityCray SupercomputerRVCM Certified MessagingTIBCO ZMSIBM MQJMS == Java Message ServiceRabbitMQMongoDBNodeJSRedisAMQP == Advanced Message Queueing ProtocolPub/Sub PatternCircuit Breaker PatternZero MQAkamaiFastlyCDN == Content Delivery NetworkAt Most OnceAt Least OnceExactly OnceAWS KinesisMemcachedSQSSegmentRudderstackPodcast EpisodeDLQ == Dead Letter QueueMQTT == Message Queueing Telemetry TransportNATS Kafka Bridge10BaseT NetworkWeb AssemblyRedPandaPodcast EpisodePulsar FunctionsmTLSAuthZ (Authorization)AuthN (Authentication)NATS Auth CalloutsOPA == Open Policy AgentRAG == Retrieval Augmented GenerationAI Engineering Podcast EpisodeHome AssistantPodcast.init EpisodeTailscaleOllamaCDC == Change Data CapturegRPCThe intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Amazon Redshift Cookbook provides practical techniques for utilizing AWS's managed data warehousing service effectively. With this book, you'll learn to create scalable and secure data analytics solutions, tackle data integration challenges, and leverage Redshift's advanced features like data sharing and generative AI capabilities. What this Book will help me do Create end-to-end data analytics solutions from ingestion to reporting using Amazon Redshift. Optimize the performance and security of Redshift implementations to meet enterprise standards. Leverage Amazon Redshift for zero-ETL ingestion and advanced concurrency scaling. Integrate Redshift with data lakes for enhanced data processing versatility. Implement generative AI and machine learning solutions directly within Redshift environments. Author(s) Shruti Worlikar, Harshida Patel, and Anusha Challa are seasoned data experts who bring together years of experience with Amazon Web Services and data analytics. Their combined expertise enables them to offer actionable insights, hands-on recipes, and proven strategies for implementing and optimizing Amazon Redshift-based solutions. Who is it for? This book is best suited for data analysts, data engineers, and architects who are keen on mastering modern data warehouse solutions using Redshift. Readers should have some knowledge of data warehousing and familiarity with cloud concepts. Ideal for professionals looking to migrate on-premises systems or build cloud-native analytics pipelines leveraging Redshift.

Supported by Our Partners • WorkOS — The modern identity platform for B2B SaaS. •⁠ Modal⁠ — The cloud platform for building AI applications • Vanta — Automate compliance and simplify security with Vanta. — What is it like to work at Amazon as a software engineer? Dave Anderson spent over 12 years at Amazon working closely with engineers on his teams: starting as an Engineering Manager (or, SDM in Amazon lingo) and eventually becoming a Director of Engineering. In this episode, he shares a candid look into Amazon’s engineering culture—from how promotions work to why teams often run like startups. We get into the hiring process, the role of bar raisers, the pros and cons of extreme frugality, and what it takes to succeed inside one of the world’s most operationally intense companies.  We also look at how engineering actually works day to day at Amazon—from the tools teams choose to the way they organize and deliver work.  We also discuss: • The levels at Amazon, from SDE L4 to Distinguished Engineer and VP • Why engineering managers at Amazon need to write well • The “Bar Raiser” role in Amazon interview loops  • Why Amazon doesn’t care about what programming language you use in interviews • Amazon’s oncall process • The pros and cons of Amazon’s extreme frugality  • What to do if you're getting negative performance feedback • The importance of having a strong relationship with your manager • The surprising freedom Amazon teams have to choose their own stack, tools, and ways of working – and how a team chose to use Lisp (!) • Why startups love hiring former Amazon engineers • Dave’s approach to financial independence and early retirement • And more! — Timestamps (00:00) Intro (02:08) An overview of Amazon’s levels for devs and engineering managers (07:04) How promotions work for developers at Amazon, and the scope of work at each level (12:29) Why managers feel pressure to grow their teams (13:36) A step-by-step, behind-the-scenes glimpse of the hiring process  (23:40) The wide variety of tools used at Amazon (26:27) How oncall works at Amazon (32:06) The general approach to handling outages (severity 1-5) (34:40) A story from Uber illustrating the Amazon outage mindset (37:30) How VPs assist with outages (41:38) The culture of frugality at Amazon   (47:27) Amazon’s URA target—and why it’s mostly not a big deal  (53:37) How managers handle the ‘least effective’ employees (58:58) Why other companies are also cutting lower performers (59:55) Dave’s advice for engineers struggling with performance feedback  (1:04:20) Why good managers are expected to bring talent with them to a new org (1:06:21) Why startups love former Amazon engineers (1:16:09) How Dave planned for an early retirement  (1:18:10) How a LinkedIn post turned into Scarlet Ink  — The Pragmatic Engineer deepdives relevant for this episode: • Inside Amazon’s engineering culture • A day in the life of a senior manager at Amazon • Amazon’s Operational Plan process with OP1 and OP2 — See the transcript and other references from the episode at ⁠⁠https://newsletter.pragmaticengineer.com/podcast⁠⁠ — Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email [email protected].

Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

Hammerspace just made headlines with its game-changing funding round! Altimeter’s Jamin Ball and Hammerspace CEO David Flynn, join us on this episode of Data Unchained to talk about Hammerspace's go-to-market (GTM) strategy. We also discuss the future of distributed data, how Hammerspace is powering performance across multi-cloud, on-prem, and hyperscale systems with zero migration pain, and how David’s pioneering legacy in NVMe laid the groundwork for the next wave of AI-driven data architecture and why now is Hammerspace’s moment. Cyberpunk by jiglr | https://soundcloud.com/jiglrmusic Music promoted by https://www.free-stock-music.com Creative Commons Attribution 3.0 Unported License https://creativecommons.org/licenses/by/3.0/deed.en_US

DataUnchained #Hammerspace #AIInfrastructure #EnterpriseAI #DataArchitecture #CloudComputing #HybridCloud #MulticloudStrategy #DataStorage #AltimeterCapital #TechInnovation #FutureOfData #DataOrchestration #GlobalNamespace #GPUEngineering #AIPodcast #TechPodcast #AIInvestments #DigitalTransformation #EdgeComputing

Hosted on Acast. See acast.com/privacy for more information.

Data projects are an intrinsic part of an organization's technical ecosystem, but data engineers in many companies continue to work on problems that others have already solved. This hands-on guide shows you how to provide valuable data by focusing on various aspects of data engineering, including data ingestion, data quality, idempotency, and more. Author Bartosz Konieczny guides you through the process of building reliable end-to-end data engineering projects, from data ingestion to data observability, focusing on data engineering design patterns that solve common business problems in a secure and storage-optimized manner. Each pattern includes a user-facing description of the problem, solutions, and consequences that place the pattern into the context of real-life scenarios. Throughout this journey, you'll use open source data tools and public cloud services to apply each pattern. You'll learn: Challenges data engineers face and their impact on data systems How these challenges relate to data system components Useful applications of data engineering patterns How to identify and fix issues with your current data components Technology-agnostic solutions to new and existing data projects, with open source implementation examples Bartosz Konieczny is a freelance data engineer who's been coding since 2010. He's held various senior hands-on positions that allowed him to work on many data engineering problems in batch and stream processing.

Just wrapped up a whirlwind tour, giving a workshop in Atlanta and then attending Google Cloud Next. B2b nonstop action, and I'm glad to home for a bit.

While at Next, I had a conversation with another tech old timer friend. We talked about how much we're having using AI as a coding assistant. I'm having fun coming up with wild stuff and seeing if it's possible to build with code. AI's made coding fun again!

📈 This episode is brought to you by GoodData. Design and deploy custom data applications and integrate AI-assisted analytics capabilities wherever your users need them.

For more information, visit https://www.gooddata.com

AI Hypercomputer is a revolutionary system designed to make implementing AI at scale easier and more efficient. In this session, we’ll explore the key benefits of AI Hypercomputer and how it simplifies complex AI infrastructure environments. Then, learn firsthand from industry leaders Shopify, Technology Innovation Institute, Moloco, and LG AI Research on how they leverage Google Cloud’s AI solutions to drive innovation and transform their businesses.

APIs dominate the web, accounting for the majority of all internet traffic. And more AI means more APIs, because they act as an important mechanism to move data into and out of AI applications, AI agents, and large language models (LLMs). So how can you make sure all of these APIs are secure? In this session, we’ll take you through OWASP’s top 10 API and LLM security risks, and show you how to mitigate these risks using Google Cloud’s security portfolio, including Apigee, Model Armor, Cloud Armor, Google Security Operations, and Security Command Center.

This panel explores the potential of cloud technologies for operational technology (OT) and the critical need for proactive cybersecurity measures. The convergence of OT and IT, driven by cloud adoption, presents both opportunities and challenges. Panelists will examine the benefits of cloud-based OT, such as increased efficiency, scalability, data-driven insights, and resilience, along with the opportunity to build with security in mind.

Migrating from AWS or Azure to Google Cloud runtimes can feel like navigating a maze of complex services and dependencies. In this session, we’ll explore key considerations for migrating legacy applications, emphasizing the “why not modernize?” approach with a practice guide. We’ll share real-world examples of successful transformations. And we’ll go beyond theory with a live product demo that showcases migration tools, and a code assessment demo powered by Gemini that demonstrates how you can understand and modernize legacy code.

JavaScript gets a lot of flak for not being strongly typed. But if you’re running JavaScript in production today, you don’t need to wait for runtime errors to catch problems. TypeScript has taken JavaScript from a loosely typed language, where a variable can change from a string to a number without warning, and made it strongly typed. Now Zod and Effect are here to tame even the wildest unknown parameters from your users. We’ll demonstrate using these tools in an application and we’ll deploy that application to Google Cloud.

Learn how to leverage Google Cloud’s innovations to build the most secure and compliant solutions with Confidential Computing, offloads, and Assured Workload capabilities.

Simplify real-time data analytics and build event-driven, AI-powered applications using BigQuery and Pub/Sub. Learn to ingest and process massive streaming data from users, devices, and microservices for immediate insights and rapid action. Explore BigQuery's continuous queries for real-time analytics and ML model training. Discover how Flipkart, India’s leading e-commerce platform, leverages Google Cloud to build scalable, efficient real-time data pipelines and AI/ML solutions, and gain insights on driving business value through real–time data.