Real Time Intelligence is a transformative technology designed to empower businesses with real-time data insights and analytics. This session will cover the journey of Real Time Intelligence from its inception to its current state, highlighting key features, capabilities, and the impact it has on operational efficiency and decision-making processes. We will also highlight customer success stories and showcase how you can revolutionize your data strategy and unlock new opportunities for growth.

𝗦𝗽𝗲𝗮𝗸𝗲𝗿𝘀: * Yitzhak Kesselman * Tessa Kloster * Niraj Revankar * Vijay Sankaran * Ksenia Suresh

𝗦𝗲𝘀𝘀𝗶𝗼𝗻 𝗜𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻: This is one of many sessions from the Microsoft Ignite 2024 event. View even more sessions on-demand and learn about Microsoft Ignite at https://ignite.microsoft.com

BRK199 | English (US) | Data

MSIgnite

As AI reshapes software development, platform engineering teams must adopt AI-powered tools, including autonomous agents, or risk falling behind. Learn how Palo Alto Networks has transformed its platform engineering practice, using AI to autonomously execute 89,000+ production changes with zero incidents, saving thousands of engineering hours and $3.5M in cloud costs.

Discover Palo Alto Networks’ comprehensive transformation:

• Learn their framework for integrating AI across the software development lifecycle, from requirements to autonomous operations, empowering thousands of engineers with AI-powered tools
• How they're managing Kubernetes infrastructure at scale with autonomous agents, making optimization decisions without human intervention while ensuring reliability and performance
• Get a step-by-step guide to safely implementing autonomous agents in both dev/test and production environments, with a live demonstration of their approach

After a 30-minute talk there’ll be a 15-minute Q&A, for which we encourage you to submit questions in advance. A webinar recording and related materials will be shared with all attendees after the event.

Speakers: Ramesh Nampelly, Sr. Director of Cloud Infrastructure and Platform Engineering, Palo Alto Networks Ramesh is an experienced engineering leader driving user-centric solutions, cloud platforms, SRE, and infrastructure teams for cutting-edge technology products. Ramesh currently leads cloud infrastructure and platform engineering at Palo Alto Networks, where his team is responsible for building internal platforms to help developers and SREs enhance their productivity. Prior to joining Palo Alto Networks, Ramesh was the Head of Engineering Effectiveness at Cohesity.

Suresh Mathew, CEO and Founder, Sedai Suresh Mathew is the CEO and Founder of Sedai, an autonomous cloud management platform. Prior to Sedai, Suresh was a Sr. MTS Architect at Paypal where he built an AI/ML-driven platform that autonomously detected and resolved performance and availability issues, executing over 2 million remediations a year and becoming the only trusted system to operate independently in production during peak holiday traffic. During his career, he has contributed to multiple publications and holds patents in the field of infrastructure and distributed systems.

As AI reshapes software development, platform engineering teams must adopt AI-powered tools, including autonomous agents, or risk falling behind. Learn how Palo Alto Networks has transformed its platform engineering practice, using AI to autonomously execute 89,000+ production changes with zero incidents, saving thousands of engineering hours and $3.5M in cloud costs.

Discover Palo Alto Networks’ comprehensive transformation:

• Learn their framework for integrating AI across the software development lifecycle, from requirements to autonomous operations, empowering thousands of engineers with AI-powered tools
• How they're managing Kubernetes infrastructure at scale with autonomous agents, making optimization decisions without human intervention while ensuring reliability and performance
• Get a step-by-step guide to safely implementing autonomous agents in both dev/test and production environments, with a live demonstration of their approach

After a 30-minute talk there’ll be a 15-minute Q&A, for which we encourage you to submit questions in advance. A webinar recording and related materials will be shared with all attendees after the event.

Speakers: Ramesh Nampelly, Sr. Director of Cloud Infrastructure and Platform Engineering, Palo Alto Networks Ramesh is an experienced engineering leader driving user-centric solutions, cloud platforms, SRE, and infrastructure teams for cutting-edge technology products. Ramesh currently leads cloud infrastructure and platform engineering at Palo Alto Networks, where his team is responsible for building internal platforms to help developers and SREs enhance their productivity. Prior to joining Palo Alto Networks, Ramesh was the Head of Engineering Effectiveness at Cohesity.

Suresh Mathew, CEO and Founder, Sedai Suresh Mathew is the CEO and Founder of Sedai, an autonomous cloud management platform. Prior to Sedai, Suresh was a Sr. MTS Architect at Paypal where he built an AI/ML-driven platform that autonomously detected and resolved performance and availability issues, executing over 2 million remediations a year and becoming the only trusted system to operate independently in production during peak holiday traffic. During his career, he has contributed to multiple publications and holds patents in the field of infrastructure and distributed systems.

Summary In this episode of the Data Engineering Podcast, the creators of Feldera talk about their incremental compute engine designed for continuous computation of data, machine learning, and AI workloads. The discussion covers the concept of incremental computation, the origins of Feldera, and its unique ability to handle both streaming and batch data seamlessly. The guests explore Feldera's architecture, applications in real-time machine learning and AI, and challenges in educating users about incremental computation. They also discuss the balance between open-source and enterprise offerings, and the broader implications of incremental computation for the future of data management, predicting a shift towards unified systems that handle both batch and streaming data efficiently.

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data managementImagine catching data issues before they snowball into bigger problems. That’s what Datafold’s new Monitors do. With automatic monitoring for cross-database data diffs, schema changes, key metrics, and custom data tests, you can catch discrepancies and anomalies in real time, right at the source. Whether it’s maintaining data integrity or preventing costly mistakes, Datafold Monitors give you the visibility and control you need to keep your entire data stack running smoothly. Want to stop issues before they hit production? Learn more at dataengineeringpodcast.com/datafold today!As a listener of the Data Engineering Podcast you clearly care about data and how it affects your organization and the world. For even more perspective on the ways that data impacts everything around us you should listen to Data Citizens® Dialogues, the forward-thinking podcast from the folks at Collibra. You'll get further insights from industry leaders, innovators, and executives in the world's largest companies on the topics that are top of mind for everyone. They address questions around AI governance, data sharing, and working at global scale. In particular I appreciate the ability to hear about the challenges that enterprise scale businesses are tackling in this fast-moving field. While data is shaping our world, Data Citizens Dialogues is shaping the conversation. Subscribe to Data Citizens Dialogues on Apple, Spotify, Youtube, or wherever you get your podcasts.Your host is Tobias Macey and today I'm interviewing Leonid Ryzhyk, Lalith Suresh, and Mihai Budiu about Feldera, an incremental compute engine for continous computation of data, ML, and AI workloadsInterview IntroductionCan you describe what Feldera is and the story behind it?DBSP (the theory behind Feldera) has won multiple awards from the database research community. Can you explain what it is and how it solves the incremental computation problem?Depending on which angle you look at it, Feldera has attributes of data warehouses, federated query engines, and stream processors. What are the unique use cases that Feldera is designed to address?In what situations would you replace another technology with Feldera?When is it an additive technology?Can you describe the architecture of Feldera?How have the design and scope evolved since you first started working on it?What are the state storage interfaces available in Feldera?What are the opportunities for integrating with or building on top of open table formats like Iceberg, Lance, Hudi, etc.?Can you describe a typical workflow for an engineer building with Feldera?You advertise Feldera's utility in ML and AI use cases in addition to data management. What are the features that make it conducive to those applications?What is your philosophy toward the community growth and engagement with the open source aspects of Feldera and how you're balancing that with sustainability of the project and business?What are the most interesting, innovative, or unexpected ways that you have seen Feldera used?What are the most interesting, unexpected, or challenging lessons that

Until recently, Nik Suresh wrote under a mysterious blog that had several viral posts, including the famous "I Will F*cking Piledrive You If You Mention AI Again." For the longest time, he was an underground sensation, with nobody (not even his friends) knowing his identity.

In this episode, we chat about his blog posts (I'm a huge fan), the realities of data science and data engineering, and much more. This is a very candid and fun chat where I'm actually the fanboy, so enjoy!

Blog: https://ludic.mataroa.blog/

Send us a text Today we are joined by Suresh Srinivas and Harsha Chintalapani, co-founders of Collate. Collate is the company behind the open-source OpenMetadata.

Check out more on OpenMetadata here: open-metadata.org or getcollate.iovia SlackGithub Music from Uppbeat (free for Creators!): https://uppbeat.io/t/roo-walker/excusez-moi

Summary A significant source of friction and wasted effort in building and integrating data management systems is the fragmentation of metadata across various tools. After experiencing the impacts of fragmented metadata and previous attempts at building a solution Suresh Srinivas and Sriharsha Chintalapani created the OpenMetadata project. In this episode they share the lessons that they have learned through their previous attempts and the positive impact that a unified metadata layer had during their time at Uber. They also explain how the OpenMetadat project is aiming to be a common standard for defining and storing metadata for every use case in data platforms and the ways that they are architecting the reference implementation to simplify its adoption. This is an ambitious and exciting project, so listen and try it out today.

Announcements

Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show! Struggling with broken pipelines? Stale dashboards? Missing data? If this resonates with you, you’re not alone. Data engineers struggling with unreliable data need look no further than Monte Carlo, the world’s first end-to-end, fully automated Data Observability Platform! In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Monte Carlo monitors and alerts for data issues across your data warehouses, data lakes, ETL, and business intelligence, reducing time to detection and resolution from weeks or days to just minutes. Start trusting your data with Monte Carlo today! Visit dataengineeringpodcast.com/impact today to save your spot at IMPACT: The Data Observability Summit a half-day virtual event featuring the first U.S. Chief Data Scientist, founder of the Data Mesh, Creator of Apache Airflow, and more data pioneers spearheading some of the biggest movements in data. The first 50 to RSVP with this link will be entered to win an Oculus Quest 2 — Advanced All-In-One Virtual Reality Headset. RSVP today – you don’t want to miss it! Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? Hightouch is the easiest way to sync data into the platforms that your business teams rely on. The data you’re looking for is already in your data warehouse and BI tools. Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems. No more scripts, just SQL. Supercharge your business teams with customer data using Hightouch for Reverse ETL today. Get started for free at dataengineeringpodcast.com/hightouch. Your host is Tobias Macey and today I’m interviewing Sriharsha Chintalapani and Suresh Srinivas about OpenMetadata, an open standard for metadata and a reference implementation for a central metadata store

Interview

Introduction How did you get involved in the area of data management? Can you describe what the OpenMetadata project is and the story behind it?

What are the goals of the project?

What are the common challenges faced by engineers and data practitioners in organizing the metadata for their systems? What are the capabilities that a centralized and holis

INTERMITTENT DEMAND FORECASTING The first text to focus on the methods and approaches of intermittent, rather than fast, demand forecasting Intermittent Demand Forecasting is for anyone who is interested in improving forecasts of intermittent demand products, and enhancing the management of inventories. Whether you are a practitioner, at the sharp end of demand planning, a software designer, a student, an academic teaching operational research or operations management courses, or a researcher in this field, we hope that the book will inspire you to rethink demand forecasting. If you do so, then you can contribute towards significant economic and environmental benefits. No prior knowledge of intermittent demand forecasting or inventory management is assumed in this book. The key formulae are accompanied by worked examples to show how they can be implemented in practice. For those wishing to understand the theory in more depth, technical notes are provided at the end of each chapter, as well as an extensive and up-to-date collection of references for further study. Software developments are reviewed, to give an appreciation of the current state of the art in commercial and open source software. “Intermittent demand forecasting may seem like a specialized area but actually is at the center of sustainability efforts to consume less and to waste less. Boylan and Syntetos have done a superb job in showing how improvements in inventory management are pivotal in achieving this. Their book covers both the theory and practice of intermittent demand forecasting and my prediction is that it will fast become the bible of the field.” — Spyros Makridakis, Professor, University of Nicosia, and Director, Institute for the Future and the Makridakis Open Forecasting Center (MOFC). “We have been able to support our clients by adopting many of the ideas discussed in this excellent book, and implementing them in our software. I am sure that these ideas will be equally helpful for other supply chain software vendors and for companies wanting to update and upgrade their capabilities in forecasting and inventory management.” — Suresh Acharya, VP, Research and Development, Blue Yonder. “As product variants proliferate and the pace of business quickens, more and more items have intermittent demand. Boylan and Syntetos have long been leaders in extending forecasting and inventory methods to accommodate this new reality. Their book gathers and clarifies decades of research in this area, and explains how practitioners can exploit this knowledge to make their operations more efficient and effective.” — Thomas R. Willemain, Professor Emeritus, Rensselaer Polytechnic Institute.