PostgreSQL is highly favored by developers and enterprises for its extensibility and scalability. Azure Database for PostgreSQL is introducing new innovations in scalability, performance, and optimizations, allowing developers to focus more on application logic and less on tuning and optimization. In this demo-rich session, discover all the latest enhancements to Azure Database for PostgreSQL that enable building mission critical applications.

𝗦𝗽𝗲𝗮𝗸𝗲𝗿𝘀: * Guy Bowerman * Denzil Ribeiro

𝗦𝗲𝘀𝘀𝗶𝗼𝗻 𝗜𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻: This is one of many sessions from the Microsoft Ignite 2024 event. View even more sessions on-demand and learn about Microsoft Ignite at https://ignite.microsoft.com

BRK191 | English (US) | Data

MSIgnite

The success of GenAI apps is decided by the accuracy of their responses. Using Retrieval Augmented Generation (RAG), you can improve accuracy by grounding GenAI app responses in your data. In this session, explore advanced RAG techniques in Azure Database for PostgreSQL including new vector search algorithms, parameter tuning, hybrid search, semantic ranking, and the GraphRAG approach. See how customers are using these techniques to deploy corporate development platform for GenAI apps.

𝗦𝗽𝗲𝗮𝗸𝗲𝗿𝘀: * Maxim Lukiyanov * Orhun Oezbek * Jay Yang

𝗦𝗲𝘀𝘀𝗶𝗼𝗻 𝗜𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻: This is one of many sessions from the Microsoft Ignite 2024 event. View even more sessions on-demand and learn about Microsoft Ignite at https://ignite.microsoft.com

BRK190 | English (US) | Data

MSIgnite

Send us a text Welcome to the cozy corner of the tech world where ones and zeros mingle with casual chit-chat. Datatopics Unplugged  is your go-to spot for relaxed discussions around tech, news, data, and society. Dive into conversations that should flow as smoothly as your morning coffee (but don't), where industry insights meet laid-back banter. Whether you're a data aficionado or just someone curious about the digital age, pull up a chair, relax, and let's get into the heart of data, unplugged style! In this episode, we cover: ChatGPT Search: Exploring OpenAI's new web-browsing capability, and how it transforms everything from everyday searches to complex problem-solving.ChatGPT is a Good Rubber Duck: Discover how ChatGPT makes for an excellent companion for debugging and brainstorming, offering more than a few laughs along the way.What’s New in Python 3.13: From the new free-threaded mode to the just-in-time (JIT) compiler, we break down the major (and some lesser-known) changes, with additional context from this breakdown and Reddit insights.UV is Fast on its Feet: How the development of new tools impacts the Python packaging ecosystem, with a side discussion on Poetry and the complexities of Python lockfiles.Meta’s Llama Training Takes Center Stage: Meta ramps up its AI game, pouring vast resources into training the Llama model. We ponder the long-term impact and their ambitions in the AI space.OpenAI’s Swarm: A new experimental framework for multi-agent orchestration, enabling AI agents to collaborate and complete tasks—what it means for the future of AI interactions.PGrag for Retrieval-Augmented Generation (RAG): We explore Neon's integration for building end-to-end RAG pipelines directly in Postgres, bridging vector databases, text embedding, and more.OSI’s Open Source AI License: The Open Source Initiative releases an AI-specific license to bring much-needed clarity and standards to open-source models.We also venture into generative AI, the future of AR (including Apple Vision and potential contact lenses), and a brief look at V0 by Vercel, a tool that auto-generates web components with AI prompts.

In the world of observability, metrics and logs are the usual suspects for monitoring system health and diagnosing issues. But what happens when you don't know what to look for in advance? We tackle this challenge by incorporating business-critical events into our observability stack. Join me for this talk as I delve into how events can fill the gaps left by traditional metrics and logs. I'll share our journey in identifying which events are worth storing and how our technical setup evolved from periodic PostgreSQL pulls to real-time streaming with AWS Firehose. You'll see real-world examples through our Grafana dashboards and learn how this approach allows us to perform ad-hoc analyses spanning over two years without incurring huge costs.

In the fast-paced world of mortgage lending, speed and accuracy are crucial. To support their underwriters, Vontive transformed written rules for loan eligibility from a Google Doc into SQL queries for evaluation in a Postgres database. However, while functional, this setup struggled to scale with business growth, resulting in slow, cumbersome processing times. Executing just a handful of loan eligibility rules could take up to 27 seconds–far too long for user-friendly interactions.

In this session, we’ll explore how Vontive reimagined its underwriting operations using Materialize. By offloading complex SQL queries from Postgres to Materialize, Vontive reduced eligibility check times from 27 seconds to under a second. This not only sped up decision-making but also removed limitations on the number of SQL-based underwriting rules, allowing underwriters to process more loans with greater accuracy and confidence. Additionally, this shift enabled the team to implement more automated checks throughout the underwriting process, catching errors earlier and further streamlining operations. Engineering needs were minimal, since DBT supports both cloud-based Postgres and Materialize.

Whether you're in financial services or any data-driven industry, this session offers valuable insights into leveraging fast-changing data for high-stakes decision-making with confidence.

Speakers: Steffen Hausmann Field Engineer Materialize

Wolf Rendall Director of Data Products Vontive

Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements

SpotOn works with FIS (formerly WorldPay) to handle payment processing, allowing for more detailed transaction management than other processors. Our data team took on the challenge of transitioning to FIS to gain better control over transaction details.

The legacy data pipelines we inherited were problematic and unreliable. They consisted of an SFTP file server, cron jobs, and Python/Shell scripts that moved data from SFTP to S3 and then processed it into Postgres. These systems were fragile, often breaking when new or different data arrived, requiring manual intervention and frequent restarts.

We recognized the need for a better solution. Our team decided to use Snowpipe and dbt to streamline our data processing. This approach allowed us to manage and parse complex data formats efficiently. We used dbt to create models that could handle the varied and detailed specifications provided by FIS, ensuring that as updates came in, they could be easily integrated.

With this new setup, we have significantly reduced the fragility of our pipelines. Using dbt Cloud, we've improved collaboration and error detection, ensuring data integrity and better insights into usage patterns. This new system supports not only payment processing but also other critical functions like customer loyalty and marketing, aggregating and cleaning data from various sources.

As we continue migrating from older systems like TSYS, we see the clear benefits of this modernization. Our experience with dbt has proven invaluable in supporting our business-critical data operations and ensuring smooth transitions and reliable data handling.

Speakers: Kevin Hu CEO Metaplane

Daniel Corley Senior Analytics Engineer SpotOn

Read the blog to learn about the latest dbt Cloud features announced at Coalesce, designed to help organizations embrace analytics best practices at scale https://www.getdbt.com/blog/coalesce-2024-product-announcements

Simplify your GenAI journey and unlock the hidden power within your databases. Businesses often feel pressured to adopt new, specialized technologies to stay ahead. However, the power to revolutionize your applications with GenAI may already reside within your current database infrastructure. 

We’ll build understanding of vector capabilities, ease of use/ROI, and how PostgreSQL, enhanced with the pgvector extension, can address 80% of common GenAI use cases, providing a streamlined and cost-effective path to AI-driven solutions.

Join us to demystify the hype around dedicated vector databases and explore how built-in vector capabilities existing databases can efficiently support your GenAI workloads without extra overhead.

The massive interest in AI solutions has sparked a huge and pervasive wave of AI projects. We are now entering a second phase where the AI projects that have proven value are looking for operational landing places in enterprise environments. This is visible through the big hype for AI data systems like vector databases and feature stores. This phase of AI operationalizing is the hour of databases, which have proven already to be the battle-proof bedrock for data management enterprise environments. 

Postgres is naturally a front runner in this space. AI workloads are entirely tied to data, they start with data, they run on data and they produce data. Join this talk for a walkthrough on popular AI application flows, their strong ties to data and Postgres' strong operational qualities and demonstrate how they form the perfect environment for mission critical AI solutions in an enterprise.

An intro to RAGHack, a global hackathon to develop apps using LLMs and RAG. A large language model (LLM) like GPT-4 can be used for summarization, translation, entity extraction, and question-answering. Retrieval Augmented Generation (RAG) is an approach that sends context to the LLM so that it can provide grounded answers. RAG apps can be developed on Azure using a wide range of programming languages and retrievers (such as AI Search, Cosmos DB, PostgreSQL, and Azure SQL). Get an overview of RAG in this session before diving deep in our follow-up streams.

Una introducción a RAGHack, un hackathon global para desarrollar aplicaciones utilizando modelos de lenguaje grandes (LLM) y RAG. Se presentarán conceptos de RAG y se demostrará una aplicación RAG de muestra que se puede empezar a usar hoy.

Dive into DuckDB and start processing gigabytes of data with ease—all with no data warehouse. DuckDB is a cutting-edge SQL database that makes it incredibly easy to analyze big data sets right from your laptop. In DuckDB in Action you’ll learn everything you need to know to get the most out of this awesome tool, keep your data secure on prem, and save you hundreds on your cloud bill. From data ingestion to advanced data pipelines, you’ll learn everything you need to get the most out of DuckDB—all through hands-on examples. Open up DuckDB in Action and learn how to: Read and process data from CSV, JSON and Parquet sources both locally and remote Write analytical SQL queries, including aggregations, common table expressions, window functions, special types of joins, and pivot tables Use DuckDB from Python, both with SQL and its "Relational"-API, interacting with databases but also data frames Prepare, ingest and query large datasets Build cloud data pipelines Extend DuckDB with custom functionality Pragmatic and comprehensive, DuckDB in Action introduces the DuckDB database and shows you how to use it to solve common data workflow problems. You won’t need to read through pages of documentation—you’ll learn as you work. Get to grips with DuckDB's unique SQL dialect, learning to seamlessly load, prepare, and analyze data using SQL queries. Extend DuckDB with both Python and built-in tools such as MotherDuck, and gain practical insights into building robust and automated data pipelines. About the Technology DuckDB makes data analytics fast and fun! You don’t need to set up a Spark or run a cloud data warehouse just to process a few hundred gigabytes of data. DuckDB is easily embeddable in any data analytics application, runs on a laptop, and processes data from almost any source, including JSON, CSV, Parquet, SQLite and Postgres. About the Book DuckDB in Action guides you example-by-example from setup, through your first SQL query, to advanced topics like building data pipelines and embedding DuckDB as a local data store for a Streamlit web app. You’ll explore DuckDB’s handy SQL extensions, get to grips with aggregation, analysis, and data without persistence, and use Python to customize DuckDB. A hands-on project accompanies each new topic, so you can see DuckDB in action. What's Inside Prepare, ingest and query large datasets Build cloud data pipelines Extend DuckDB with custom functionality Fast-paced SQL recap: From simple queries to advanced analytics About the Reader For data pros comfortable with Python and CLI tools. About the Authors Mark Needham is a blogger and video creator at @‌LearnDataWithMark. Michael Hunger leads product innovation for the Neo4j graph database. Michael Simons is a Java Champion, author, and Engineer at Neo4j. Quotes I use DuckDB every day, and I still learned a lot about how DuckDB makes things that are hard in most databases easy! - Jordan Tigani, Founder, MotherDuck An excellent resource! Unlocks possibilities for storing, processing, analyzing, and summarizing data at the edge using DuckDB. - Pramod Sadalage, Director, Thoughtworks Clear and accessible. A comprehensive resource for harnessing the power of DuckDB for both novices and experienced professionals. - Qiusheng Wu, Associate Professor, University of Tennessee Excellent! The book all we ducklings have been waiting for! - Gunnar Morling, Decodable

Discover how to design and optimize modern databases efficiently using PostgreSQL and MySQL. This book guides you through database design for scalability and performance, covering data modeling, query optimization, and real-world application integration. What this Book will help me do Build efficient and scalable relational database schemas for real-world applications. Master data modeling with normalization and denormalization techniques. Understand query optimization strategies for better database performance. Learn database strategies such as sharding, replication, and backup management. Integrate relational databases with applications and explore future database trends. Author(s) Alkin Tezuysal and Ibrar Ahmed are seasoned database professionals with decades of experience. Alkin specializes in database scalability and performance, while Ibrar brings expertise in database systems and development. Together, they bring a hands-on approach, providing clear and insightful guidance for database professionals. Who is it for? This book is oriented towards software developers, database administrators, and IT professionals looking to enhance their knowledge in database design using PostgreSQL and MySQL. Beginners in database design will find its structured approach approachable. Advanced professionals will appreciate its depth on cutting-edge topics and practical optimizations.

Summary Postgres is one of the most widely respected and liked database engines ever. To make it even easier to use for developers to use, Nikita Shamgunov decided to makee it serverless, so that it can scale from zero to infinity. In this episode he explains the engineering involved to make that possible, as well as the numerous details that he and his team are packing into the Neon service to make it even more attractive for anyone who wants to build on top of Postgres. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data managementData lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake. Trusted by teams of all sizes, including Comcast and Doordash. Want to see Starburst in action? Go to dataengineeringpodcast.com/starburst and get $500 in credits to try Starburst Galaxy today, the easiest and fastest way to get started using Trino.Your host is Tobias Macey and today I'm interviewing Nikita Shamgunov about his work on making Postgres a serverless database at Neon.Interview IntroductionHow did you get involved in the area of data management?Can you describe what Neon is and the story behind it?The ecosystem around Postgres is large and varied. What are the pain points that you are trying to address with Neon? What does it mean for a database to be serverless?What kinds of products and services are unlocked by making Postgres a serverless database?How does your vision for Neon compare/contrast with what you know of PlanetScale?Postgres is known for having a large ecosystem of plugins that add a lot of interesting and useful features, but the storage layer has not been as easily extensible historically. How have architectural changes in recent Postgres releases enabled your work on Neon?What are the core pieces of engineering that you have had to complete to make Neon possible?How have the design and goals of the project evolved since you first started working on it?The separation of storage and compute is one of the most fundamental promises of the cloud. What new capabilities does that enable in Postgres?How does the branching functionality change the ways that development teams are able to deliver and debug features?Because the storage is now a networked system, what new performance/latency challenges does that introduce? How have you addressed them in Neon?Anyone who has ever operated a Postgres instance has had to tackle the upgrade process. How does Neon address that process for end users?The rampant growth of AI has touched almost every aspect of computing, and Postgres is no exception. How does the introduction of pgvector and semantic/similarity search functionality impact the adoption and usage patterns of Postgres/Neon?What new challenges does that introduce for you as an operator and business owner?What are the lessons that you learned from MemSQL/SingleStore that have been most helpful in your work at Neon?What are the most interesting, innovative, or unexpected ways that you have seen Neon used?What are the most interesting, unexpected, or challenging lessons that you have learned while working on Neon?When is Neon the wrong choice? Postgres?What do you have planned for the future of Neon?Contact Info @nikitabase on TwitterLinkedInParting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.init covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story.Links NeonPostgreSQLNeon GithubPHPMySQLSQL ServerSingleStorePodcast EpisodeAWS AuroraKhosla VenturesYugabyteDBPodcast EpisodeCockroachDBPodcast EpisodePlanetScalePodcast EpisodeClickhousePodcast EpisodeDuckDBPodcast EpisodeWAL == Write-Ahead LogPgBouncerPureStoragePaxos)HNSW IndexIVF Flat IndexRAG == Retrieval Augmented GenerationAlloyDBNeon Serverless DriverDevinmagic.devThe intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

Build faster, more reliable Rails apps by taking the best advanced PostgreSQL and Active Record capabilities, and using them to solve your application scale and growth challenges. Gain the skills needed to comfortably work with multi-terabyte databases, and with complex Active Record, SQL, and specialized Indexes. Develop your skills with PostgreSQL on your laptop, then take them into production, while keeping everything in sync. Make slow queries fast, perform any schema or data migration without errors, use scaling techniques like read/write splitting, partitioning, and sharding, to meet demanding workload requirements from Internet scale consumer apps to enterprise SaaS. Deepen your firsthand knowledge of high-scale PostgreSQL databases and Ruby on Rails applications with dozens of practical and hands-on exercises. Unlock the mysteries surrounding complex Active Record. Make any schema or data migration change confidently, without downtime. Grow your experience with modern and exclusive PostgreSQL features like SQL Merge, Returning, and Exclusion constraints. Put advanced capabilities like Full Text Search and Publish Subscribe mechanisms built into PostgreSQL to work in your Rails apps. Improve the quality of the data in your database, using the advanced and extensible system of types and constraints to reduce and eliminate application bugs. Tackle complex topics like how to improve query performance using specialized indexes. Discover how to effectively use built-in database functions and write your own, administer replication, and make the most of partitioning and foreign data wrappers. Use more than 40 well-supported open source tools to extend and enhance PostgreSQL and Ruby on Rails. Gain invaluable insights into database administration by conducting advanced optimizations - including high-impact database maintenance - all while solving real-world operational challenges. Take your new skills into production today and then take your PostgreSQL and Rails applications to a whole new level of reliability and performance. What You Need: A computer running macOS, Linux, or Windows and WSL2 PostgreSQL version 16, installed by package manager, compiled, or running with Docker An Internet connection

Send us a text Welcome to the cozy corner of the tech world where ones and zeros mingle with casual chit-chat. Datatopics Unplugged is your go-to spot for relaxed discussions around tech, news, data, and society. Dive into conversations that should flow as smoothly as your morning coffee (but don't), where industry insights meet laid-back banter. Whether you're a data aficionado or just someone curious about the digital age, pull up a chair, relax, and let's get into the heart of data, unplugged style! In this episode, we dive deep into the fascinating and complex world of AI with our special guest, Senne Batsleer: De Mol + AI Voices: Exploring the use of AI-generated voices to disguise the mole in the Belgian TV show "The Mole". Our guest, Senne Batsleer, shares insights from their experience with AI voice technology. Scarlett Johansson vs OpenAI: Delving into the controversy of OpenAI using a voice eerily similar to Scarlett Johansson's in their new AI model. Read more in The Guardian and The Washington Post. Elon Musk’s xAI Raises $6B: A look into Elon Musk’s latest venture, xAI, and its ambitious funding round, aiming to challenge AI giants like OpenAI and Microsoft. OpenAI and News Corp’s $250M Deal: The implications of OpenAI’s data deal with News Corp.  Google AI Search Risks: Examining Google's AI search providing potentially dangerous answers based on outdated Reddit comments. Find out more on The Verge and BBC.  Humane’s AI Pin Looking for a Buyer: Discussing the struggles of Humane’s wearable AI device and its search for a buyer following a rocky debut. PostgREST Turns Databases into APIs: Exploring the concept of turning PostgreSQL databases directly into RESTful APIs, enhancing real-time applications. Risks of Expired Domain Names: Highlighting the dangers of expired domains and how they can be exploited by hackers.  The 'Dead Internet' Theory: Debating the rise of bots on the web and their potential to surpass human activity online. 

Databases are ubiquitous, and you don’t need to be a data practitioner to know that all data everywhere is stored in a database—or is it? While the majority of data around the world lives in a database, the data that helps run the heart of our operating systems—the core functions of our computers— is not stored in the same place as everywhere else. This is due to database storage sitting ‘above’ the operating system, requiring the OS to run before the databases can be used. But what if the OS was built ‘on top’ of a database? What difference could this fundamental change make to how we use computers? Mike Stonebraker is a distinguished computer scientist known for his foundational work in database systems, he is also currently CTO & Co-Founder At DBOS. His extensive career includes significant contributions through academic prototypes and commercial startups, leading to the creation of several pivotal relational database companies such as Ingres Corporation, Illustra, Paradigm4, StreamBase Systems, Tamr, Vertica, and VoltDB. Stonebraker's role as chief technical officer at Informix and his influential research earned him the prestigious 2014 Turing Award. Stonebraker's professional journey spans two major phases: initially at the University of California, Berkeley, focusing on relational database management systems like Ingres and Postgres, and later, from 2001 at the Massachusetts Institute of Technology (MIT), where he pioneered advanced data management techniques including C-Store, H-Store, SciDB, and DBOS. He remains a professor emeritus at UC Berkeley and continues to influence as an adjunct professor at MIT’s Computer Science and Artificial Intelligence Laboratory. Stonebraker is also recognized for his editorial work on the book "Readings in Database Systems." In the episode, Richie and Mike explore the the success of PostgreSQL, the evolution of SQL databases, the shift towards cloud computing and what that means in practice when migrating to the cloud, the impact of disaggregated storage, software and serverless trends, the role of databases in facilitating new data and AI trends, DBOS and it’s advantages for security, and much more.  Links Mentioned in the Show: DBOSPaper: What Goes Around Comes Around[Course] Understanding Cloud ComputingRelated Episode: Scaling Enterprise Analytics with Libby Duane Adams, Chief Advocacy Officer and Co-Founder of AlteryxRewatch sessions from RADAR: The Analytics Edition New to DataCamp? Learn on the go using the DataCamp mobile appEmpower your business with world-class data and AI skills with DataCamp for business

Want to use Dart on the server to share code and complement your Flutter app? Learn about Serverpod, Flutter's full-stack Dart solution that uses code generation to create matching client-server code and a feature-rich Postgres ORM based on your schema.

Click the blue “Learn more” button above to tap into special offers designed to help you implement what you are learning at Google Cloud Next 25.

Reduce IT overhead and drive innovation with Database Service for Google Distributed Cloud Hosted. Focus on strategic application development by automating time-consuming tasks. Leverage PostgreSQL, Oracle, and the cutting-edge performance and AI capabilities of AlloyDB Omni for a competitive edge.

Click the blue “Learn more” button above to tap into special offers designed to help you implement what you are learning at Google Cloud Next 25.