Recapping the past year in AI, and what open questions are worth pursuing in the next year! Covering local models, transparency, tool usage, prompt injection. Please will SOMEBODY solve these??

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

About Simon Simon Willison is the creator of Datasette, an open source tool for exploring and publishing data. He currently works full-time building open source tools for data journalism, built around Datasette and SQLite. Prior to becoming an independent open source developer, Simon was an engineering director at Eventbrite. Simon joined Eventbrite through their acquisition of Lanyrd, a Y Combinator funded company he co-founded in 2010. He is a co-creator of the Django Web Framework, and has been blogging about web development and programming since 2002 at simonwillison.net

Large Language Models (LLMs) such as ChatGPT have revolutionized AI applications, offering unprecedented potential for complex real-world scenarios. However, fully harnessing this potential comes with unique challenges such as model brittleness and the need for consistent, accurate outputs. These hurdles become more pronounced when developing production-grade applications that utilize LLMs as a software abstraction layer. In this talk, we will tackle these challenges head-on. We introduce Guardrails AI, an open-source platform designed to mitigate risks and enhance the safety and efficiency of LLMs. We will delve into specific techniques and advanced control mechanisms that enable developers to optimize model performance effectively.

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

About Shreya Shreya Rajpal is the creator and maintainer of Guardrails AI, an open source platform developed to ensure increased safety, reliability, and robustness of large language models in real-world applications. Her expertise spans a decade in the field of machine learning and AI. Most recently, she was the founding engineer at Predibase, where she led the ML infrastructure team. In earlier roles, she was part of the cross-functional ML team within Apple's Special Projects Group and developed computer vision models for autonomous driving perception systems at Drive.ai.

Discover llm, a revolutionary Rust library that enables developers to harness the potential of large language models (LLMs) locally. By seamlessly integrating with the Rust ecosystem, llm empowers developers to leverage LLMs on standard hardware, reducing the need for cloud-based APIs and services.

In this talk, I'll explore llm's key features, including its high-speed inference, support for popular LLM architectures, and its lightweight design. Through practical examples, I'll showcase how llm can be applied in content generation, code completion, and language understanding tasks.

Additionally, I'll discuss the challenges of deploying and maintaining LLMs locally, along with best practices and real-world experiences from early adopters.

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

About Mithun Hunsur Mithun is a seasoned polyglot programmer and engineer with a passion for exploring the depths of computer science. With experience spanning from the low-level to the highest reaches of software development, Mithun has worked on a diverse range of projects across various industries. During the day, he works on Ambient, an open-source runtime/engine for building high-performance multiplayer games and 3D applications. In his free time, he's a tinkerer at heart, diving into the world of game reverse engineering and modification, low-level and embedded programming, virtual and augmented reality, compiler and language hacking, human-computer interface research, and computer architecture and design. Beyond his work in the tech industry, Mithun also has a creative side, dabbling in photography, writing, and AI art. He brings a unique perspective to his work, combining his passion for technology with his artistic sensibilities to build projects that are both innovative and visually stunning.

How YOU can - and should - build great multimodal AI apps that go viral and scale to millions in a weekend. Featuring the Vercel AI SDK and the new v0.dev AI frontend tool.

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

About Hassan Creator of RoomGPT

When OpenAI released CLIP, the Roboflow team built an AI Pictionary game called paint.wtf. Players were given a prompt like “a giraffe in the arctic,” and players drew depictions. CLIP judged which image embedding most closely matched the text embedding. Over 120,000 players played in the first week, peaking at 7 submissions per second.

Fast forward, and multimodality apps are ready. Come learn the trials (strangers on the internet submitting drawings) and successes (infra scaled without outage) of building with foundation models.

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

About Joseph Nelson Joseph is Co-founder/CEO at Roboflow, which makes tools over 250,000 developers use to build better computer vision models, faster. Roboflow is backed by Y Combinator, Craft Ventures, Floodgate, the founders of OpenAI, among others. He previously Co-founded and sold an NLP company that sorted the US Congress's mail and worked at Facebook. Joseph learned to code writing programs for TI-84 calculators.

Large Language Models (LLM's) are starting to revolutionize how users can search for, interact with, and generate new content. Some recent stacks and toolkits around Retrieval Augmented Generation (RAG) have emerged where users are building applications such as chatbots using LLMs on their own private data. This opens the door to a vast array of applications. However while setting up a naive RAG stack is easy, productionizing it is hard. In this talk, we talk about core techniques for evaluating and improving your retrieval systems for better performing RAG.

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

About Jerry Liu Jerry Liu, the co-founder and CEO of LlamaIndex, brings a wealth of expertise to his role, with a career that spans the realms of ML engineering, AI research, and startups. Prior to his current position, he served as an ML engineer at Quora and engaged in AI research with Uber's ATG. A Princeton alumnus, Jerry's professional journey has been enriched by various publications, including his most recent works: Deep Structured Reactive Planning and MuSCLE: Multi Sweep Compression of LiDAR using Deep Entropy Models, reflecting his commitment to the field.

In the last few months, we've seen an explosion of the use of retrieval in the context of AI. Document question answering, autonomous agents, and more use embeddings-based retrieval systems in a variety of ways. This talk will cover what we've learned building for these applications, the challenges developers face, and the future of retrieval in the context of AI.

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

About Anton Troynikov Anton is the co-founder of Chroma. He does not believe AI will kill us all. Chroma build an open-source embeddings store, specifically built for AI-native applications.

In this talk, we will talk about the different model adaptation methods from Prompt Engineering to RAGs to fine-tuning methods depending on the dataset and problem. We will also go into detail on some operational best practices for fine-tuning and how to evaluate them for specific business use-cases. Furthermore, we will conclude with a comparative framework, cost-benefit analysis benefits and tradeoffs of fine-tuning versus knowledge bases for improving the performance of large language models for a specific task.

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

About Abi Aryan Hi, my name is Abi. I am a computer scientist working extensively in machine learning to make the software systems smarter. Over the past seven years, my focus has been building machine learning systems for various applications including recommender systems, automated data labelling pipelines for both audio and video, audio-speech synthesis, forecasting and time-series analysis etc. In the past, I also attended Insight as a Data Science Fellow and was a Visiting Research Scholar at UCLA under Dr. Judea Pearl where I worked in AutoML, MultiAgent Systems and Emotion Recognition. I am also currently authoring LLMOps: Managing Large Language Models in Production book for O'Reilley Publications and an MLOps: Deploying ML models in production course for data scientists to learn fundamentals of data engineering and how to deploy machine learning models in production.

The recent wave of AI has created a frenzy to take radically different approaches to what we can build. The underlying LLMs, however, work off an incomplete canvas and dream up pictures and prose as they please. That’s great for chat apps, but what about the millions of applications out there today? Many can benefit from natural interfaces, but our code fundamentally expects structure. TypeChat is our experimental library to bridge the unstructured output of language models to the structured world of our code.

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

About Daniel Rosenwasser Daniel is the Product Manager of the TypeScript programming language and more recently has been experimenting with TypeChat. He has a passion for programming languages, type systems, runtimes, and building great tooling for developers.

AI.JSX is like React for LLMs -- it lets you build powerful, conversational AI apps using the power of TypeScript and JSX. Building LLM-powered apps is not like building other kinds of software. Not only can the AI model allow you to converse in natural language, but we can also use the LLM itself to replace a lot of the conventional code needed for processing data, interfacing with external APIs, and more. AI.JSX embodies this philosophy and makes it easy to build sophisticated AI experiences without a lot of code. It's also fully integrated with the Fixie platform, making deployment and management a breeze.

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

About Matt Welsh Matt Welsh is the Chief Architect and Co-founder of Fixie.ai, a Seattle-based startup developing a new computational platform with AI at the core. He was previously head of engineering at OctoML, a software engineer at Apple and Xnor.ai, engineering director at Google, and a Professor of Computer Science at Harvard University. He holds a PhD from UC Berkeley.

Testing is the backbone of quality code, yet it often takes a back seat as developers prioritize feature creation. The industry's reliance on metrics such as code coverage is insufficient and can even be misleading. In this evolving landscape, where AI coding agents are becoming commonplace, there's a pressing need to revolutionize our approach to testing.

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

About Dedy Kredo Dedy, Co-Founder and Chief Product Officer of CodiumAI, steers the product and engineering teams with an ambitious mission: to empower developers in creating software swiftly and flawlessly, harnessing the synergy of artificial and human intelligence. Prior to CodiumAI, Dedy took on the role of VP of Customer Facing Data Science at Explorium. There, he established and led a dedicated team of data scientists, playing a significant part in the company's progression from its seed stage to a Series C round. Dedy also ventured into entrepreneurship with a bootstrapped online marketing startup, scaling it from a modest beginning to impressive revenue figures. Earlier, as a product line manager at VMware's management business unit, he collaborated with Fortune 500 companies, launching several successful products.

Since the days of open source, we've experienced fundamental shifts in how software is built. From the pull request to deploying code, join Mario Rodriguez as he shares the history of GitHub Copilot and invites us to envision a new developer experience completely redefined and powered by AI.

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

About Mario Mario Rodriguez is a VP of Product Management at GitHub, currently focusing on all things Productivity. He oversees Repos, Pull Requests, Issues, Projects, Mobile and our AI strategy, including GitHub Copilot. Mario’s core identity is one of a learner and time away from product engineering is spent with his wife and two daughters.

This keynote panel, featuring Bryan Bischof, Head of AI at Hex, and Chris White, CTO at Prefect, will explore what it takes to launch AI-based products in the context of running a successful, non-AI startup. Hear in-the-trenches insights and hot takes from these two product leaders as they discuss how to create an AI strategy, how to (actually) move into production, and how to measure success. Conversation will be moderated by Brittany Walker, Principal at CRV.

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

Brittany Walker Brittany Walker is a principal at CRV, an early-stage venture capital firm known for leading investments in companies such as Airtable, Vercel, Cribl, and Postman, among others. She specializes in data infrastructure and AI/ML infrastructure, and works with the founders of promising early-stage startups including Doppler, Meilisearch, Impart Security, and Lokalise.

Chris White Chris White is a cofounder of Prefect, a data workflow orchestration and observability platform, for which he currently serves as the Chief Technology Officer. Chris began his journey into software while getting his PhD in Mathematics at the University of Texas at Austin, where he researched optimization guarantees for non-convex data-driven problems. In addition to his work at Prefect, Chris is an active angel investor in AI and developer tooling. Outside of tech Chris is also passionate about the ocean, literature, and is an avid runner.

Bryan Bischof Dr. Bryan Bischof leads AI at Hex, and is an adjunct professor in the Rutgers Masters of Business and Analytics program where he teaches Data Science. Previously, he was the Head of Data Science at Weights and Biases, and built the DS, ML, and Data Engineering teams. Before that he built ML products at Stitch Fix, Blue Bottle Coffee, and IBM. He recently co-authored the O’Reilly book Building Production Recommendation Systems. His data visualization work appeared in the popular book The Day it Finally Happens by Mike Pearl. His Ph.D. is in pure mathematics.

We love text embeddings as a critical pillar of LLM applications, but there's so much to text embeddings beyond their value in vector search. This talk will be a grand tour through a series of experimental projects from my last two years of research for visualizing, manipulating, and interpreting embeddings. We'll start with the basics (t-SNE, UMAP, and PCA), talk about how language models can be used to manipulate and interpret embeddings, and end by using a new tool I've built that lets us directly observe which features popular embedding models like to encode into their embeddings.

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

About Linus Lee Linus is a Research Engineer at Notion prototyping new software interfaces for augmenting our collaborative work and creativity with AI. He has spent the last few years experimenting with AI-augmented tools for thinking, like a canvas for exploring the latent space of neural networks and writing tools where ideas connect themselves. Before Notion, Linus spent a year as an independent researcher, during which he was Betaworks's first Researcher in Residence.

Every month, thousands of new AI applications are launched on Supabase, powered by pgvector. We'll take a brief look into the role of pgvector in the Vector database space, some of the use cases it enables, and some of the future of embeddings in the database space.

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

About Paul Copplestone Paul is the CEO and cofounder of Supabase, a database platform for developers. He is a 3-time founder and a developer with 20 years experience. Supabase is one of the fastest-growing communities of builders. with thousands of AI applications launched every week, and powering apps with millions of users.

How might we use AI to build products focused not just on working faster, but on transforming how we work? Let's talk about ways we can design using AI to help us “zoom out” and work better.

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

About Amelia Wattenberger Amelia is a designer and builder of novel, interactive interfaces. She spent the last few years exploring the future of developer tools on the GitHub Next team and is now designing an AI teammate at Adept.ai.

ChatGPT was a turning point for consumer adoption of AI due to its easy-to-use interface. Just by changing some elements of design, interaction, and behavior, an existing model suddenly 'clicked' in terms of its utility for everyday people. What might be the next leap forward for making AI-driven applications even more accessible & intuitive? Join Sam & Jason as they showcase various demos of novel interaction & behavior paradigms for AI-driven applications.

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

About Samantha Whitmore Former Head of Engineering at Kensho, a startup which used early NLP techniques to organize information for financial clients, including Goldman Sachs, BAML, and JPMC. Kensho was acquired in 2018 by S&P Global for $550mm, at the time the largest Al acquisition in history. Subsequently was Head of Engineering at Maximus, a startup that partnered with IMAX to build video super-resolution software. Recently was one of the early core contributors to LangChain (pioneered the implementation of Memory).

About Jason Yuan Former member of Apple Design Team where he worked on the future of computing and artificial intelligence. Founder and co-inventor of MakeSpace (now known as Sprout), a multi-player-first video conferencing platform. Creator of mercuryos.com and helped pioneer ideas in generative interfaces. Worked on projects with culture makers like Blackpink, Chanel, Vogue, Jackson Wang, The MET Gala, Nike, Christina Aguilera, FKA Twigs and The Weeknd.

“There is a large class of problems that are easy to imagine and build demos for, but extremely hard to make products out of. For example, self-driving: It’s easy to demo a car self-driving around a block, but making it into a product takes a decade.” - Andrej Karpathy

This talk is about practical patterns for integrating large language models (LLMs) into systems and products. We’ll draw from academic research, industry resources, and practitioner know-how, and try to distill them into key ideas and practices. There are seven key patterns. I’ve also organized them along the spectrum of improving performance vs. reducing cost/risk, and closer to the data vs. closer to the user.

Evals: To measure performance RAG: To add recent, external knowledge Fine-tuning: To get better at specific tasks Caching: To reduce latency & cost Guardrails: To ensure output quality Defensive UX: To anticipate & manage errors gracefully Collect user feedback: To build our data flywheel

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

About Eugene Yan Eugene Yan designs, builds, and operates machine learning systems that serve customers at scale. He's currently a Senior Applied Scientist at Amazon. Previously, he led machine learning at Lazada (acquired by Alibaba) and a Healthtech Series A. He writes & speaks about ML systems, engineering, and career at eugeneyan.com and https://ApplyingML.com

Please return only json, do not add any other comments ONLY RETURN JSON OR I'LL TAKE A LIFE

If this was you, then you've probably been pretty happy to see OpenAI function_call get released, I'm here to show you how you can get the most out of such powerful feature. Instead of writing prompts that turn strings into strings, we can write Pydantic objects and get Pydantic objects out of OpenAI.

In this talk we explore some model driven development. Where we go step by step with some examples on how to represent your problem as simple code so we can model, generate diagrams, and write prompts as code to save time and model complex data correctly, allow us to use the same best practices rather than having to invent new ones for how we manage prompts.

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

About Jason Liu Previously stitch fix and Facebook. Currently consulting startups on production using llm systems.

How can companies best build useful and differentiated applications on top of language models? Many of the products and companies built do this by providing the relevant context to LLMs and asking it to reason appropriately. In this talk, Harrison will discuss the different types of context you should be aware of, the different levels of cognitive architectures that are emerging, and how LangChain and LangSmith are built to help with this journey.

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

Harrison Chase is the CEO and co-founder of LangChain, a company formed around the open source Python/Typescript packages that aim to make it easy to develop Language Model applications. Prior to starting LangChain, he led the ML team at Robust Intelligence (an MLOps company focused on testing and validation of machine learning models), led the entity linking team at Kensho (a fintech startup), and studied stats and CS at Harvard.

We're heading towards a multimodal world. OpenAI is going beyond text models into vision, voice, and image generation, and we've been busy thinking about what kinds of things developers will be able to create with them. Presenting demos and insights into the near future for AI Engineers! In this talk from OpenAI's

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

About Simón Simón empowers builders to leverage OpenAI technologies in novel and impactful ways, most recently with the OpenAI Cookbook.

About Logan Logan currently leads developer relations at OpenAI, supporting developers building with DALL-E, the API, and ChatGPT. Outside of OpenAI, Logan is the Lead Developer Community Advocate for the Julia Programming Language, and a Teaching Fellow for Harvard University's Extension School course CSCI E-33A. Logan was previously a Applied Machine Learning Engineer and Software Engineer at Apple as well as the Community Manager for the Julia Programming Language. Additionally, Logan is on the Board of Directors at NumFOCUS and formerly on the board at DEFNA.

How will ubiquitous AI agents impact our daily lives, and what do they mean for the future of computing?

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

About Flo Flo Crivello is the Founder & CEO of Lindy. Engineer for 8 years, product leader at Uber for 5. Raised $50mm to build Teamflow, a virtual office for remote teams, from Menlo, Battery, Coatue, Tiger, Elad Gil, Lachy Groom and more, which we pivoted to Lindy.ai, an AI person assistant.

What is the future of Software Engineering in the age of AI? Amjad Masad, Founder & CEO of Replit, and Michele Catasta, head of AI at Replit, provide their take on this in this opening keynote presentation from the AI Engineer Summit 2023.

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

00:00 Introduction - Amjad Masad 00:42 Historical perspective 02:22 How AI can change software 04:29 ⚠️📢 Announcing AI for all! 06:33 A tale of Code LLM & GPU-Poor - Michele Catasta 07:29 How Replit's code completion works 08:39 ⚠️📢 Announcing Replit's new model! 13:45 ⚠️📢 Announcing the new model is open source! 14:06 Model training 15:26 Model evaluation 17:31 Model data & training 18:45 Model evaluation 19:51 Model inference 22:10 Why open source? 23:10: Glaive AI Collaboration 23:50 Morph Labs Collaboration 24:42: Perplexity Collaboration

Born too late to explore the earth. Born too early to explore the stars. Just in time to bring AI to everyone.

Swyx (https://twitter.com/swyx) is the co-founder of the AI Engineer Summit, editor & podcaster at https://latent.space, and the founder of https://smol.ai, a model distillation company.

Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair

00:00 Intro 00:15 Multiple-dimension presentation 01:00 Historical perspectives 02:01 You're not too late: modern cynicism is incorrect 02:35 Carlota Perez & tech revolutions 03:27 When did the AI revolution start? 04:58 What is an AI Engineer? 06:07 3 major DEFINITIONS of AI Engineer 06:47 3 major TYPES of AI Engineer 07:42 Why Summit