Language models are incredible engineering breakthroughs but require auditing and risk management before productization. These systems raise concerns about toxicity, transparency and reproducibility, intellectual property licensing and ownership, disinformation and misinformation, supply chains, and more. How can your organization leverage these new tools without taking on undue or unknown risks? While language models and associated risk management are in their infancy, a small number of best practices in governance and risk are starting to emerge. If you have a language model use case in mind, want to understand your risks, and do something about them, this presentation is for you! We'll be covering the following: 

• Studying past incidents in the AI Incident Database and using this information to guide debugging.
• Adhering to authoritative standards, like the NIST AI Risk Management Framework. 
• Finding and fixing common data quality issues.
• Applying general public tools and benchmarks as appropriate (e.g., BBQ, Winogender, TruthfulQA).
• Binarizing specific tasks and debugging them using traditional model assessment and bias testing.
• Engineering adversarial prompts with strategies like counterfactual reasoning, role-playing, and content exhaustion. 
• Conducting random attacks: random sequences of attacks, prompts, or other tests that may evoke unexpected responses. 
• Countering prompt injection attacks, auditing for backdoors and data poisoning, ensuring endpoints are protected with authentication and throttling, and analyzing third-party dependencies. 
• Engaging stakeholders to help find problems system designers and developers cannot see. 
• Everyone knows that generative AI is going to be huge. Don't let inadequate risk management ruin the party at your organization!

Talk by: Patrick Hall

Here’s more to explore: LLM Compact Guide: https://dbricks.co/43WuQyb Big Book of MLOps: https://dbricks.co/3r0Pqiz

Connect with us: Website: https://databricks.com Twitter: https://twitter.com/databricks LinkedIn: https://www.linkedin.com/company/databricks Instagram: https://www.instagram.com/databricksinc Facebook: https://www.facebook.com/databricksinc

The past decade has witnessed the broad adoption of artificial intelligence and machine learning (AI/ML) technologies. However, a lack of oversight in their widespread implementation has resulted in some incidents and harmful outcomes that could have been avoided with proper risk management. Before we can realize AI/ML's true benefit, practitioners must understand how to mitigate its risks. This book describes approaches to responsible AI—a holistic framework for improving AI/ML technology, business processes, and cultural competencies that builds on best practices in risk management, cybersecurity, data privacy, and applied social science. Authors Patrick Hall, James Curtis, and Parul Pandey created this guide for data scientists who want to improve real-world AI/ML system outcomes for organizations, consumers, and the public. Learn technical approaches for responsible AI across explainability, model validation and debugging, bias management, data privacy, and ML security Learn how to create a successful and impactful AI risk management practice Get a basic guide to existing standards, laws, and assessments for adopting AI technologies, including the new NIST AI Risk Management Framework Engage with interactive resources on GitHub and Colab

Machine learning is a hot topic in business. Even data-driven organizations that have spent years developing successful data analysis platforms, with many accurate statistical models in place, are now looking into this decades-old discipline. But how can companies turn hyped opportunities for machine learning into real business value? This report examines the growing momentum of machine learning in the analytics landscape, the challenges machine learning presents to businesses, and examples of how organizations are actively seeking to incorporate modern machine learning techniques into their production data infrastructures. Authors Patrick Hall, Wen Phan, and Katie Whitson look at two companies in depth—one in healthcare and one in finance—that are seeing the real impact of machine learning. Discover how machine learning can help your organization: Analyze and generate insights from large amounts of varied, messy, and unstructured data unfit for traditional statistical analysis Increase the predictive accuracy beyond what was previously possible Augment aging analytical processes and other decision-making tools