NIST Drops 232 Pages of AI Guidelines Partly due to duties from the 2023 AI Executive Order, the National Institute of Standards and Technology (@NIST) released four new documents with guidelines for reducing AI risks. First, NIST applied the principles in its original AI Risk Management Framework (#RMF) to focus on generative AI (GAI). The GAI RMF outlines twelve different risk areas exacerbated by GAI: data privacy, environmental impacts, human-AI interactions, misinformation, cybersecurity, intellectual property, chemical, biological, radiological, or nuclear (#CBRN) weapons information, as well as GAI systems confidently stating erroneous content (“hallucinations”), recommending harmful actions, generating obscene content, generating biased content, and utilizing bad data or other third-party components. The document then recommends straightforward actions to manage these risks for organizations working with GAI models. Here are some examples: 📣 “Disclose use of GAI to end users.” ⚠️ “Establish policies to identify and disclose GAI system incidents to downstream AI actors.” 🕵️ “Bolster oversight of GAI systems with independent audits or assessments.” 🔄 “Re-evaluate risks when adapting GAI models to new domains.” 🔍 “Conduct adversarial role-playing exercises, AI red-teaming, or chaos testing to identify anomalous or unforeseen failure modes.” And so on. What is remarkable is the sheer number of useful practices for reducing various risks. In total, NIST recommends over 450 different actions. Next, NIST adapted its Secure Software Development Framework (#SSDF) in a second publication targeting GAI and large general-purpose AI models. This view of AI software can be useful, even though there are many important differences between AI and traditional software in the context of risk management. Here are some sample recommendations from the AI-specific SSDF: 📂 “Store reward models separately from AI models and data.” 📋 “Conduct periodic audits of AI models.” 🔙 “Be prepared to roll back to a previous AI model.” Lastly, NIST’s other two documents offer (1) plans for crafting global AI standards and (2) technical approaches for reducing risks posed by synthetic content. If it wasn't clear already, the US Government is piling loads of critical AI work on NIST. But NIST cannot enforce any of its AI suggestions, because NIST is not a regulatory agency. With Big Tech companies already reneging on their voluntary commitments to the UK @AISafetyInst, it's looking increasingly important to create an AI regulatory authority to ensure that companies actually adopt these best practices. #AI #AIPolicy #NIST