The National Institute of Standards and Technology (NIST) published its Artificial Intelligence Risk Management Framework (NIST AI 100-1) in January 2023.
The NIST AI Framework consists of 19 categories and 72 subcategories within the following four core functions:
- Govern
- Map
- Measure
- Manage
In prior articles, we focused on considerations when assessing and implementing the Govern and Map functions within the NIST AI Risk Management Framework. In this article, we focus on the implementation of the Measure function of the NIST AI Risk Management Framework.
The Measure function includes five categories and 22 subcategory controls as listed in Table 1 below.
How can organizations use the NIST AI Risk Management Framework Controls to assess activities that involve AI systems for the Measure function?
Along with the NIST AI Risk Management Framework, NIST also provided the AI Risk Management Playbook which contains supporting actions and considerations for each subcategory control.
Below are example questions to focus on when assessing an organization’s current AI compliance posture relative to the Measure function within the NIST AI Risk Management Framework:
- How will the appropriate performance metrics, such as the accuracy of the AI, be monitored after the AI is deployed? 1
- What corrective actions has the entity taken to enhance the quality, accuracy, reliability, and representativeness of the data? 2
- What are the roles, responsibilities, and delegation of authorities of personnel involved in the design, development, deployment, assessment, and monitoring of the AI system? 3
- How has the entity identified and mitigated potential impacts of bias in the data, including inequitable or discriminatory outcomes? 4
- As time passes and conditions change, is the training data still representative of the operational environment? 5
What should companies consider implementing to support alignment with the NIST AI Risk Management Framework Measure function?
After assessing and documenting activities that involve AI systems against the Measure function, below are examples of AI compliance management activities to assist organizations in implementation for remediation of gaps or to demonstrate AI compliance readiness and maturity:
- Establish approaches for detecting, tracking, and measuring known risks, errors, incidents, or negative impacts. 6
- Document reports of errors, incidents, and negative impacts and assess the sufficiency and efficacy of existing metrics for repairs, and upgrades. 7
- Utilize separate testing teams established in the Govern function to enable independent decisions and course correction for AI systems. Track processes and measure and document changes in performance. 8
- Measure and document performance criteria such as validity (false positive rate, false negative rate, etc.) and efficiency (training times, prediction latency, etc.). 9
- Monitor and document how metrics and performance indicators observed in production differ from the same metrics collected during pre-deployment testing. 10
The Measure function is focused on developing business processes to measure and then remediate or improve topics such as false positives, bias, and the intended uses of the AI systems. The Measure function aligns with Article 17 of the EU AI Act requirements where providers of AI systems must create a “quality management system” to properly assess or measure techniques, procedures, and systematic actions in the design, control, and verification of high-risk AI systems including testing and validation processes.
Notes:
1. NIST AI 100-1. NIST AI RMF Playbook. January 2023. Page 106.
2. NIST AI 100-1. NIST AI RMF Playbook. January 2023. Page 109.
3. NIST AI 100-1. NIST AI RMF Playbook. January 2023. Page 110.
4. NIST AI 100-1. NIST AI RMF Playbook. January 2023. Page 116.
5. NIST AI 100-1. NIST AI RMF Playbook. January 2023. Page 120.
6. NIST AI 100-1. NIST AI RMF Playbook. January 2023. Page 105.
7. NIST AI 100-1. NIST AI RMF Playbook. January 2023. Page 108.
8. NIST AI 100-1. NIST AI RMF Playbook. January 2023. Page 110.
9. NIST AI 100-1. NIST AI RMF Playbook. January 2023. Page 119.
10. NIST AI 100-1. NIST AI RMF Playbook. January 2023. Page 122.