Business View Magazine l October 2022

94 BUSINESS VIEW MAGAZINE VOLUME 9, ISSUE 10 identify and evaluate different risks,” Baird explained. “The only difference is that ML is going to fail in slightly different ways than how software typically fails.” Prior to publishing their report, the task force conducted a literature review of AI/ML failures in multiple other industries in an attempt to collect valuable lessons learned. “One of the things we noticed in the literature about ML systems is that many times failures occurred because, although the development team had data, they didn’t have knowledge,” Baird said. “Developers had logical assumptions regarding the use of their product, but the reality was different, leading to failure.” Sparnon provided an example for healthcare technology professionals during the recent AAMI eXchange conference. “Imagine a system looking at lung x-rays is tasked to identify the sickest patients.” As the algorithm is fed more information, it learns that x-rays from patients getting imaged up on the upper floors are often the sickest, because they are too sick to go down to radiology. From that logic alone, completing the task then boils down to efficiently “picking on where the image was taken,” said Sparnon. “It’s not actually assessing the patient’s lung at all. All the data scientist sees is ‘hey, look, this system is great at finding out who’s too sick.’” There’s no one with the right contextual knowledge to realize the program has failed to grasp the true purpose of the task. “To be successful, we really need to understand the context of use, and leverage the wisdom around us. We felt it was important to stress this point when discussing risk management,” Baird concluded. As a result, the CR details safety-related characteristics and considerations for data management, bias, security and privacy, adaptive systems, and even troubles that may arise from too much trust in AI. The CR also includes annexes covering the risk management process, risk management examples, considerations for autonomous systems, and personnel qualifications. AAMI and BSI next plan to use this CR as the basis for an AAMI technical report and a British Standard. Longer term, AAMI and BSI expect to propose these resources to the International Organization for Standardization as guidance, informative, or annex documents to ISO 14971 and its sister document, ISO 24971. “I’ll be the first to admit that AI has been promising us great things for decades and has consistently failed to meet people’s expectations,” said Baird, “but I think this time we will succeed.” Parties interested in joining the AAMI AI Committee can email Standards@aami.org for more information.