Even as claims of artificial intelligence taking over the jobs meant for physicians are frequently made, many organizations seek to understand how machine learning will benefit them prior to investing in the same. At Partners Connected Health, the Algorithm Science team has been working hard to identify potential pitfalls and develop the best measures to implement and evaluate machine learning solutions.
At Partners Connected Health/Harvard Medical School, Sujay Kakarmath, MD, who is a post-doctoral research fellow, advises the IT staff and CIOs of healthcare organizations on working with algorithms. For any given task, the technical performance of the algorithm is merely one of its intended uses. Evaluating the true cost of implementing an algorithm should account for the human resources and technical infrastructure, cost of acting on false positives or not acting on false negatives, and eventual decay in algorithm performance in diseases where medical science progresses rapidly.
At Partners Connected, Kakarmath and his colleagues focus on the problem that an algorithm seeks to solve. An interdisciplinary team comprising data scientists, human-centric design experts, software engineers, and physicians consults with the intended end-users of the algorithm to figure out their requirements from the solution. This simultaneously helps in understanding how an algorithmic solution can best fit into their workflow. After this, tests are conducted to evaluate the strength of the algorithm’s performance when data conditions are varied. This test is expected to reveal any overlooked weak spots in the performance of the algorithm that might fail it during challenging situations if it is implemented in electronic health record systems.
The resulting information is analyzed by looking into the efficiency and cost outcomes of the algorithm. Eventually, when organizations invest in new algorithms, they know exactly which metrics they seek an improvement upon, and can choose solutions that provide the required benefits.