Lay Summary for Accuracy Testing of Patient-Reported Outcome Measure Results on Pathpoint®

Open Medical Ltd. received funding from a programme called NIHR i4i FAST (Invention for Innovation) to create and test a method for checking digital health score calculators. These health scores are often called Patient Reported Outcome Measures (PROMs). We discussed the results of our research with patient representatives and members of the public on the 21st of March 2024. Their comments have helped create this summary of our work. 

 

As a result of the funding provided by the NIHR, we were able to achieve the following outcomes:

 

1. Making sure our results are accurate: We have come up with a way to check the results of our PROMs. PROMs is a tool we use to ask patients about their health and wellbeing. We need to make sure the results we get are right and can be trusted. We have set up a system that tests the results we get from our computer programme. This means we can trust the results and use them to help doctors make decisions about patient care and keep an eye on patients remotely (from a distance).
    

2. Expanding our programme: With the help of FAST funding, we've improved our computer programme. It can now calculate PROM scores from additional questionnaires. These questionnaires ask patients about different health conditions and experiences. This makes our programme more useful for different areas of healthcare, from bone doctors to mental health and cancer care. We're also better prepared to add more questionnaires to our programme. This is a big step towards helping doctors make decisions and monitor patients remotely using PROMs.
    

3. Explaining our work: We know that our work affects patients and the public, so we've written a lay summary of what we've been doing. This summary is easy to understand and includes feedback from our Patient and Public Involvement and Engagement (PPIE) group. We hope this will help people understand our work better and trust PROMs more. We believe in putting patients first and involving them in decision-making. 

What is a PROM? 

PROMs are usually forms that patients fill out to tell their doctor how a treatment or procedure has affected their health. They often ask simple questions, like rating pain before and after surgery on a scale of one to ten, but some are much more detailed. A PROM usually asks multiple questions about someone's health, which are then used to calculate a score.

 

How are PROMs used?

These forms are sometimes used to help decide how to take care of someone. PROMs can also be used to monitor someone remotely, for example, if they are recovering at home after a surgery.  Now, healthcare providers are starting to get paid based on the answers people give on these forms. This is part of a new way of doing healthcare called 'value-based healthcare', which is still very new. 

 

Digital PROMs

Open Medical creates software that allows patients to complete a PROM form digitally, on their phone, computer, or tablet device. The score of a PROM is then calculated by our calculator and shown to a doctor or nurse. Previously, doctors and nurses would calculate the score themselves. We created a process to test our calculator, so we can be confident that the result is accurate. 

 

Testing our calculators

We looked at six PROM calculators, each used for a different PROM. Some of these forms ask a lot of questions, and as a result, there are over a trillion different ways a patient could complete such a form. Instead of calculating every possible response, we used the following approach: 

 

First, we created a dataset of so-called ‘edge-cases’. We created fake patient records where someone had entered extreme values or where there were missing values. This way, we could test what would happen if someone answered all the questions with the highest score, or work out what would happen if a patient didn’t respond to a certain question. We compared how the calculator performed against our own dataset and checked it by hand. 

 

Then, we created more fake records of random responses—we created three datasets of 100, 1000, and 1 million combinations of responses. Again, we compared how the calculator performed against our own dataset, although these millions of responses were not checked by hand. 

 

Lastly, we tested how the calculators performed when given false data, like providing text instead of a number, or something that is not a valid response (like “Always” if the question has a maximum of “Nearly every day”). 

 

Involving patient representatives and members of the public

We presented the process and results to our PPIE committee on the 21st of March, 2024. Our PPIE partners provided some insightful comments on the use of PROMs in clinical practice, raising important concerns about making sure that patients are informed about how their PROM responses are used. Some participants were also interested in knowing if artificial intelligence is used to examine these PROMs; this is not currently the case, but that could change in the future. 

 

The feedback our PPIE group shared about using PROMs and calculators mostly focused on how it could change their relationship with their doctor. They felt that these tools could help them better understand their health and make decisions, for example, by seeing how they scored compared to other patients. They also wanted to talk about the results with their doctor.