Better metrics? Measuring progress in prosthetics users
Researchers at Brown University have devised a series of "metrics" designed to monitor the progress of prosthetics patients. The metrics measure the performance of patients with prosthetic arms when carrying out 18 household tasks such as putting on a shirt, pouring soda and tying shoelaces.
The particular tasks selected have been identified as fundamental to the independence of the prosthetic user. "Patients can't just take a prosthesis out of the box and start using it skillfully," said lead author Linda Resnik, who says training is required in order for prosthetic arms to be used to their full potential. "We need measures to let us know if our patients are improving the way that we expect them to. When they get a new device, what are the benefits? Are they able do more with it?"
All good questions. So, the researchers have come up with AM-ULA (or Activities Measure for Upper Limb Amputees), and conducted trials with 49 military veterans at various Department of Veterans Affairs facilities and at Fort Sam Houston.
Though metrics for prosthetic use do exist, these tend to rely on self-reporting by the user, which, Resnik asserts, may overlook compensation used by other parts of the body. "That's important, because we know that upper limb amputees often develop problems in their neck and back," she says.
Crucial to the AM-ULA system is that it relies on monitoring by an observer. It has been adapted over a period of testing to increase reliability, so that independent raters would draw the same conclusions. The 18 tasks were reduced from 24 because the ratings drawn from the extra six were deemed too unreliable. The research has also sought to identify by how much a score need change to be identifiable as actual progress, rather than a statistical blip (3.7 points, apparently).
The metrics were devised to assist Resnik's own research but it's thought they have the potential to become a standard across the field. The development may not be as eye-catching as the latest neural-interface bionic arm, but it sounds as if this research has the potential to make a real and lasting contribution to the field of prosthetics. It'll be interesting to see if the technique is adopted more widely.
We've asked Brown for more details about the 18 tests, and specifically the assessment methods used. The word metric implies measurement, and clearly measurable results are preferable, being open to the least interpretation. We'll update this story with the relevant details when we hear them.
Update: Brown University has been in touch to let us see the report accompanying the research, titled Development and Evaluation of the Activities Measure for Upper Limb Amputees (AM-ULA), which includes the original list of 26 tasks. In addition to those mentioned above, these include writing the word "Letter" with a pen, dialing a number with a cellphone and folding a towel.
Each task is assessed on five criteria: the extent of its completion, the time it took, movement quality, skillfulness of prosthetic use and independence, the last three of which sound rather subjective. The paper also reveals that each of these criteria are graded from 0 to 4, so rather than actually timing the task, observers instead grade the time taken, with one point award for "very slow to slow," and three for "medium-fast to normal," for example. One wonders why observers wouldn't just record the actual time taken to establish benchmarks for each task, against which scores 0 to 4 can later be assigned.
Similarly, 4 points are available for "independence" if the user avoids the use of an assistive device. However, the observer's guidance for scores 0 to 3 states "may or may not use assistive device" in each case.
Strangely, it's perhaps the "skillfulness of prosthesis use" criterion that has the least ambiguity attached, as there is precise guidance for scoring. This includes monitoring loss of grip of an object. Once would record a score of 2, for instance, while multiple losses would score only 1.
Clearly the point of the 0 to 4 points system is to be able to assign scores which account for a number of variables. Whether there's more work to be done to minimize ambiguity and subjectivity is perhaps open to question. After all, if the results are to be subjective, should an observer's view count for more than the user's?
Source: Brown University