The MeganePro data: a resource to improve hand prostheses control, its embodiment, and to understand the effect of hand amputation on cognition

The MeganePro data: a resource to improve hand prostheses control, its embodiment, and to understand the effect of hand amputation on cognition

Have you ever tried to open a bottle of water using just one hand? If you think of this straightforward example for a second, you can probably have a feeling of the consequences that the loss of a hand has on the life of a person. 

A hand prosthesis that can be controlled at will is a life changer.

Hand prostheses with remarkable mechanical capabilities are now available. They can have one motor controlling each finger, allowing the prosthesis to perform numerous complex grasp types. However, the control strategy usually represents the bottleneck of these devices.  

Remarkable progress has been made in scientific research over the years. The results are often impressive in controlled environments, but the robustness required for daily use is, unfortunately, still unmet. Some of us experienced such limitations in the NinaPro project, during which we tried to recognize more than 50 hand movements, including basic fingers and wrist motions as well as grasp types by applying machine learning to sEMG. 

However, despite the impressive advances in the field of hand prosthetics, many amputees still tend to neglect the use of a hand prosthesis to carry out their everyday life activities, resulting in severe impairments. Prostheses may be experienced as a mere extension of the physical body that does not resemble an intact limb and that is located in extracorporeal space. Also felt in extracorporeal space are phantom limb sensations. Experienced by almost all amputees, they represent the missing limb as it was still present. Some peculiarities in an individual’s phantom limb experience may offer unique insights into how prostheses may be optimally incorporated. Take, for instance, the way a phantom limb interacts with the solid characteristics of an artificial limb occupying the same space. Will it fade away or “bend back” in order to avoid violation of the law of impenetrability of physical matter? Will it continue its ghostly existence and be felt in superposition with the prosthesis? Any profound experience of embodiment of an artificial limb likely depends on an individual’s interaction between mind and matter and may significantly influence the willingness to use a prosthesis. 

The ultimate goal in the domain is to develop a prosthesis that can be both controlled and felt as closely as possible to a biological hand.

The pursuit of this goal drove the conception of the Myo-Electricity, Gaze, and Artificial intelligence for Neurocognitive Examination & Prosthetics (MeganePro) project. MeganePro aimed at i) increasing the robustness of grasp-type recognition by including additional sources of information (linked to eye-hand coordination) and at ii) investigating neurocognitive aspects of amputation particularly as far as phantom limb sensations are concerned.

Cognolato et al. focus on the former aim and present the MeganePro Dataset 1 (MDS1) containing sEMG, accelerometry, gaze, and first-person video from transradial amputees and able-bodied subjects while grasping several objects with 10 grasp types. Saetta et al. focus on the second aim of the project and describes the MeganePro Dataset 2 (MDS2), MeganePro Dataset 4 (MDS4), and MeganePro Dataset Info (MDSInfo) containing gaze, behavioral, and clinical data on the phantom limb after hand amputation recorded in the same participants. 

We recorded data from 15 transradial amputees, with different characteristics, such as age, gender, years since the amputation, cause of amputation, residual limb length, and prosthesis usage. The control group consisted of 30 able-bodied subjects in Cognolato et al., of which 29 served in Saetta et al. as well. All controls matched the amputees with respect to gender and age as closely as possible.

In MDS1, we devoted particular efforts on designing an acquisition protocol that can deliver a natural grasping behavior. We chose common household objects and paired them with grasp types among the most useful in everyday life. Most of the objects were paired with multiple grasp types and vice versa, as typical in real life manipulative tasks. Numerous objects were simultaneously placed in front of the subject during each grasp session and the guidance was given by vocal messages, with the possibility to select the most comfortable language for the subject among Italian, English, French, and German. We included grasping from a seated and standing position, which is common in everyday life, and added a session in which the subjects were asked to perform an action, such as to drink from a can or to open a door. 

Eye-hand and eye-phantom coordination are crucial components that might predict the embodiment of the prosthesis. In MDS2, we looked at how the imagery of phantom limb movement is coupled with eye movements during an imaginary pointing task where participants were asked to aim at different squares with the tip of a pen. As a control condition, imagined and executed pointing movements were performed by the amputees with their intact hand and by the controls with their dominant hand. 

Prostheses are physical matter, while a phantom limb is a mere “mind product”. However, mind and matter can be experienced as co-located in extracorporeal space. How can we exploit this individually variable phantom behavior to predict the embodiment of a prosthesis? The MDS4 specifically addresses this question. Amputees are asked to reposition their hand/phantom from a starting to an end-point in the presence of hindering physical matter, an “obstacle”, in between these points. 

MDSInfo offers an extensive assessment of both qualitative and quantitative aspects of phantom limb sensations performed by a trained neuropsychologist in a clinical interview lasting 90 minutes. 

The MeganePro data required the interaction of experts with various backgrounds as diverse as biomedical engineering, hand surgery, neuropsychology, and computer science, all related to prosthesis control and phantom limb sensations. Given the multifaceted nature of the datasets we are offering, we believe that its sharing will be of great interest to researchers in all these communities and that it will allow researchers to reach important advances in these fields when using the data.

The MeganePro Consortium