Evaluation of Physiotherapy Exercise by Motion Capturing Based on Artificial Intelligence: A Review

Section: Review Paper
Issue
Vol. 28 No. 2 (2023): Volume 28 Issue 2
Published
Sep 1, 2023
Pages
237-251

Abstract

Physical therapy is an important form of rehabilitation for patients suffering from a variety of disorders. Since professional physiotherapists are not always available, there is a need to introduce an intelligent system that assets the patients to perform the exercise by themselves. Any evaluation system consists of hardware interfacing, computers, processing, and evaluation tools. These tools made it easier to build methods for automating the evaluation of patient performance and advancement in functional rehabilitation. In this research, about one hundred research papers are classified according to the above-mentioned system parts. The review of current tools for capturing rehabilitative motions shows that the Kinect camera has been used in about 35% of the studies. This review concentrates on using machine learning techniques to evaluate motion in rehabilitation. The most relevant research for physiotherapy evaluation using deep learning have shown that the Convolutional Neural Network (CNN) is widely used by 44% of the researcher. A useful overview the collection of the reference datasets illuminates that the KIMORE dataset is popular and used by 38% as compared with other types of datasets. The advanced literature in the present peer-reviewed paper (20162022), includes primary studies and organized reviews.

References

  1. -J. Su, C.-Y. Chiang, and J.-Y. Huang, "Kinect-enabled home-based rehabilitation system using Dynamic Time Warping and fuzzy logic," Applied Soft Computing, vol. 22, pp. 652-666, 2014.
  2. -S. Hosseini, H. Peyrovi, and M. Gohari, "The effect of early passive range of motion exercise on motor function of people with stroke: a randomized controlled trial," Journal of caring sciences, vol. 8, no. 1, p. 39, 2019.
  3. A. Felipe et al., "Evaluation instruments for physical therapy using virtual reality in stroke patients: a systematic review," Physiotherapy, vol. 106, pp. 194-210, 2020.
  4. A. Khedher, D. A. Alkababji, and O. Hadi, "Improving the Reliability of Object Recognition Based On Template Matching," Al-Rafidain Engineering Journal (AREJ), vol. 23, no. 5, pp. 81-88, 2015.
  5. A. Sultan and M. Ghanim, "Comprehensive Study and Evaluation of Commonly used Dimensionality Reduction Techniques in Biometrics Field," Al-Rafidain Engineering Journal (AREJ), vol. 25, no. 2, pp. 152-163, 2020.
  6. Alzubaidi et al., "Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions," Journal of big Data, vol. 8, pp. 1-74, 2021.
  7. Anaz, M. Skubic, J. Bridgeman, and D. M. Brogan, "Classification of therapeutic hand poses using convolutional neural networks," in 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2018: IEEE, pp. 3874-3877.
  8. Anton, I. Berges, J. Bermdez, A. Goi, and A. Illarramendi, "A telerehabilitation system for the selection, evaluation and remote management of therapies," Sensors, vol. 18, no. 5, p. 1459, 2018.
  9. Bejakovi, "Eurofound: Living, Working and Covid-19, Covid-19 Series," Revija za socijalnu politiku, vol. 28, no. 1, pp. 115-117, 2021.
  10. Ben, P. Adeline, H. Sion, and M. Majid, "Skeleton-Free Body Pose Estimation from Depth Images for Movement Analysis," in Proceedings of the 2015 IEEE International Conference on Computer Vision Workshop (ICCVW), Santiago, Chile, 2015, pp. 7-13.
  11. Boukhennoufa, X. Zhai, K. D. McDonald-Maier, V. Utti, and J. Jackson, "Improving the activity recognition using GMAF and transfer learning in post-stroke rehabilitation assessment," in 2021 ieee 19th world symposium on applied machine intelligence and informatics (SAMI), 2021: IEEE, pp. 000391-000398.
  12. Butepage, M. J. Black, D. Kragic, and H. Kjellstrom, "Deep representation learning for human motion prediction and classification," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 6158-6166.
  13. C-H. Huang, C.-F. Lin, C.-A. Chen, C.-H. Hwang, and D.-C. Huang, "Real-time rehabilitation exercise performance evaluation system using deep learning and thermal image," in 2020 IEEE International Instrumentation and Measurement Technology Conference (I2MTC), 2020: IEEE, pp. 1-6.
  14. C. Alarcn-Aldana, M. Callejas-Cuervo, and A. P. L. Bo, "Upper limb physical rehabilitation using serious videogames and motion capture systems: A systematic review," Sensors, vol. 20, no. 21, p. 5989, 2020.
  15. C. Quatman-Yates et al., "Physical therapy evaluation and treatment after concussion/mild traumatic brain injury: Clinical practice guidelines linked to the international classification of functioning, disability and health from the academy of orthopaedic physical therapy, American Academy of sports physical therapy, academy of neurologic physical therapy, and academy of pediatric physical therapy of the American Physical therapy association," Journal of Orthopaedic & Sports Physical Therapy, vol. 50, no. 4, pp. CPG1-CPG73, 2020.
  16. Campbell, E. H. Coulter, P. G. Mattison, L. Miller, A. McFadyen, and L. Paul, "Physiotherapy rehabilitation for people with progressive multiple sclerosis: a systematic review," Archives of physical medicine and rehabilitation, vol. 97, no. 1, pp. 141-151. e3, 2016.
  17. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, "Realtime multi-person 2d pose estimation using part affinity fields," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 7291-7299.
  18. Capecci et al., "A Hidden Semi-Markov Model based approach for rehabilitation exercise assessment," Journal of biomedical informatics, vol. 78, pp. 1-11, 2018.
  19. Capecci et al., "An instrumental approach for monitoring physical exercises in a visual markerless scenario: A proof of concept," Journal of biomechanics, vol. 69, pp. 70-80, 2018.
  20. Capecci et al., "Physical rehabilitation exercises assessment based on hidden semi-markov model by kinect v2," in 2016 IEEE-EMBS international conference on biomedical and health informatics (BHI), 2016: IEEE, pp. 256-259.
  21. Coskun, D. J. Tan, S. Conjeti, N. Navab, and F. Tombari, "Human motion analysis with deep metric learning," in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 667-683.
  22. D. Correia et al., "Medium-term outcomes of digital versus conventional home-based rehabilitation after total knee arthroplasty: prospective, parallel-group feasibility study," JMIR rehabilitation and assistive technologies, vol. 6, no. 1, p. e13111, 2019.
  23. D. Hssayeni, J. L. Adams, and B. Ghoraani, "Deep learning for medication assessment of individuals with Parkinsons disease using wearable sensors," in 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2018: IEEE, pp. 1-4.
  24. D. Tsakanikas et al., "Evaluating the performance of balance physiotherapy exercises using a sensory platform: The basis for a persuasive balance rehabilitation virtual coaching system," Frontiers in digital health, vol. 2, p. 545885, 2020.
  25. Dehbandi et al., "Using data from the Microsoft Kinect 2 to quantify upper limb behavior: a feasibility study," IEEE journal of biomedical and health informatics, vol. 21, no. 5, pp. 1386-1392, 2016.
  26. Elkholy, M. E. Hussein, W. Gomaa, D. Damen, and E. Saba, "Efficient and robust skeleton-based quality assessment and abnormality detection in human action performance," IEEE journal of biomedical and health informatics, vol. 24, no. 1, pp. 280-291, 2019.
  27. Escalona, E. Martinez-Martin, E. Cruz, M. Cazorla, and F. Gomez-Donoso, "EVA: EVAluating at-home rehabilitation exercises using augmented reality and low-cost sensors," Virtual Reality, vol. 24, pp. 567-581, 2020.
  28. Fabbri et al., "A systematic review of the psychometric properties of the JebsenTaylor Hand Function Test (JTHFT)," Hand Surgery and Rehabilitation, vol. 40, no. 5, pp. 560-567, 2021.
  29. Fan, "Cerebral Infarction Rehabilitation Evaluation with Posture Analyses," in IOP Conference Series: Materials Science and Engineering, 2019, vol. 612, no. 2: IOP Publishing, p. 022082.
  30. Ferraris et al., "A self-managed system for automated assessment of UPDRS upper limb tasks in Parkinsons disease," Sensors, vol. 18, no. 10, p. 3523, 2018.
  31. Galna, G. Barry, D. Jackson, D. Mhiripiri, P. Olivier, and L. Rochester, "Accuracy of the Microsoft Kinect sensor for measuring movement in people with Parkinson's disease," Gait & posture, vol. 39, no. 4, pp. 1062-1068, 2014.
  32. Ge, H. Liang, J. Yuan, and D. Thalmann, "3d convolutional neural networks for efficient and robust hand pose estimation from single depth images," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1991-2000.
  33. Gu, S. Pandit, E. Saraee, T. Nordahl, T. Ellis, and M. Betke, "Home-based physical therapy with an interactive computer vision system," in Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2019, pp. 0-0.
  34. H. Osgouei, D. Soulsbv, and F. Bello, "An objective evaluation method for rehabilitation exergames," in 2018 IEEE Games, Entertainment, Media Conference (GEM), 2018: IEEE, pp. 28-34.
  35. Hachaj and M. R. Ogiela, "Rule-based approach to recognizing human body poses and gestures in real time," Multimedia Systems, vol. 20, pp. 81-99, 2014.
  36. Halilaj, A. Rajagopal, M. Fiterau, J. L. Hicks, T. J. Hastie, and S. L. Delp, "Machine learning in human movement biomechanics: Best practices, common pitfalls, and new opportunities," Journal of biomechanics, vol. 81, pp. 1-11, 2018.
  37. Hart, H. Smith, and Y. Zhang, "Systematic review of automatic assessment systems for resistance-training movement performance: A data science perspective," Computers in Biology and Medicine, vol. 137, p. 104779, 2021.
  38. Houmanfar, M. Karg, and D. Kuli, "Movement analysis of rehabilitation exercises: Distance metrics for measuring patient progress," IEEE Systems Journal, vol. 10, no. 3, pp. 1014-1025, 2014.
  39. Huan and H. Zhou, "Continuous human pose estimation by machine learning and computer vision," 2022.
  40. Izadmehr, H. F. Satizbal, K. Aminian, and A. Perez-Uribe, "Depth Estimation for Egocentric Rehabilitation Monitoring Using Deep Learning Algorithms," Applied Sciences, vol. 12, no. 13, p. 6578, 2022.
  41. K. Deters and Y. Rybarczyk, "Hidden Markov Model approach for the assessment of tele-rehabilitation exercises," International Journal of Artificial Intelligence, vol. 16, no. 1, pp. 1-19, 2018.
  42. K. O'Brien et al., "Activity recognition for persons with stroke using mobile phone technology: toward improved performance in a home setting," Journal of medical Internet research, vol. 19, no. 5, p. e184, 2017.
  43. Klishkovskaia, A. Aksenov, A. Sinitca, A. Zamansky, O. A. Markelov, and D. Kaplun, "Development of Classification Algorithms for the Detection of Postures Using Non-Marker-Based Motion Capture Systems," Applied Sciences, vol. 10, no. 11, p. 4028, 2020.
  44. Kramer, N. Schmidt, R. Memmesheimer, and D. Paulus, "Evaluation of physical therapy through analysis of depth images," in 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), 2019: IEEE, pp. 1-6.
  45. Kurillo, J. J. Han, A. Nicorici, and R. Bajcsy, "Tele-MFAsT: Kinect-Based Tele-Medicine Tool for Remote Motion and Function Assessment," in MMVR, 2014, pp. 215-221.
  46. L. Chmielewski et al., "Low-versus high-intensity plyometric exercise during rehabilitation after anterior cruciate ligament reconstruction," The American journal of sports medicine, vol. 44, no. 3, pp. 609-617, 2016.
  47. Lee, Y.-S. Lee, and J. Kim, "Automated evaluation of upper-limb motor function impairment using Fugl-Meyer assessment," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 26, no. 1, pp. 125-134, 2017.
  48. Leightley, J. S. McPhee, and M. H. Yap, "Automated analysis and quantification of human mobility using a depth sensor," IEEE journal of biomedical and health informatics, vol. 21, no. 4, pp. 939-948, 2016.
  49. Liao, A. Vakanski, and M. Xian, "A deep learning framework for assessing physical rehabilitation exercises," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 28, no. 2, pp. 468-477, 2020.
  50. Liao, A. Vakanski, M. Xian, D. Paul, and R. Baker, "A review of computational approaches for evaluation of rehabilitation exercises," Computers in biology and medicine, vol. 119, p. 103687, 2020.
  51. Liu et al., "Feature boosting network for 3D pose estimation," IEEE transactions on pattern analysis and machine intelligence, vol. 42, no. 2, pp. 494-501, 2019.
  52. Liu, J. Zhu, J. Bu, and C. Chen, "A survey of human pose estimation: the body parts parsing based methods," Journal of Visual Communication and Image Representation, vol. 32, pp. 10-19, 2015.
  53. Lu, Z. Deng, J. Luo, W. Chen, S.-K. Yeung, and Y. He, "3D articulated skeleton extraction using a single consumer-grade depth camera," Computer Vision and Image Understanding, vol. 188, p. 102792, 2019.
  54. M. Burns, N. Leung, M. Hardisty, C. M. Whyne, P. Henry, and S. McLachlin, "Shoulder physiotherapy exercise recognition: machine learning the inertial signals from a smartwatch," Physiological measurement, vol. 39, no. 7, p. 075007, 2018.
  55. M. Hulteen, N. J. Lander, P. J. Morgan, L. M. Barnett, S. J. Robertson, and D. R. Lubans, "Validity and reliability of field-based measures for assessing movement skill competency in lifelong physical activities: a systematic review," Sports medicine, vol. 45, pp. 1443-1454, 2015.
  56. Maciejasz, J. Eschweiler, K. Gerlach-Hahn, A. Jansen-Troy, and S. Leonhardt, "A survey on robotic devices for upper limb rehabilitation," Journal of neuroengineering and rehabilitation, vol. 11, no. 1, pp. 1-29, 2014.
  57. Meng et al., "Exploration of human activity recognition using a single sensor for stroke survivors and able-bodied people," Sensors, vol. 21, no. 3, p. 799, 2021.
  58. Omelina, B. Jansen, B. Bonnechre, M. Oravec, P. Jarmila, and S. V. S. Jan, "Interaction detection with depth sensing and body tracking cameras in physical rehabilitation," Methods of information in medicine, vol. 55, no. 01, pp. 70-78, 2016.
  59. Panuccio et al., "Internal consistency and validity of the Italian version of the JebsenTaylor hand function test (JTHFT-IT) in people with tetraplegia," Spinal Cord, vol. 59, no. 3, pp. 266-273, 2021.
  60. Panwar et al., "Rehab-net: Deep learning framework for arm movement classification using wearable sensors for stroke rehabilitation," IEEE Transactions on Biomedical Engineering, vol. 66, no. 11, pp. 3026-3037, 2019.
  61. Paraskevopoulos, E. Spyrou, D. Sgouropoulos, T. Giannakopoulos, and P. Mylonas, "Real-time arm gesture recognition using 3D skeleton joint data," Algorithms, vol. 12, no. 5, p. 108, 2019.
  62. Piero-Fuentes, S. Canas-Moreno, A. Rios-Navarro, M. Domnguez-Morales, J. L. Sevillano, and A. Linares-Barranco, "A deep-learning based posture detection system for preventing telework-related musculoskeletal disorders," Sensors, vol. 21, no. 15, p. 5236, 2021.
  63. Pogrzeba, T. Neumann, M. Wacker, and B. Jung, "Analysis and quantification of repetitive motion in long-term rehabilitation," IEEE journal of biomedical and health informatics, vol. 23, no. 3, pp. 1075-1085, 2018.
  64. R. Shareef and Y. F. M. Al-Irhayim, "Comparison Between Features Extraction Techniques for Impairments Arabic Speech," Al-Rafidain Engineering Journal (AREJ), vol. 27, no. 2, pp. 190-197, 2022.
  65. S. Ravali et al., "A systematic review of artificial intelligence for pediatric physiotherapy practice: past, present, and future," Neuroscience Informatics, vol. 2, no. 4, p. 100045, 2022.
  66. Saraee et al., "Exercisecheck: remote monitoring and evaluation platform for home based physical therapy," in Proceedings of the 10th international conference on PErvasive technologies related to assistive environments, 2017, pp. 87-90.
  67. Sarafianos, B. Boteanu, B. Ionescu, and I. A. Kakadiaris, "3d human pose estimation: A review of the literature and analysis of covariates," Computer Vision and Image Understanding, vol. 152, pp. 1-20, 2016.
  68. Sardari, A. Paiement, S. Hannuna, and M. Mirmehdi, "Vi-netview-invariant quality of human movement assessment," Sensors, vol. 20, no. 18, p. 5258, 2020.
  69. Sarsfield et al., "Clinical assessment of depth sensor based pose estimation algorithms for technology supervised rehabilitation applications," International journal of medical informatics, vol. 121, pp. 30-38, 2019.
  70. Spasojevi, A. Rodi, and J. Santos-Victor, "Kinect-based approach for upper body movement assessment in stroke," in New Trends in Medical and Service Robotics: Advances in Theory and Practice, 2019: Springer, pp. 153-160.
  71. Suzuki, Y. Amemiya, and M. Sato, "Deep learning assessment of child gross-motor," in 2020 13th International Conference on Human System Interaction (HSI), 2020: IEEE, pp. 189-194.
  72. T. Um et al., "Parkinson's disease assessment from a wrist-worn wearable sensor in free-living conditions: Deep ensemble learning and visualization," arXiv preprint arXiv:1808.02870, 2018.
  73. Tang, "Hybridized hierarchical deep convolutional neural network for sports rehabilitation exercises," IEEE Access, vol. 8, pp. 118969-118977, 2020.
  74. Tao et al., "A comparative study of pose representation and dynamics modelling for online motion quality assessment," Computer vision and image understanding, vol. 148, pp. 136-152, 2016.
  75. Tschuggnall, V. Grote, M. Pirchl, B. Holzner, G. Rumpold, and M. J. Fischer, "Machine learning approaches to predict rehabilitation success based on clinical and patient-reported outcome measures," Informatics in Medicine Unlocked, vol. 24, p. 100598, 2021.
  76. V. Gauthier et al., "Video Game Rehabilitation for Outpatient Stroke (VIGoROUS): protocol for a multi-center comparative effectiveness trial of in-home gamified constraint-induced movement therapy for rehabilitation of chronic upper extremity hemiparesis," BMC neurology, vol. 17, no. 1, pp. 1-18, 2017.
  77. Vakanski, J. Ferguson, and S. Lee, "Mathematical modeling and evaluation of human motions in physical therapy using mixture density neural networks," Journal of physiotherapy & physical rehabilitation, vol. 1, no. 4, 2016.
  78. Vakanski, J. M. Ferguson, and S. Lee, "Metrics for performance evaluation of patient exercises during physical therapy", International journal of physical medicine & rehabilitation, Vol. 5, Issue:3, 2017.
  79. Vasileiadis, C.-S. Bouganis, and D. Tzovaras, "Multi-person 3D pose estimation from 3D cloud data using 3D convolutional neural networks," Computer Vision and Image Understanding, vol. 185, pp. 12-23, 2019.
  80. W. X. Cejnog, T. de Campos, V. M. C. Elui, and R. M. Cesar Jr, "A framework for automatic hand range of motion evaluation of rheumatoid arthritis patients," Informatics in Medicine Unlocked, vol. 23, p. 100544, 2021.
  81. Wei, C. McElroy, and S. Dey, "Towards on-demand virtual physical therapist: Machine learning-based patient action understanding, assessment and task recommendation," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 27, no. 9, pp. 1824-1835, 2019.
  82. Wei, C. Mcelroy, and S. Dey, "Using sensors and deep learning to enable on-demand balance evaluation for effective physical therapy," IEEE Access, vol. 8, pp. 99889-99899, 2020.
  83. Williams, A. Vakanski, S. Lee, and D. Paul, "Assessment of physical rehabilitation movements through dimensionality reduction and statistical modeling," Medical engineering & physics, vol. 74, pp. 13-22, 2019.
  84. Yuan et al., "Depth-based 3d hand pose estimation: From current achievements to future goals," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 2636-2645.
  85. Zhang, C. Su, and C. He, "Rehabilitation exercise recognition and evaluation based on smart sensors with deep learning framework," IEEE Access, vol. 8, pp. 77561-77571, 2020.
  86. Zhang, Z. C. Lipton, M. Li, and A. J. Smola, "Dive into deep learning," arXiv preprint arXiv:2106.11342, 2021.
  87. Zhao, R. Lun, D. D. Espy, and M. A. Reinthal, "Realtime motion assessment for rehabilitation exercises: Integration of kinematic modeling with fuzzy inference," Journal of Artificial Intelligence and Soft Computing Research, vol. 4, no. 4, pp. 267-285, 2014.
  88. Zhao, R. Lun, D. D. Espy, and M. A. Reinthal, "Rule based realtime motion assessment for rehabilitation exercises," in 2014 IEEE Symposium on Computational Intelligence in Healthcare and e-health (CICARE), 2014: IEEE, pp. 133-140.

Authors

Nawal Y. Abdullah

Department of Computer Techniques Engineering, Technical Engineering College/Mosul, Northern Technical University, Mosul, Iraq

ORCID
Sa’ad Ahmed Al-Kazzaz

Department of Mechatronics Engineering, College of Engineering, University of Mosul, Mosul, Iraq

ORCID

Identifiers

https://doi.org/10.33899/rengj.2023.138562.1235
Download this PDF file
PDF

Statistics

How to Cite

[1]
N. Y. Abdullah and S. Ahmed Al-Kazzaz, “Evaluation of Physiotherapy Exercise by Motion Capturing Based on Artificial Intelligence: A Review”, AREJ, vol. 28, no. 2, pp. 237–251, Sep. 2023.
Crossref
Scopus
Google Scholar
Europe PMC