- A new class of data-based learning algorithms built on neural networks has revolutionised computer vision over the last decade.
- Modern human motion analysis can play a key role in measuring gait parameters, providing insights into the walking patterns of elderly people.
- Less than one and a half years ago, the error rates of state-of-the-art 3D pose estimators were at a range of ~9-13 cm. Today, state-of-the-art 3D models show error rates of ~2-3 cm.
- The Lindera motion analysis and algorithm pipeline for estimating mobility parameters is one novel approach to delivering highly precise gait parameters that aid in the clinical decision process.
Blogpost based on the publication by Azhand, A., Rabe, S., Müller, S. et al. Algorithm based on one monocular video delivers highly valid and reliable gait parameters. Scientific Reports 11, 14065 (2021). https://doi.org/10.1038/s41598-021-93530-z
Computerised, vision-based mobility measurement systems using single monocular videos already demonstrate their potential in a range of applications (medicine, sports, entertainment, and the arts). Using neural networks to estimate human poses was initiated less than a decade ago. Today, they build the foundation for highly valid and reliable gait assessment systems at a fraction of the cost of high-tech motion assessment laboratories. Meanwhile, the developments in the field of computer vision and human pose estimation have been progressing at a breath-taking pace, with error rates in the millimetre range just on the horizon.
The challenge of motion analysis
Analysing human motion has occupied many scientists through the ages. Interestingly, by following how walking and motion have developed throughout the history of humankind, we get a summary of human development in the utilised scientific methods and tools [1-3]. In the modern era, starting from the second half of the 20th century, the computer is the primary tool that drives society. The advent of computers has made it possible to process large amounts of data much more efficiently. This also enabled the development of laboratory systems that analyse human motion. These systems, equipped mostly with four to ten video cameras, highly sophisticated motion sensor systems, wearable sensors, pressure-sensitive walking mats, etc. made measuring human mobility with highprecision.
Until today, these systems were widely utilised in various settings, spanning clinical areas, professional sports, and the entertainment business. In one way or the other, this technology has already been integrated into everyday life. Modern computer game consoles use motion analysis to allow players to bring their avatars to life on screen. Smartwatches and fitness trackers use various forms of motion detection and analysis. At the other end of the spectrum is the film industry, among others, with productions such as Avatar making extensive use of motion capture capabilities.
But there is one major issue that prevents these systems from being widely accessible, especially in the medical sector and in clinical settings. This issue is two-sided: price and operational complexity. Building a modern gait measurement lab can amount to hundreds of thousands and up to millions of euros. Additionally, only highly trained personnel can operate the lab properly. On the other hand, examples like the ones we’re familiar with from the gaming and leisure worlds are less complex, but they are not strong enough. However, motion analysis is vital, especially in the clinical setting. Our movements reflect our state of health and our state of mind. People communicate with body language. We are convinced that in the future, the whole world will have access to tools for high-precision gait and mobility analysis with negligible costs and operation complexities compared to current systems. In this vein, we understand the notion of democratising mobility analysis. Therefore, Lindera is pushing the limits of motion analysis.
Lindera’s approach to mobility analysis
The Lindera motion analysis and algorithm pipeline for estimating mobility parameters is one novel approach to delivering highly precise gait parameters that aid in the clinical decision process. The novelty of this assessment system is that it directly and mathematically calculates the gait parameters from three-dimensional skeletal poses. As input, the Lindera algorithm pipeline only needs one monocular video of a person walking, which can effortlessly be obtained with a smartphone camera. For each frame of the input video, the first two modules estimate the 2D and 3D skeletal poses, respectively, via Convolutional Neural Networks (CNNs). The third module takes these 2D and 3D skeleton data for all frames and optimises the 3D skeletons according to the given height of the person in the video. Additionally, the optimisation module estimates the 3D coordinates of the camera in the skeleton’s centred space. The pipeline’s final module is a mathematical algorithm that calculates the gait parameters using the anatomically optimised 4D skeleton data (three spatial coordinates plus time) and the camera’s position in space.
We recently published a study in Nature Scientific Reports showing how accurate and reliable the estimated parameters from the Lindera algorithm pipeline are compared to GAITRite, the gold standard of gait parameter estimation systems [4]. The study on statistical validity shows excellent results on all measured parameters of walking (gait speed, cadence, step length, and step time), for multiple walking trials from more than 40 elderly people at normal and fast walking speeds. In conclusion, to the best of our knowledge, the Lindera algorithm pipeline demonstrates the highest validity and repeatability for a skeletal tracking system based on a monocular video compared to a gold-standard assessment system and can, therefore, be considered state of the art. All that is required is a smartphone that can be employed whenever and wherever needed. The GAITRite system, on the other hand, is a 5.2-metre-long pressure-sensitive walking mat that costs 40,000 euros and must be operated by trained personnel.
Outlook – Imagining a mobility assessment tool in everyone’s pocket
Less than one and a half years ago, the error rates of state-of-the-art 3D pose estimators were at a range of ~9-13 cm [5]. Today, state-of-the-art 3D models show error rates of ~2-3 cm [6-8]. In the last one and a half years, 3D estimators have shown a 400 per cent improvement in terms of accuracy. With this progress in research and development together with the collection of more data (millions to billions of pose images with 2D and 3D ground truth labels), we expect to reach error rates in the millimetre range. Furthermore, the huge amount of data introduces more variability across environmental situations (clothing, background, lighting, etc.) and will further improve the reliability of measuring all body parts. Accuracy is just one potential point of progress for the models. Run time on devices also shows potential (e.g., mobile phones). State-of-the-art 2D estimators are running on mobile phones in real-time (20-24 frames per second) [9]. Combined with a state-of-the-art 3D model that is already running in real-time, it is possible to create a potential gait and mobility assessment system that can process input images with high accuracy and reliability on low-cost mobile devices everywhere on the planet, even without internet access. Lindera uses this leap in technological advancement to help the elderly and ultimately benefit the entire health care system and any kind of clinical setting – because the greatest technology is useless if it is not useful.