Soline Calla | Jean Tomaszewski | Paul Henry | Hugo Villi | Guillaume Thomann* | Nicolas Pinsault
OPEN ACCESS
Idiopathic scoliosis is a tri-dimensional deformation of the spine of unknown causes, impacting 2 to 3% of the population. It’s a chronic pathology starting during childhood or adolescence and it is associated to long term physical and psychological prejudices for the patient, leading to important healthcare costs for the system. The weight of these consequences is linked to the early diagnostic of the pathology and to the subsequent treatment. Therefore, the early screening of AIS is of major importance. In France, the actual screening technique is clinical (Adam’s test), it’s a technique reserved to personnel of the medical field and with an imperfect sensibility (71-84% depending on the literature). The final goal of our study is to develop a non-ionizing, automatic evaluation tool of the idiopathic scoliosis, that would allow an accurate screening and if possible to evaluate its severity. The preliminary work conducted with this goal is presented in this paper. After a first literature review concerning the idiopathic scoliosis, the depth camera Microsoft Kinect V2 is used to generate a tri-dimensional point cloud of the back’s surface of the patient. This point cloud is computed with the objective of quantifying an asymmetry of the patient’s back, indicator of the spine’s deformation in the axial plan, a specificity of the scoliosis. The two main axes of work were about the guarantee of the data’s reproducibility and the choice of a pertinent parameter for the back’s surface asymmetry, based on clinical and physio-pathological hypotheses and on the available literature data. The outcome of this preliminary work is a novelty surface parameter acquired in a patient-related coordinate system, with the goal to maximize its reproducibility. The first results on a few healthy patients and a scoliotic patient were encouraging. This work will obviously need to be followed by a validation study on a larger cohort of healthy and scoliotic patient. A key objective in recommending assistive technologies is to promote personal independence, social participation and life-building.
scoliosis, depth camera, screening, spine pathology, Kinect, motion capture, dynamic, forward bending test
1.1 Context
1.1.1 Definition
The idiopathic scoliosis is an evolutive, three-dimensional deformation of the spine of unknown causes [1]. It is defined as a rotation of the vertebrae around their axis, which is specific to the pathology. This leads to a more global deformation of the ribcage, because of the link between the twelve pair of ribs to the twelve thoracic vertebrae. There also is a lateral deviation of the spine in the frontal plane. Anomalies in the physiological curves of the spine, in the sagittal plane (kyphosis and lordosis), can also be excessive or insufficient compared to the norm. It's important to note that only the axial deformation is specific to the idiopathic scoliosis. Indeed, deformations in the other two planes can exist without scoliosis: A deformation in the frontal plane without axial deformation is called a « scoliotic attitude », and can be linked to chronic postural anomalies or to a lower member length discrepancy, constituting a completely different nosological setting [2]. A deformation in the sagittal plane without axial deformation is common in a number of degenerative pathologies of the spine.
1.1.2 Diagnosis criterion
The Gold Standard in the diagnosis of scoliosis is the radiography of the spine. It is based on two criterions obtained on the frontal radiography: Asymmetries on the vertebral pedicles (proof of an axial deformation) and a Cobb angle (angle between the line drawn on the upper end plate of the most tilted vertebra above the apex of the curvature and the lower end plate of the vertebra under the curvature’s apex) higher than 10°, proof of a significative deviation in the frontal plane (Figure 1). Different levels of severity can be established with this diagnosis.
Figure 1. The Cobb angle is the α angle evaluated on this radiography of the spine in the frontal plane
1.1.3 Care and treatment
The management of scoliosis depends mainly on its severity [1]. It consists in rehabilitation, for low Cobb angles (<10°), which still does not constitute a "true scoliosis" from the point of view of diagnostic criterions. Patient with a more severe scoliosis will have to wear a brace combined with rehabilitation, for Cobb angles between 10 and 25°. For Cobb angles above 25°, there will be discussion of surgery with rehabilitation because the above-mentioned non-invasive techniques are usually insufficient to correct the deformity.
1.1.4 Epidemiology
The prevalence of idiopathic scoliosis is estimated to be between 2% and 3% worldwide according to the literature [1]. The pathology generally begins in childhood or adolescence with a peak in the deformity’s evolution between 10 and 15 years old. It is associated with a decrease in the quality of life [3] and is linked to physical and psycho-social prejudices in the long term.
1.1.5. Screening methods
The screening method used in France is the Forward Bending Test (FBT), also called Adams test, a clinical method performed by a physician (pediatrician, general practitioner or school doctor) on children between 7 and 18 years of age, according to the recommendations of the HAS (Haute Autorité de Santé). The child stands upright, presenting his or her back to the doctor who examines him or her (Figure 2). First the physician will be looking for direct (lateral inflection of the spine) or indirect (asymmetry of shoulder or scapula height) signs of frontal spinal deformity. Then the child will bend forward to try and touch the ground with his hands without bending his knees: the goal here is to unmask a gibbosity, i.e. a prominence of a thoracic hemi-cage (right or left) indirectly witnessing the axial spinal deformation.
It should be noted that these signs can be very discreet at the start of the disease, hence difficult to detect, especially for an untrained examiner. The sensitivity of this method is therefore imperfect, reported between 71.1% and 84.4% according to the studies [4]. Other screening methods are used in other countries, such as Moiré topography in Japan, Canada and Singapore, based on the projection of light fringes on the patient's back to detect symmetry of the patient's back surface, which is a sign of spinal deformation. This method has a better sensitivity (94-100%) and a similar specificity (95-99%) but is more expensive and time consuming.
Figure 2. The Adams test: First straight back, (a): lateral inflection of the spine and asymmetry of height of the scapulae and shoulders are noted to a lesser extent. Then during anterior flexion of the back; (b): a right gibbosity appears, i.e., the prominence of the right hemi-thorax
1.2 Objectives
It is important to distinguish the objectives of the main study from those of our work, which is preliminary to this study. Concerning the main study, its objective is to develop on the one hand an automated and non-irradiating assessment tool for scoliosis allowing to screen it efficiently and on the other hand to assess its severity.
1.2.1 Main objective: Efficient and generalized screening of scoliosis
The tool will have to measure a parameter able to detect scoliosis with better performances (higher sensitivity, similar specificity) than the current reference screening method in France (FBT) and have a good reproducibility. Its automated nature could make it accessible to people with less medical expertise and notably non-medical school professionals in the framework of large school screening campaigns. It should also be moderate in cost and non-irradiating.
1.2.2 Secondary objective: Assessment of the severity of scoliosis
The tool will have to measure a parameter that can be correlated to the Cobb angle measured on the radiograph, the gold standard for the evaluation of the severity of the pathology.
Concerning our preliminary study, the present work focuses on two main objectives: To ensure the reproducibility of the point cloud data acquired by the depth camera, either between two different acquisitions in the same patient or between two different patients and to extract a surface asymmetry parameter from the point cloud data that would allow for the automatic detection of a gibbosity in the form of a right/left asymmetry, indicator of the deformation of the spine in the specific axial plane of the scoliosis.
2.1 Material
For this study, a Microsoft Kinect V2 depth camera was used. It is a low-cost camera initially developed for a ludic use in video games for the XBOX console. The production of the Kinect V2 has been discontinued and has been replaced by a new improved model, the Azure Kinect DK.
2.1.1 Working principles of the camera
The Kinect V2 is a “RGB-D” camera, meaning it combine a classic RGB camera with a definition of 1920x1080 pixels and a framerate of 30 fps with a depth camera (D for “Depth”) composed of an infrared emitter-receptor, with the same framerate but with a lower resolution (512*424 pixels). Depth data is obtained with the use of infrared light sent by an emitter, reflected by the object in the environment and received by the infrared captor. Over the years, these types of camera have been used for many applications in the medical field [10-12].
2.1.2 Points cloud of the back surface
In the developed application, an acquisition of the patient’s back with the depth camera will generate a points cloud placed in a fixed 3-dimensional camera-dependent frame defined by 3 axes and 3 planes (Figure 3 (a)). The 3 axes of this reference frame are:
The 3 planes thus formed are the following:
For each axial cross-section of the patient (i.e. each y), we obtain an axial surface curve of the patient's back (Figure 3 (b)) which is noted z(x). This curve generally presents a local minimum corresponding to the surface point located at the aplomb of the spine, as shown by the study of Xu et al. [5]. This local minimum is located between two local maxima on either side corresponding to the two para-vertebral muscle masses. Note that the presence and size of this local minimum is dependent to the morphology of the patient.
Figure 3. (a): Camera-dependent 3-dimensional landmark; (b): 3D point cloud of the patient's back surface and surface curve z(x) for a given y (i.e. a given axial section of the patient)
2.1.3 Technical characteristics and limitations
Based on precedent work [6-9], some technical characteristics of the equipment have to be aware of. Firstly, the resolution data: The spatial resolution in x and y is about 2-3mm when the patient is at a distance of 1m from the camera (operating distance). It degrades slowly when this distance increases (resolution of 5mm to 2.5m).
Then there is a depth shift, of 18mm on average, between the depth calculated by the camera and the depth actually measured. This offset is not a problem if it is homogeneous over the entire surface of the patient's back since we are interested in relative (side-to-side) depth values. However, some sources of heterogeneity have been identified: The offset is larger for pixels at the periphery of the image. It is more important for the darker pixels than for the lighter ones: the illumination of the patient's back must therefore be homogeneous. Finally, it increases proportionally to the temperature of the camera: There is a progressive shift increase during the warm-up of the camera until a plateau is reached after 25-30 minutes. This warm-up time must therefore be considered in our protocol. Two main artifacts are related to the technique: The "flying pixel artifact", associated with depth discontinuity zones, and the "multipath interference" related to the possible heterogeneity of the same infrared wave. These two artifacts will be considered in our acquisition protocol and data processing.
2.2 Methods
2.2.1 Acquisition protocol
For this study, based the acquisition protocol on an FBT has been chosen: the patient stands upright, with his back to the camera, about 1m from it. When the patient is in a satisfactory position, the recording starts. After a few seconds in a standing position, the patient is asked to perform an anterior flexion with the arms hanging forward, without bending the knees. The recording stops at the end of a complete flexion.
It is therefore a dynamic acquisition protocol, i.e. a succession of acquisitions (about 60) in the straight position and then during flexion. This protocol is innovative because, although other studies have analyzed the back surface using a depth camera in the upright position and at the end of anterior flexion, none has done so dynamically [13, 14].
2.2.2 Reproducibility of acquired data
In order to extract a robust surface asymmetry parameter from the point cloud data, it is essential to ensure a good reproducibility of the acquired data between two different patients and between two different acquisitions of the same patient. The general idea is that the measurement variability between two capture should be small enough not to erase pathological variation (i.e. spinal deformation).
The main source of non-pathological variability corresponds to the initial positioning of the patient before the beginning of the acquisition with a potential variability in terms of distance and rotation of the back with respect to the camera. We propose two combined precautionary methods to avoid this variability. For extrinsic precautions there are markers on the floor on which the patient should place his feet. These guarantee a fixed distance of the patient from the camera and minimize the initial rotation of the patient. For intrinsic precautions, the default three-dimensional reference frame in which the point cloud is acquired is fixed, camera-dependent, with the x-axis corresponding to the longitudinal axis of the camera, and the y and z axes corresponding to the two axes orthogonal to the x axis (respectively "up/down" and "front/back").
The best way to exclude the variability related to the initial positioning of the patient is to create a new patient-dependent and acquisition-dependent coordinate system, defined axis by axis. The y-axis becomes the cranio-caudal axis of the patient and corresponds to the general orientation of the spine, determined using the local minimum of the surface curve z(x) as explained above. The local minimum is determined for each axial slice and the patient-dependent y-axis thus corresponds to the general axis connecting each of these points.
The x-axis becomes the right-left patient axis and is defined using two prominent and depth-symmetric surface points. From the literature [15, 16], it is known that there are a few prominent anatomical landmarks that are preserved from patient to patient, including the scapular spines and posterior superior iliac spines (PSIS). In contrast to the scapular spines, the PSIS have a depth that is weakly influenced by the spinal deformity. Their depth (z) is symmetrical in all patients, so the surface points located at their aplomb is used to define the patient-dependent x-axis. On the other hand, their height (y) can be made asymmetrical due to the frontal deformation of the spine. In order to avoid this, to allocate the same y coordinate to these two surface points is decided. Let the respective coordinates of the two surface points at the left and right PSIS (x1, y1, z1) and (x2, y2, z2), the patient-dependent x-axis will thus correspond to the axis passing through the coordinate points (x1, y1, z1) and (x2, y1, z2).
Figure 4. Axis by axis definition of the patient-dependent and acquisition-dependent coordinate system: (a) the y-axis is the axis passing through the local minimum of the surface curve z(x) of each axial slice; (b) the x-axis is the axis passing through the surface points located in line with the two posterior-superior iliac spines; (c) the z-axis is the axis orthogonal to the mean surface plane passing through the origin of the reference frame (i.e. intersection of x and y). There is therefore one z-axis per degree of anterior flexion.
The intersection between the x and y axes defines the origin of our new reference frame. The z-axis (depth) becomes the antero-posterior axis of the patient and is defined for each frame as follows: first an "average surface plane" of the patient's back is defined for each acquisition that corresponds to a two-dimensional linear regression of the set of points in the cloud for the given acquisition. Then the z-axis is defined for each acquisition as the axis orthogonal to this "mean surface plane" and passing through the center of our orthonormal reference frame. The different steps in the construction of this new three-dimensional patient-dependent and acquisition-dependent coordinate system are illustrated in Figure 4.
2.2.3 Definition of the surface asymmetry parameter
In order to answer the second objective of the proposed work, the definition of a parameter extracted from the point cloud of the surface able to detect the spinal deformation in the axial plane specific to scoliosis has been studied. As seen previously, the spinal deformation in the axial plane is manifested by a right/left asymmetry of the rib cage, called "gibbosity" in clinical terms.
Figure 5. Definition of the Surface Asymmetry Index (SAI) for a given axial slice (i.e. a given y) of a given acquisition: (a) Absence of scoliosis thus axial spinal deformity no surface asymmetry, SAI≈ 1; (b) Scoliosis so axial spinal deformation and surface asymmetry, SAI >1
A parameter that accurately detects and quantifies this right/left asymmetry from the surface point cloud data of the patient's back was therefore sought. After studying parameters proposed by different studies [13-16], a surface asymmetry parameter based on the area under the surface curve z(x) was selected. This parameter can be defined according to the following steps, illustrated in Figure 5.
For an axial section (i.e. a y) of a given acquisition:
The parameter thus obtained is called "max/min lateralized AUC ratio" and is noted SAI (for Surface Asymmetry Index). For a given acquisition, all the ratios obtained for each y along the entire y-axis can be integrated. This parameter is called "integrated ratio of max/min lateralized AUC" and is noted ISAI (for Integrated Surface Asymmetry Index).
Finally, for a given patient, the integrated parameter can then be integrated a second time over the entire anterior flexion motion (i.e., the entire acquisitions) to further sensitize the latter (it is not noted as “ISAI2” parameter, ISAI was kept). It is also possible to follow the evolution of the ISAI parameter over the whole flexion movement. In patients with a pathological parameter (i.e. above a threshold that will have to be defined later), it will be possible to detect the level of flexion for which this threshold is reached. This additional data could be of interest in the evaluation of the severity of the scoliosis, which is the secondary objective of our global study.
Some initial recordings could be made in four individuals free of scoliosis and one individual with low-severity dextro-convex lumbar scoliosis (Cobb angle ≈10°).
Figure 6 represents the evolution along the y axis of the maximum and minimum lateralized AUC per axial slice as well as the SAI parameter (ratio of maximum/minimum lateralized AUC for a given axial slice).
It is interesting to note that in the healthy patient case, the SAI parameter is very close to 1 and reaches a maximum of 1.4, whereas in the scoliotic patient, the SAI parameter is >1 and rises to around 1.8 at the stage of the deformity (i.e. lumbar).
Figure 6. Evolution of maximum and minimum lateralized AUC per axial slice as well as the SAI parameter along the y axis: (a) in a healthy patient: Overlapping AUC and SAI ≈ 1; (b) in a patient with moderate scoliosis: Scattered AUC and SAI>1 and maximal at the lumbar stage (i.e. stage of maximal deformation)
As explained above, this study is a preliminary work to a larger study whose objectives have been stated above. The acquisition protocol as well as the proposed surface asymmetry parameters are new, based on the combination of a review of the literature on scoliosis, knowledge and clinical and physio-pathological hypotheses. Their validity therefore remains by definition to be demonstrated and this will be the subject of the rest of the study. This validation will depend on the desired objective.
The answer to the first objective will require testing the parameters on a large number of healthy and scoliosis patients in whom the respective diagnosis of absence/presence of scoliosis will have been proven by radiography. This will allow to determine the best pathological threshold for our parameters as well as the associated sensitivity and specificity. It will then be possible to compare these performances to the reference screening method currently used in France (FBT).
The answer to the second objective will require to look for a correlation between the parameters and the Cobb angle. In this perspective, there is high hopes in the novelty of this dynamic protocol allowing to follow the variation of the surface asymmetry throughout the anterior flexion.
Scoliosis is a chronic and progressive pathology of the spine whose delay in diagnosis is associated with a heavy physical and psycho-social toll on patients. The challenge of early detection is therefore major.
In this preliminary study, keys to the diagnosis and screening of scoliosis from a three-dimensional point of view have been identified. Using data from the surface point cloud of the patient's back acquired dynamically by a depth camera during the equivalent of an FBT, novel surface asymmetry parameters has been defined, whose first results are encouraging.
Of course, these parameters will have to be tested rigorously during the rest of the study in order to establish whether or not they meet the objectives of screening and assessment of scoliosis severity defined for this study.
[1] Negrini, S., Donzelli, S., Aulisa, A.G., Czaprowski, D., Schreiber, S., de Mauroy, J.C. (2018). 2016 SOSORT guidelines: Orthopaedic and rehabilitation treatment of idiopathic scoliosis during growth. Scoliosis, 13(1): 3. https://doi.org/10.1186/s13013-017-0145-8
[2] Harrison, D.E., Betz, J.W., Cailliet, R., Colloca, C.J., Harrison, D.D., Haas, J.W., Janik, T.J. (2006). Radiographic pseudoscoliosis in healthy male subjects following voluntary lateral translation (side glide) of the thoracic spine. Archives of Physical Medicine and Rehabilitation, 87(1): 117-122. https://doi.org/10.1016/j.apmr.2005.08.004
[3] Tones, M., Moss, N., Polly Jr, D.W. (2006). A review of quality of life and psychosocial issues in scoliosis. Spine, 31(26): 3027-3038. https://doi.org/10.1097/01.brs.0000249555.87601.fc
[4] Dunn, J., Henrikson, N.B., Morrison, C.C., Blasi, P.R., Nguyen, M., Lin, J.S. (2018). Screening for adolescent idiopathic scoliosis: Evidence report and systematic review for the US preventive services task force. JAMA, 319(2): 173-187. https://doi.org/10.1001/jama.2017.11669
[5] Xu, Z., Zhang, Y., Fu, C., Liu, L., Chen, C., Xu, W., Guo, S. (2020). Back shape measurement and three-dimensional reconstruction of spinal shape using one kinect sensor. In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pp. 1-5. https://doi.org/10.1109/ISBI45749.2020.9098481
[6] Wasenmüller, O., Stricker, D. (2017). Comparison of kinect v1 and v2 depth images in terms of accuracy and precision. In Asian conference on computer vision, pp. 34-45. https://doi.org/10.1007/978-3-319-54427-4_3
[7] Khoshelham, K., Elberink, S.O. (2012). Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors, 12(2): 1437-1454. https://doi.org/10.3390/s120201437
[8] Corti, A., Giancola, S., Mainetti, G., Sala, R. (2016). A metrological characterization of the Kinect V2 time-of-flight camera. Robotics and Autonomous Systems, 75: 584-594. https://doi.org/10.1016/j.robot.2015.09.024
[9] Ballester, J., Pheatt, C. (2013). Using the Xbox Kinect sensor for positional data acquisition. American journal of Physics, 81(1): 71-77. https://doi.org/10.1119/1.4748853
[10] Mentiplay, B.F. Perraton L.G., Bower, K.J., Pua, Y.H., McGaw, R., Heywood, S., Clark, R.A. (2015). Gait assessment using the Microsoft Xbox One Kinect: Concurrent validity and inter-day reliability of spatiotemporal and kinematic variables. Journal of Biomechanics, 48(10): 2166 2170. https://doi.org/10.1016/j.jbiomech.2015.05.021
[11] Barry, G., Van Schaik, P., MacSween, A., Dixon, J., Martin, D. (2016). Exergaming (XBOX KinectTM) versus traditional gym-based exercise for postural control, flow and technology acceptance in healthy adults: A randomised controlled trial. BMC Sports Sci Med Rehabil, 8(1): 25. https://doi.org/10.1186/s13102-016-0050-0
[12] Lecrosnier, L., Khemmar, R., Ragot, N., Rossi, R., Ertaud, J.Y., Decoux, B., Dupuis, Y. (2021). Object detection, localization and tracking-based deep learning for smart wheelchair. Modelling, Measurement and Control C, 82(1-4): 1-5. https://doi.org/10.18280/mmc_c.821-401
[13] Kokabu, T., Kawakami, N., Uno, K., Kotani, T., Suzuki, T., Abe, Y., Sudo, H. (2019). Three-dimensional depth sensor imaging to identify adolescent idiopathic scoliosis: A prospective multicenter cohort study. Scientific Reports, 9(1): 9678. https://doi.org/10.1038/s41598-019-46246-0
[14] Sudo, H., Kokabu, T., Abe, Y., Iwata, A., Yamada, K., Ito, Y.M., Kanai, S. (2018). Automated noninvasive detection of idiopathic scoliosis in children and adolescents: A principle validation study. Scientific Reports, 8(1): 17714. https://doi.org/10.1038/s41598-018-36360-w
[15] Patias, P., Grivas, T.B., Kaspiris, A., Aggouris, C., Drakoutos, E. (2010). A review of the trunk surface metrics used as Scoliosis and other deformities evaluation indices. Scoliosis, 5(1): 1-20. https://doi.org/10.1186/1748-7161-5-12
[16] Bonnet, V., Yamaguchi, T., Dupeyron, A., Andary, S., Seilles, A., Fraisse, P., Venture, G. (2016). Automatic estimate of back anatomical landmarks and 3D spine curve from a Kinect sensor. In 2016 6th IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob), pp. 924-929. https://doi.org/10.1109/BIOROB.2016.7523746
