Ever wondered about the wealth of information facial thermograms can reveal about our health? Just by analyzing face thermal images/videos we can estimate four four crucial health indicators: body temperature, heart rate, respiration rate and blood pressure. In this blog, lets delve into how thermal AI has been used to estimate these vital signs.
What are Vital Signs?
Vital signs serve as crucial indicators of our overall health, reflecting the body's fundamental functions. Heathcare professionals routinely monitor four key vital signs to assess and track various medical conditions. These vitals include
- Body Temperature: This indicates the body's ability to regulate heat. Abnormal temperatures can signal infections, inflammation, or metabolic issues.
- Pulse Rate or Heart Rate: This measures the heart's beats per minute, offering insights into cardiovascular health, exercise tolerance, and stress levels.
- Respiratory Rate: This quantifies the number of breaths taken per minute, aiding in diagnosing respiratory disorders, assessing lung function, and monitoring anesthesia.
- Blood Pressure: This records the force of blood against arterial walls during heart contractions and relaxations, helping evaluate cardiovascular health and risk factors like hypertension.
Traditional methods of obtaining vital signs rely on contact-based approaches, typically involving the attachment of sensors to the body. While these methods are generally effective for one-time data collection in adults, continuous monitoring of vital signs using these approaches presents challenges. Moreover, these conventional methods can be particularly problematic for vulnerable populations such as newborns and young children. They may cause skin irritation, discomfort, or require frequent adjustments due to the child's movements.
A Non-Invasive and Non-Contact Thermal Imaging for Health Vitals
Thermal artificial intelligence (AI) technology has potential to revolutionize vital sign monitoring in this era. The use of thermal AI for monitoring vital signs offers several advantages. It eliminates the need for physical contact based sensors, reducing the risk of skin irritation and discomfort, particularly in delicate newborn skin. Thermal imaging provides a non-invasive and non-contact monitoring, allowing for continuous real-time assessment of vital signs. This would be essential to monitor patients or new-borns remotely. This non-invasive nature of thermal AI monitoring improves the overall patient experience, reducing stress and anxiety for children and their caregivers. Further, thermal Imaging provides a privacy aware information as they do not capture the visual images.
To achieve this, thermal patterns of human face are analyzed. Human face comprises an extensive network of blood vessels, and blood flow is closely regulated to maintain thermal equilibrium. Changes in core body functions are often reflected in the thermal patterns of the face due to alterations in blood flow. Additionally, facial tissues are thin, enabling a direct transfer of heat and making the face sensitive to temperature changes.
Therefore, the first step in automation of vital sign estimation involves the identification and detection of bounding boxes corresponding to potential face regions in the images. To achieve this, widely used face detector deep learning networks such as MobileNet-SSD, YOLO, among others, were popularly employed. Once the face region is estimated, specific key points are analyzed to estimate the four basic health vitals.
Body Temperature
The utilization of body heat as an indicator of disease has been in existence since ancient times. Elevated body temperature serves as a common symptom across a spectrum of conditions, including infections (such as viral, bacterial, or fungal), inflammatory disorders (like rheumatoid arthritis, lupus, and inflammatory bowel disease), medication reactions, certain cancers (such as lymphoma and leukemia), autoimmune diseases, as well as other medical conditions like blood clots, certain types of seizures, and some endocrine disorders. Given that thermal infrared cameras directly measure temperatures, body temperature estimation from thermal imaging is one of the most prominent and straightforward use case.
Upon detecting the facial region, the subsequent step involves estimating core body temperatures based on facial temperatures. Various heuristics have been employed to determine which facial regions are most suitable for approximating body temperature. These include measuring temperatures on the inner canthus region of the eye, forehead, nose, ears, mouth (both open and closed), cheeks, among other potential indicators.
Recent investigations have indicated that full-face maximum temperatures offer optimal performance, closely followed by the maximum temperature of a broader inner canthi region, in approximating body temperature. Hence, it is possible to directly estimate variations in core body temperature by identifying the maximum temperature on the facial region. Below, I've provided some research articles that delve into body temperature estimation from thermal imaging cameras.
[1] Katte P, Kakileti ST, Madhu HJ, Manjunath G. Automated thermal screening for COVID-19 using machine learning. InMICCAI Workshop on Medical Image Assisted Blomarkers' Discovery 2022 Sep 18 (pp. 73-82). Cham: Springer Nature Switzerland.
[2] Ferrari C, Berlincioni L, Bertini M, Del Bimbo A. Inner eye canthus localization for human body temperature screening. In2020 25th International Conference on Pattern Recognition (ICPR) 2021 Jan 10 (pp. 8833-8840). IEEE.
[3] Zhou Y, Ghassemi P, Chen M, McBride D, Casamento JP, Pfefer TJ, Wang Q. Clinical evaluation of fever-screening thermography: impact of consensus guidelines and facial measurement location. Journal of Biomedical optics. 2020 Sep;25(9):097002.
Respiration Rate
Respiration rate (RR), also known as breathing rate, represents the number of breaths a person takes per minute. This physiological parameter is vital in assessing respiratory health and can vary under different circumstances, such as fever, illness, and various medical conditions. For instance, during the COVID-19 pandemic, patients with COVID-19 exhibited abnormal respiration rates. Similarly, individuals with pneumonia often present with irregular respiration rates.
Estimating the respiration rate involves utilizing the nostril region as a key reference point. As mentioned, the RR reflects the frequency of both inhalation and exhalation. During inhalation, the temperature of the tissue around the nostrils remains relatively stable. However, during exhalation, the local temperature may rise due to the release of warm, exhaled air into the surrounding environment. Thus, monitoring temperature fluctuations in the nostril region can offer valuable insights into estimating respiration rate accurately.
One of the primary challenges associated with tracking respiration rate lies in accurately identifying the nostril region throughout the entirety of a video recording. Any deviation in predicting the nostril region could lead to significant errors in RR estimation. To address this issue, many researchers have proposed the use of visual cameras for robust tracking of the nostril region. Additionally, many thermal cameras available on the market are equipped with built-in visual cameras, allowing thermal images to be synchronized with corresponding RGB images. By first obtaining the nostril region from RGB images, the corresponding temperatures can then be utilized for RR estimation.
Once the nostril region is identified, the temperatures of the nostril region are averaged over time and plotted against time to create a temperature-time curve. To ensure accuracy, smoothening and filtering techniques are applied to remove noise and artifacts from the signal. Subsequently, Discrete Fourier Transform (DFT) analysis can be employed to estimate the respiration rate from the processed temperature data. This comprehensive approach enables the reliable estimation of respiration rate using thermal imaging technology. Below, I've provided some research articles that delve into RR estimation from thermal imaging cameras.
Taken from 'Maurya L, Zwiggelaar R, Chawla D, Mahapatra P. Non-contact respiratory rate monitoring using thermal and visible imaging: A pilot study on neonates. Journal of Clinical Monitoring and Computing. 2023 Jun;37(3):815-28.'
[1] Al-Khalidi FQ, Saatchi R, Burke D, Elphick H. Tracking human face features in thermal images for respiration monitoring. InACS/IEEE International Conference on Computer Systems and Applications-AICCSA 2010 2010 May 16 (pp. 1-6). IEEE.
[2] Maurya L, Zwiggelaar R, Chawla D, Mahapatra P. Non-contact respiratory rate monitoring using thermal and visible imaging: A pilot study on neonates. Journal of Clinical Monitoring and Computing. 2023 Jun;37(3):815-28.
[3] Ruminski J, Kwasniewska A. Evaluation of respiration rate using thermal imaging in mobile conditions. Application of Infrared to Biomedical Sciences. 2017:311-46.
Heart Rate (HR)
Heart rate, also known as pulse rate, represents the number of times the heart beats per minute (bpm). This vital physiological parameter mirrors the rhythm of the heart's contractions, crucial for pumping blood throughout the body to supply oxygen and nutrients to tissues and organs. Heart rate detection provides manifold benefits across various domains, encompassing healthcare, fitness, sports performance monitoring, stress management, and beyond.
As the heart beats, it induces periodic vasoconstriction and vasodilation in the blood vessels. The human face hosts an intricate network of blood vessels, and alterations in heart rate cause fluctuations in blood flow. These minute changes manifest as subtle variations in thermal patterns, detectable using state-of-the-art thermal cameras.
Nevertheless, estimating heart rate presents a slightly more challenging task compared to respiratory rate (RR) or body temperature estimation due to the minute temperature changes associated with blood flow. Identifying the region of interest (ROI) plays a pivotal role in heart rate estimation. To scrutinize blood flow changes induced by pulsations, pinpointing an accurate ROI corresponding to a vessel with heightened temperature is imperative. Traditionally, regions such as the temple (the area between the eyes and ears), jugular region (neck), pupil region, cheek region, nasal bridge have been utilized for heart rate estimation.
Once the ROI is identified, signal processing techniques involving smoothening, filtering, and Discrete Fourier Transform (DFT) are employed to estimate heart rate. Some researchers have also proposed the utilization of WaveNet-like architectures to estimate heart rate from the time-series ROI data. Below, I have provided some research articles that delve into heart rate estimation from thermal imaging cameras.
[1] Jing B, Li H. A Novel Thermal Measurement for Heart Rate. J. Comput.. 2013 Sep 1;8(9):2163-6.
[2] Gault T, Farag A. A fully automatic method to extract the heart rate from thermal video. InProceedings of the IEEE conference on computer vision and pattern recognition workshops 2013 (pp. 336-341).
[3] Chen DY, Zou HS, Hsieh AT. Thermal image based remote heart rate measurement on dynamic subjects using deep learning. In2020 IEEE International Conference on Consumer Electronics-Taiwan (ICCE-Taiwan) 2020 Sep 28 (pp. 1-2). IEEE.
Blood Pressure
Blood Pressure (BP) refers to the force exerted by the blood against the walls of the arteries as it is pumped by the heart throughout the body. Hypertension, commonly known as high blood pressure, emerges as a significant risk factor for cardiovascular diseases such as heart disease, stroke, and peripheral artery disease. Daily blood pressure monitoring is important for prevention or early detection of these diseases.
The above image showing the importance of BP estimation during Hemodialysis. The picture is taken from 1] Oiwa K, Suzuki S, Maeda Y, Jinnai H. Applicability of deep learning for blood pressure estimation during hemodialysis based on facial images. Renal Replacement Therapy. 2024 Jan 20;10(1):2.
Estimating BP directly from thermal imaging is a challenging task and there is only a limited research on its estimation. As changes in blood pressure can influence blood flow and subsequently affect skin temperature, it is possible to detect BP using latest thermal imaging cameras. Similar to above vitals estimation, identification of face bounding box is the foremost step for estimation of BP. The existing methods consider the full face temperatures and either perform traditional image and signal processing (Independent Component Analysis followed by regression techniques) or directly use Convolutional Neural Networks (CNNs) on the facial temperature data to estimate the mean blood pressure. If you are interested to know more details, I have provided some research articles that delve into BP estimation from thermal imaging cameras.
[1] Oiwa K, Suzuki S, Maeda Y, Jinnai H. Applicability of deep learning for blood pressure estimation during hemodialysis based on facial images. Renal Replacement Therapy. 2024 Jan 20;10(1):2.
[2] Iwashita Y, Nagumo K, Oiwa K, Nozawa A. Estimation of resting blood pressure using facial thermal images by separating acute stress variations. Artificial Life and Robotics. 2021 Nov;26:473-80.
[3] Shaikh MR, Rao M, Subramaniam G. A Novel Thermal Imaging Based Transfer-Learning Model To Estimate Blood Pressure. In2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI) 2023 Apr 18 (pp. 1-5). IEEE.
[4] Liu Z, Li H, Li W, Zhuang D, Zhang F, Ouyang W, Wang S, Bertolaccini L, Alskaf E, Pan X. Noncontact remote sensing of abnormal blood pressure using a deep neural network: a novel approach for hypertension screening. Quantitative Imaging in Medicine and Surgery. 2023 Dec 12;13(12):8657.
Current Status of Thermal AI for Vital Estimation
As explored in this blog, thermal AI holds promise in estimating vital signs in a non-contact, safe, and privacy-aware manner. However, a critical question remains: How reliable are these results when compared to traditional contact-based measurements? While current research demonstrates promising outcomes, the majority of vital estimation techniques have not yet transitioned into clinical practice due to the need for thorough validation. If you're interested in tackling emerging problem statements, exploring this evolving field could be highly rewarding.
For researchers and entrepreneurs eager to venture into developing vital estimation using thermal AI, here are some recommended datasets that could significantly contribute to your investigations
1. https://youngjuncho.com/datasets/ : This resource offers different datasets for robust estimation of respiration rate estimation. Further, you can use this dataset for automated stress recognition.
2. https://paperswithcode.com/dataset/ntic-screening-dataset: This dataset is focused on COVID-19 screening and has a labeled dataset for thermal face detection. This dataset can be instrumental in identifying the face region and also the detection of key landmarks for vital estimation.
What Else?
Apart from the above, thermal AI technology has the potential to offer insights beyond traditional vital sign measurements. By analyzing thermal patterns, researchers can explore additional indicators of health and well-being, such as stress levels, circadian rhythms, and autonomic nervous system activity. This holistic approach to monitoring may enhance early detection of health issues and facilitate personalized interventions.
Interesting to know the applications of thermal imaging
ReplyDelete