Lakshaga Jyothi Murugesan* | Shanmugasundaram Ramasamy Seeranga Chettiar
© 2021 IIETA. This article is published by IIETA and is licensed under the CC BY 4.0 license (http://creativecommons.org/licenses/by/4.0/).
OPEN ACCESS
Intelligent classrooms are becoming famous. Multimodal sensor data fusion technique, where data generated from different sensors are fused to derive at some valuable insights from the classroom settings. In this paper, the proposed framework model tries to enable intelligence in a traditional Classroom Environment by experimenting on modules such as Deep Learning based Face recognition systems, Interactive Smart Mirror Assistant, Indoor Classroom Air quality monitors. Sensor hub (Jedi One) helps to visualize and analyse streaming data in real-time. Based on the proposed framework design, experimentations are carried out. The accuracy achieved in the Face Recognition System of 71% has to be increased with 80-90% by finetuning the parameters. In future, Interactive dashboards can be activated via PowerBI or Excel worksheets. Based on the questionnaire study & responses from the participants on AI & IoT systems inside the classrooms, more than 50% responded positively to support the usage of these technologies in a Classroom Environment. The future classrooms will be (DLeIC) Deep Learning enabled IoT Classrooms to lift the educational space into a new dimension. Incorporating Deep Learning on IoT systems can be a savvy and fruitful path to collaborate with generations to come.
classroom environment, classroom experiment, convolutional neural networks, intelligent systems, interactive systems, IoT devices, multimodal data, multimodal approach, sensor data fusion, edge devices
Nowadays, few Higher Education Classrooms integrate various educational technologies into their academic arena. Whereas the shift is a concept in the entire educational sector, it is not entirely implemented based on the findings of different authors in the development of a reasonable solution for their industry-ready execution. Urges such as Cost, availability, assistance, efficiency, accommodation, etc., are the prime reasons behind the incompleteness. There would be a need and scope for better and more promising strategies to fill the multiple shortages. Even when IoT acts as an intermediary between the real-life scenario and the digital version. Automation seems to be everywhere across our everyday lives, from remotely operated toys to robots or computer controls. The core aspect lies in integrating knowledge into such models with computer processors or embedded systems. As well as the rise of AI across IoT applications seems to be another sub-set of emerging jargon that many researchers are exploring. And incorporating Deep Learning into an IoT network can be a savvy and fruitful path to collaborate with generations to come. As Gul et al. said in their paper [1], the deepest technologies are the ones which fade away”. Such that, logical processing will be invisible while the knowledge processing around us will grow. Task on applying IoT in higher education institutions is addressed for governance, classrooms, teaching, learning, or other college departments. Inside this article, attention is paid to the traditional classroom or the circumstance for assessing prospects of enabling insights in the classroom setting through the use of the IoT platform. Classroom observations where obtained sensor data would then be recorded and analyzed to make long-term decisions about improved academic practices for their profit. Classroom plays a vital role as it enables us to receive information on several below-mentioned examples, such as teaching/learning, classroom management, etc., and tracking the student bodies overall within the class environment. With that said, IoT-based systems in a traditional classroom will efficiently and cheaply bring the normal activities carried in favor of their preferences and routines. The compilation of class data would provide automation or smartness and effective service by constant inspections of the daily tasks and discovering patterns and trends formed in the classroom. The meaning of smartness can be described as fast delivery. Still, the intelligence can be named auto-computed content delivery, i.e., it will help consumers through the use of certain computations to deliver highly demanded results. IoT 's supreme function is to connect anything at any time, or even anywhere, with anything, and also with anybody using any route or service [2]. The main objective of this framework design is idea is to transform IoT-based systems equipped within a classroom settings to act intelligently based on the model or algorithms invested in the computer machines or the IoT devices without or with less human intervention whenever needed for them to interact among them.
2.1 Intelligent Classrooms
As with current systems, the classroom setting in which programs/services where cameras were introduced and those camera feed work on the principle of intelligent computer vision systems/services were deemed "intelligent" in a different sense to create an "intelligent classroom." Let us know what meanings the authors provided for an intelligent class framework. In Ref. [3], for AmI (Ambient Intelligence) and RFID, cameras were used to evaluate user activity and render it flexible to their context and user requirements. Dooley et al. [4] defined how to construct an intelligent classroom setup by supporting Mixed Reality to augment the experience in the classroom. The study by Zhang and Wang [5] showed a smart classroom with an intelligent atmosphere would be feasible. Temkar et al. [6] has developed an intelligent campus (ICIoT) that can run the campus and with Sensor nodes and WSN channels using camera feeds with 16.7 percent power management efficiency. In Ref. [7], the story utilizes the Class setup in which the camera is used for user satisfaction and behavioral analysis. As studied by Gupta et al. [8], the cameras were set to enforce machine learning techniques in the classroom to identify students' progress within the lecture hall for attendance and personality psychology. Yasodharan [9] studied the robust energy management design and automation on INTEL GALILEO and Z-HOME exploration. Similarly, there was research in developing an automatic smart classroom to manage energy efficiently using Mega2560 and Blynk, based on the same scenario. Chang [10] polled how to develop an intelligent roll caller system to call the students in the classroom. Rafiq et al. [11] noted that cameras could be introduced within the classroom to track classroom activities. Aguilar et al. [12] proposed a conceptual model design that can deliver a smart classroom based on a multiagent framework. Singh et al. [13] attempted to use camera feeds to control students' actions inside the classroom using Gestures, facial recognition, lip tracking to incorporate Machine Learning algorithms. And Huang et al. [14], based on the SCADA infrastructure model, also attempt to construct a classroom model based on IoT architecture to implement an intelligent classroom. In Ref. [15], the author suggested the introduction of classrooms emotionally aware of AI. Whereas the study [16], had worked with deep learning and osmotic computing to have a smart classroom.
2.2 Deep learning-based intelligent classroom experiences
Usage of camera sensors inside the classroom to feel their environmental conditions and the activities of the students for quality education [17]. A smartphone app using CNN where class attendance was rendered dependent on their known faces [18]. Taking attendance based on SOTA techniques and obtaining 98.67% accuracy on the LFW dataset and 100% on classroom known data [19]. Besides, Mobile-oriented Cloud Hybrid Architecture used deep learning focused on non-verbal signs such as gestures and facial expressions for the emotion recognition system were researched. Methodology on building face datasets relying on two quantitative and qualitative research on the classifier to work for the student attendance management framework [20]. By detecting faces using a web-based student attendance management system together with XAMPP and MySQL for CNN and k-NN classifier [21]. Student performances were studied within the classroom for 350 students with 71 percent precision on the data collection of the Gold Standard Report than Cohen Kappa on non-verbal signs [22]. To study the affective states of the student on the e-learning environment with 83 percent, 76 percent accuracy on detection and classification, based on spontaneous and posed data sets [23]. Web-based machine translation tools to research students' actions among Korean-speaking language learners (Google translate, Navel Translate) [24]. To automatically annotate for self-reports inside the classroom based on instructors recording versions such as one voice, multi-voice, no-voice, and others to identify the time consumption [25]. To immediately lip-read speakers using CNN and even RNN models by capturing lip zone utilizing VGG and similar network versions tested with 88.2 percent accuracy on datasets developed [26]. An automated student monitoring device can track teacher and student behavior in the classroom leveraging 1800 frames of 6 videos with 10-20 students participants [27], By designing and implementing a gamified framework for a group of 120 students from higher educational sector had resulted in enhancing the student engagement, enticement, and motivation [28] using ANFIS model.
Through literature works, we have formulated a framework model that consists of different modules to enable intelligence in a Traditional Classroom Environment via IoT devices, thereby using Deep Learning techniques whenever applicable to suit the context. Due to the recent breakthroughs in multimedia processing, it is explored to combine the tasks or approaches or different types of data sources that can be termed "multimodal data sources." Based on the theory of 'semiotics,' the study relates the signs and their meaning by interpreting them in terms of words, phrases, or sentences based on semantic representations. Multimodal data sources such as video, audio, text, etc., can be approached with cognitive perception.
This framework comprises four different modules that consist of sensor nodes communicating through SPI / Bluetooth or Wireless transmission technologies and serving as Wireless Sensor Network based IoT devices operating in an Indoor Classroom Environment.
Face Recognition System (FAS) is purely focused on the frontal face recognition process where the pipeline follows the general facial feature extraction process. Standardizing and data cleaning are performed in the pre-processing stage to make the dataset normalized for further processing. Initially, in this research, teachers act as a primary model for dataset collection and experimentation on the same setup, which needs minimal parameters for experimentation. The total dataset can then be broken up with more training data so that the model can generalize well to the defined face recognition problem. The IoT device embedded with the camera model is fixed, then the face dataset inputs are obtained, video data is split into image frames and can be stored in local or cloud storage such as (GCP, AWS, Azure, etc.) if needed, but in this study, it is locally stored. The Caffe model supports the OpenCV DNN module. Then train the model, and adjust the hyper-parameters to enhance accuracy. Then using the finalized trained model in the real-time scenario to test the model performance, where Neural Network accelerator, if available, can also be used essentially, and Deep Learning computation can also be transferred to local or cloud storage (GCP, AWS, Azure, etc.), ultimately to simulate and display on-device inferences using a small scale or squeezed neural network models based on model pruning, vector quantization, model optimization, etc.
Figure 1. Proposed Conceptual Framework suited for IoT based classroom environment
Various modules act as sensor nodes that can sense, process, and respond with some information to make better decisions. The Deep Learning-based Face recognition system, an Interactive Mirror system with Voice or gesture recognition, has been experimented with within this study. Furthermore, indoor classroom Air quality can be monitored and analyzed to understand a healthy classroom better. Additionally, A Sensor Hub that can act as an IoT device monitoring system as a whole and the interaction among the IoT devices such as Raspberry Pi or Arduino can be made possible via SPI mode using nRF24L01 radio signal transmission protocols or built-in Wi-Fi or Bluetooth modules integrated to the IoT devices for wireless transmission of the data efficiently. Thus, ensuring that all these Sensor nodes in the classroom environment setup can be IoT system enabled or act as an IoT-based smart classroom environment. Therefore, ultimate proof-ready IoT-based systems can help develop a more secure, smarter classroom infrastructure based on Deep Learning for IoT systems. Finally, the live streaming data from those sensor nodes can then be monitored for better insights through storytelling Charts, leading to data analytics skills and producing Interactive dashboards on the same.
3.1 Module design and implementation
The proposed approach works based on constructing a classroom setup to design, build and implement the same research work based on multimodal sensor data fusion paradigms. As described in Figure 1, the architecture was proposed to enable Deep Learning techniques through IoT systems in a Classroom Environment. The scenario considered for this setup consists of different modules from a faculty perspective since very few types of research have been conducted to focus on the teachers present in the classroom. 1. Smart Attendance System using Deep Learning algorithms 2. AI-based Smart mirrors as an interactive assistant 3. Indoor Air Quality monitoring systems and several other IoT devices generate data collected and fused in the Universal Sensor Hub. Finally, interpreting, visualizing and analysing the streaming data in real-time resulting via Interactive dashboards. With recent advances in Deep Learning architectures and algorithm designs, it is possible to run sophisticated Machine Learning algorithms in these tiny microcontroller prototyping boards. An attempt has been made to implement Deep Learning for Arduino based systems using Edge Impulse via TinyML algorithms for audio detection systems. TinyML is a field in Machine Learning that can be applied to the embedded systems, leading to 5 key advantages in real-world applications: low-latency, low-bandwidth, privacy, and low power consumption. When we use CPUs and GPUs for training and testing, it will consume more power ranging from 60 watts to 500 watts, whereas a microcontroller consumes 100x times less power than CPU and GPUs of milliwatts or microwatts. This feature makes them perfect for deploying Machine Learning or Deep Learning algorithms in such tiny devices for lasting longer, even for years on application towards edge computations or intelligent devices.
This paper is taken as a step towards designing and implementing such sensor data fusion in a classroom setup since this may lead to gathering better and valuable insights from the teachers who act as the primary stakeholders or users of this modules inside the classroom environment. Based on this investigation, in our future research, it is needed to explore the full-fledged solution. Using this Conceptual framework, Implementation work is in progress. Making Deep Learning possible for the IoT systems considering its computational limitations on working with light-weight neural networks or the embedded neural networks equipped to an IoT device, either a Raspberry Pi or a Mobile device. Optimized CPU performances can be a better option when there is very little availability of GPU-enabled systems.
3.1.1 Face based Attendance System
A face recognition system is capable of identifying a person from an image or a captured video frame. Most face recognition systems are common in their functionality, but they compare the images already present with the new unknown face images. Euclidean distance is the one that can help to find the relation from the database of face images. A face recognition system can be applied in various contexts. To deal with face based classification problems, deep learning based algorithms and frameworks, especially Convolutional Neural Network (CNN), are the best option to opt for. CNN consists of various layers such as Convolutional layer, pooling, max pooling, fully connected layers, and an output layer. Kernels available in the convolutional layers have the responsibility to figure out the feature representations of an image. These are arbitrary values produced in the feature extraction process. There are mainly three major operations required to be followed for feature extraction, namely, face detection, face recognition, and face classification. The facial recognition process is mainly needed for face classification functionality. The feature maps will help to figure out the images more precisely and accurately on the representations produced. For this purpose, in the given classroom context, since many researchers have followed the problem to recognise students and mark their attendances. In this research, the datasets of few faculty members have been taken into consideration. Those datasets were collected from a total of 7 participants with 200 datasets each on a sum of 1000 images per each participant. The datasets were randomly taken for frontal face recognition along with spontaneous postures and non-spontaneous postures in positions (left, right, top or bottom), additionally wearing face masks. During this project work, numerous different libraries were used in deep learning algorithm implementation. Caffe is a Python based open-source AI library to run these algorithms in CPU as well as GPU. Higher FPS can be achieved using OpenCV-based DNN modules while a pre-trained model is implemented under a specific Caffe framework. OpenCV DNN because of its better inference time on CPUs can be a remarkable choice in deploying them in edge devices. To use a pretrained Caffe model with OpenCV DNN, essentials are the caffemodel file with the trained weights and the .prototxt file with the architecture structured in JSON notations comprising of Neural Network Layer definitions. Matplotlib for creating data visualization problems in GPUs. OpenCV, which is the famous Computer Vision library used to read, convert, resize the face images and hence, it will be used finally for producing real-time face recognition systems based on the live streaming videos. The final system has been implemented in the Raspberry Pi 3B+, and the same has been tested in different machines such as mobile cameras, laptop web cameras, etc. Based on the below (Figures 2-5), the whole system is implemented. The project consists of the following process of steps, namely, Dataset creation, Preprocessing the embeddings, training the face using Caffe model and then recognising the person with the given database as well as with the live streaming videos. All the datasets were evaluated using the same method, and the training dataset's size largely affects the performance. Hence, all the images are cropped to 640 * 480 pixels. The data augmentation process has been used in the implementation work to increase the total size of the dataset. To see if the accuracy can be improved. If so, then how much it can. Before making the dataset forwarded to the network, the dataset needed to be preprocessed. Hence, it is need to ensure that the cropped images have face images which is succeeded by using Haar-Cascade algorithm. Face detection algorithm tries to spot the faces in the images from the following directions such as upper left, upper right, down left, down right corners in the face images. Based on the detected points, bounding boxes were built to spot the faces. The function is given in the program then iterates through the folder. OpenFace is a python based PyTorch implementation of face recognition with deep neural networks on the famous IVPR 2015. Torch framework allows the deep neural network to be executed in the CPU. Generally, raw OpenFace model is built in Torch. OpenCV DNN module helps us to integrate and consume these torch models directly. Here in this project, the PyTorch module named openface.nn4.small2.v1.t7 has been used for face detection. Using this model, multi-layer convolution will be performed to calculate 128 vectors. Comparison is done based on the cosine similarity. It is to be ensured that a threshold has to be set, otherwise false detection will take place. Feature extraction or the face embeddings will be created for a particular face. These face embeddings are a 128 dimensional vector space. These face embeddings are then stored in a pickle file. The fresh embeddings of the same person when captured in a real frame will be matched with the stored face embeddings in the pickle file. These new face embeddings will be close enough into vector space. Hence, we can recognise the person based on the extracted 128-d feature vectors. The resulting output has embeddings pickle, which is a serialized facial embeddings and a label encoder, i.e., le.pickle contains the name labels for the model we recognize. Recognizer.pickle is the Linear SVM a machine learning model for recognizing faces. It is necessary to filter out weak recognitions based on the threshold fixed. The pretrained models loaded via OpenCV DNN cannot take input image directly unless preprocessing of the images are done to mee the network requirements. After the experimentation, from the below Figures 6 and Figure 7 on the sample outcomes, it has reached an accuracy of about 64% and 71% on the face recognition system. The implemented system needs further improvement on the accuracy against the model definition and the hyperparameter tuning techniques. The FPS can also be tested against various machines.
Figure 2. Dataset creation
Figure 3. Preprocessing stage
Figure 4. Extracting face embeddings
Figure 5. Recognizing the faculty in the real-time video stream
Figure 6. 64% accuracy in recognizing the face without wearing the mask
Figure 7. 71% accuracy in recognizing the faces wearing the mask
3.1.2 Interactive Mirror System
Figure 8. Smart Mirror module with Arduino sensors connected in API communication mode and showing the classroom timetable
Imagine a scenario when you come very close to the smart mirror mounted in the classroom, it will act as your personal assistant, power-driven with Artificial Intelligence, with compact disclosure of a few tasks and updates that you demand from it, like user-customized educational feeds, class timetable, academic calendar, weather updates, dashboards if any, and an excess amount of services on the go as the faculties desire to display. Without purchasing a DIY or an acrylic mirror, the SmartMirror module can be configured and used on the go for to serve their purpose. Once booted, the mirror will act interactively to help the user based on their questions. With Nodejs installation, on 'npm start' the following MagicMirror directory, and executes the file with the .json file. The accomplishment of AI in the SmartMirror module begins from the hotword phrase 'Magic Mirror' uttered by the Faculty. Primary knowledge repositories such as motivational greetings, weather news in the current area, educational news feeds, API profile for each Faculty, mark details have been entered in the database and fetched in the smart mirror on the website, Class timetable, Academic calendar etc. as in (Figure 8). The Offline Speech Recognition module can be used with the mirror module to convert the audio to text for communication or interaction. Besides these functionalities, the Smart Mirror can communicate with other IoT boards such as Arduino Uno via Serial Communication or through Wireless communication like NRF24L01 modules integrated or using light-weight message transfer protocols like Messaging Query Telemetry Transport (MQTT) protocols, which makes it possible for the IoT devices to talk to each other.
3.1.3 Indoor Environment Monitoring System
Raspberry Pi 3B+ has been considered as the prototype for this study. The prominent sensors used for the classroom indoor environment monitoring system are MQ135, DHT11 sensor connected to different pins in the Raspberry Pi board or the Arduino board connected via SPI pins. DHT11 Sensor that can be used to measure the temperature and humidity level in the indoor based environment in real-time. MQ-135 Sensor is used to measure the air quality based on the pollutants, namely CO2, NO, Smoke present in the air. Using this schematic diagram, the module has been setup and the acquired sensor data will be sent through Wi-Fi communication protocols to an information storage. This data can be further queried and analysed for the monitoring purpose inside the classroom. The Python script used can sensibly store the sensor temperature in the form of Celsius and time in hrs:min format.
Figure 9. Arduino Configuration with the DHT11 / MQ2 / MQ135 Sensors
The circuit diagram (Figure 9) is the main module setup in which the Temperature and Humidity sensor (DHT11) and the Gas Sensor (MQ-135) are connected to the Arduino board that can later send the values to Raspberry Pi or other devices wirelessly via NodeMCU or SPI mode based data transmission to Raspberry Pi boards. Due to its flexible nature, and IoT System has the potential to improve the air quality standards inside the classroom environment. To produce a better environment condition, the sensor data representations can be graphically analyzed inside the classroom setup.
Figure 10. Indoor Environment Monitoring System configured in Arduino IDE via SPI mode
3.1.4 Wireless Sensor Nodes / Universal Sensor Hub
Wireless Sensor Networks (WSN) are integrated into this classroom environment, which forms the traditional classroom to evolve into a smart or IoT-based classroom environment. In this research work, networked sensor nodes have typically been built using Arduino UNO, Raspberry Pi 3B+, and Arduino Nano microcontroller boards as in (Figures 11 and 12). To build wireless communication among the sensor nodes, the NRF24L01+ transceiver module has been experimented with low energy consumption. Using IoT based WSN architectures inside the classroom will make the educational sector getting valuable resources or insights from the classroom environment for the transformation into future IoT-enabled classroom architectures. When we have so many sensors or IoT devices through which data is collected in a classroom setup. Suppose an Indoor Environment Monitoring system with some Air Quality sensors and temperature sensors can provide some information on the safer educational space inside the classroom environment. So, with the help of some inexpensive sensors and components, it is possible to monitor them simultaneously and displaying the results in real-time. To choose perfect sensors for data collection environmental applications. For such experimentations, Machinechar JEDI One software is configured in Raspberry Pi 3 Model B+ board to provide software as a service to manage the sensors as well as IoT devices deployed with sensors such as Air quality sensor (MQ-135 and MQ-2), Temperature sensor (DHT11), Ultrasonic sensor (HC-SR04), Passive Infrared Sensor (PIR), etc. These sensors can sense temperature, humidity, Carbon dioxide, etc. Even though DHT11 is not used for indoor environment monitoring, DHT22 can opt for more accurate and sensitive applications. In order to visualize the data values, Arduino Serial Monitor setup can help us to know the output values as in the screenshot shown below (Figure 10).
Figure 11. Transceiver Units (ARDUINO UNO / NANO) Sensor Nodes
Figure 12. Transceiver Units (RASPBERRY PI 3B +) Sensor Nodes
Based on the captured data, notifications can be sent via Email / SMS alerts, whenever key events take place. With the help of some Python programming language. Combining all the modules, our first sensor node has been built out. Sensors can be operated to collect the data, additionally with the help of Jedi One software, based on the Unique ID associated with each IoT devices present inside the classroom environment, Thus, grabbing the information and displaying them on the Data Dashboard, With the help of plug-ins these sensor data can be moved to Jedi One plugins for monitoring and display the results in the Dashboard. Like a web server, after initial setup to start the application in the Raspberry Pi's browser window using IP-address of the Raspberry Pi board. Once done, JEDI one enabled sensor hub can be used to explore more such additional sensors inside the classroom setup. Hence, with additional sensor nodes that can wirelessly communicate the sensor data values, i.e., Wireless sensor nodes inside the classroom can be built to develop a wireless sensor network, where the Wireless Sensor nodes can send the sensor data to the Sensor Hub that can be accessed from any browser over the network. Below is the System View representation of multiple sensors available in the Classroom environment. Experimental study was done in the classroom environment and tested in Centre for Research and Development (CRD) Laboratory. In this test, 3 sensor nodes have been used for the test in a closed scenario. NRF24L01 can transmit the data approximately up to 25 meters in a closed space where the range is reduced and the signal strength is not enough to support wide-area networking applications, which can be overcome by using multi-hop wireless sensor networks. Using low power, cheap and inexpensive built-in Wi-Fi boards such as NodeMCU, or using wireless modules like NRF24L01 transceiver module with the Arduino board which uses 2.4Ghz band and baud rated of 250 kbps to 2 Mbps in a closed environment. The significant advantage of this module is low power consumption during transmission.
To communicate the data from the microcontroller boards is to be done based on Serial Port Interface SPI communication, other than I2C, ADC and GPIO modes. Likewise, several NodeMCU sensor nodes can be configured to create a Wireless Sensor Network, thus putting them available in various places inside the classroom to monitor the sensors / IoT devices in the classroom with the help of Unique ID/name associated with these devices and the collected data is sent via to JEDI One Sensor Hub. Based on two different types of dashboard views for the sensor data, IoT devices are monitored in real-time or through the system view (to view the data in real-time according to the context of the sensors in the classroom environment). This module setup is suitable for short-distance wireless data acquisition and data transmission. Later, based on the data visualization paradigms, real-time streaming data can be monitored, defining Interactive dashboards for further improvement besides semantic understanding of the classroom environment.
Our study on the usage of IoT based technologies inside the classroom environment was additionally supported by a few questionnaires to collect information on what the participants representing the educational sector expect from the educational technologies. Thus, a variety of questions designed with 24 questions related to the integration of IoT enabled cameras / IoT technologies inside the classroom environment. No scores were allotted for the given answers. Out of 50 responses from the participants, more than 50% responded positively to support the usage of IoT in the classroom environment from the figures below (Figures 13-25).
Figure 13. Educational technology in the classroom for teaching and learning (more than 35 positive responses)
Figure 14. Technology for students community in the classroom (46 positive responses)
Figure 15. Using IoT technology in the classroom (more than 30 positive responses)
Figure 16. Cameras in the classroom (more than 30 positive responses)
Figure 17. Cameras in the classroom will benefit administratives and staffs (more number of positive responses)
Figure 18. Cameras are useful for attendance and class lecture recordings (more no. of positive responses)
Figure 19. Awareness of Face recognition technologies (more than 40 positive responses)
Figure 20. Face recognition systems inside the classroom (more than 30 positive responses)
Figure 21. Cameras affect the student's emotions (more than 30 positive responses)
Figure 22. Educational technology in the classroom (more than 30 positive responses)
In the future, more participants will be surveyed on the present design setup additionally to improve the implementation as per the expected outcomes.
Figure 23. Student's attention and engagement in the classroom (more than 30 positive responses)
Figure 24. AI technology that helps the educational sector (more than 50 positive responses)
Figure 25. Deep Learning-based face recognition systems in the classroom using mobile-based IoT devices (more than 30 positive responses)
A sizeable technical culture demands its use in different fields, and it is not unique for the education sector. IoT is the ideal solution for combining IoT with Deep Learning in the classroom environment to collect, coordinate, envision, and take decisions based on the given scenarios to place deep learning techniques useful for the benefit of the educational sectors. This research work helps in discussing IoT applications in the Classroom Environment, unlike other researches conducted to build an intelligent classroom based on a different context. This paper has been organized to investigate the design and implementation of Intelligent Classroom Framework via Multimodal sensor data fusion approach. In the next section, various literatures related to deploying Internet of Things in the Classroom Environment, along with embedding deep learning with IoT based devices in the classroom experiences. In the very next section, the significant applications of the proposed framework using Deep Learning-based Computer Vision systems for a Classroom Environment as approached by many researchers were discussed and then, i.e., for Face Recognition system experimented on the Faculty members that achieved an accuracy of 71% that can be improved by finetuning the hyperparameters and the network model. This design discusses the module design individually that later on correlate and collaborate to create an Intelligent Classroom Environment. Finally, the proposed approach consisting of different modules to work on the perspective of how the multimodal sensor data fusion approach works for a classroom environment have been explored. Finally in the last section based on this research work, a short questionnaire study has been done mainly to capture the opinion of the stakeholders of this proposed system. From the study, it was evident that the current generation is very much aware of integrating this trending educational technology in the classroom environment. Cameras inside the classroom are beneficial to the faculties in a special consent against their privacy issues apart from enhancing their teaching and learning process. The same has been reflected in the study from the responses received. The study's findings suggest that the huge scope prevails for using AI-based educational technology inside the classroom environment and demands its full potential to find significant valuable insights pertaining to the individuals surrounding the IoT based classroom environments. There is an immediate need for this technology to be implemented for the huge benefit of the educational sector.
To summarize, the leading objective of this research work is to explore how traditional classrooms can be added with advanced technologies like Deep Learning technologies. The future generation classrooms can be a (DLeIC) Deep learning enabled IoT Classroom to lift the educational space into a new dimension. The future of these IoT based deep learning systems seems to amuse as more connecting things will be communicating besides less human intervention resulting in a new era of the higher education sectors and its environment that will thrive to survive. This implementation work can be considered experimentation that can lead to gaining more values from the more profound analysis of the classroom semantics for better actionable insights. This proposed architecture reveals that these state-of-the-art technologies can induce the developers, researchers, and scientists in new directions to enable Intelligence in a Classroom Environment. Based on this proposed architecture, concept implementation for a proof-ready solution to build systems can give rise to intelligent classrooms. The future of Deep Learning-based IoT systems seems to be more promising for the near future. Thus, it can be concluded that future classrooms will become an IoT-connected environment for both the staff and the students. To make Deep Learning possible for IoT devices in the classroom environment, Embedded ML can be viewed as a perfect solution. The big difference is how and what you want to use to create such a solution efficiently. Finally, it was found that future generation classrooms can be deep learning allowed IoT Classroom to take the educational evolution to a new level for both the staff and the students who are part of these systems. Furthermore, based on the design, implementation systems can be improved to evaluate the system based on its performances in real-time.
Based on this conceptual setup, implementing the above design for formal classroom experimentation is in progress. Improvement is needed in the accuracy achieved in the Face Recognition System around 80-90% by finetuning the parameters. Sensors deployed and configured in the Arduino IDE connected the Raspberry Pi via SPI port can be used to visualise through using some data analytics tools like PowerBI or Google Studio or Excel worksheets. Sensor nodes are to be tested on their communication via nRF24L01, Wi-Fi, or Bluetooth communication protocols to send the data suited for short-range transmission requirements. Interactive Smart Mirror is yet to be tested for Voice / Gesture Recognition in the classroom setup as an assistant in showing helpful sheets to the staff in the Classrooms. Sensor Hub via Machinechat is to be implemented with all those sensor nodes. The breakthrough in Deep Learning to solve complex problems and meet IoT technologies led to the beginning of this research work. This framework was thus helpful to progress the research work to the next stage of concept implementation, and the result is in the progress of further testing and report preparation. This could be taken as the first step to making the implementation more successful in the classroom scenario. Considering this research as a starting point, it can be possible to focus on building intelligent classroom infrastructure.
This work is supported by Vinayaka Mission's Kirupananda Variyar Engineering College, Salem, Tamilnadu, India, by facilitating the research through the Center for Research and Development (CRD).
[1] Gul, S., Asif, M., Ahmad, S., Yasir, M., Majid, M., Malik, M.S.A., Arshad, S. (2017). A survey on role of Internet of Things in education. IJCSNS International Journal of Computer Science and Network Security, 17(5): 159-165. http://paper.ijcsns.org/07_book/201705/20170520.pdf.
[2] Patel, K.K., Patel, S.M. (2016). Internet of Things-IOT: Definition, characteristics, architecture, enabling technologies, application & future challenges. International Journal of Engineering Science and Computing, 6(5): 6122-6131. https://doi.org/10.4010/2016.1482
[3] Ramadan, R.A., Hagras, H., Nawito, M., El Faham, A., Eldesouky, B. (2010). The intelligent classroom: Towards an educational ambient intelligence testbed. Intelligent Environments (IE), Sixth International Conference, pp. 344-349. https://doi.org/10.1109/IE.2010.70
[4] Dooley, J., Callaghan, V., Hagras, H., Gardner, M., Al_Ghazzawi, D. (2011). The intelligent classroom: Beyond four Walls. Presented at the Intelligent Campus 2011 (iC’11), Nottingham. https://doi.org/10.1109/IE.2012.31
[5] Zhang, P., Wang, J.Z. (2015). Management of intelligent campus wireless sensor networks based on runtime model. Journal of Computer and Communications, 3(7): 22-31. https://doi.org/10.4236/jcc.2015.37003
[6] Temkar, R., Gupta, M., Kalgaonkar, S. (2016). Internet of Things for smart classrooms. International Research Journal of Engineering and Technology (IRJET), 3(7): 203-207.
[7] Lim, J.H., Teh, E.Y., Geh, M.H., Lim, C.H. (2017) Automated classroom monitoring with connected visioning system. Proceedings of APSIPA Annual Summit and Conference 2017, pp. 386-393. https://doi.org/10.1109/APSIPA.2017.8282063
[8] Gupta, A., Gupta, P., Chhabra, J. (2015) IoT based power efficient system design using automation for classrooms. 3rd International Conference on Image Information Processing, pp. 285-289. https://doi.org/10.1109/ICIIP.2015.7414782
[9] Yasodharan, R., Bennaiah, D., Harikrishnan, V., Karthick, S., Prince. R. (2018). IoT based classroom automation using Arduino. International Journal of Trend in Scientific Research and Development (IJTSRD), 2(2): 2456-6470. https://doi.org/10.31142/ijtsrd9404
[10] Chang, C.H. (2011). Smart classroom roll caller system with IOT architecture. Proceedings of 2nd International Conference, IBICA, pp. 356-360. https://doi.org/10.1109/IBICA.2011.94
[11] Rafiq, N.R., Mohammed, S.F., Pandey, J., Singh, A.V. (2017). Classic from the outside, smart from the inside: The era of smart buildings. 2017 6th International Conference on Reliability, Infocom Technologies and Optimization (ICRITO) (Trends and Future Directions), pp. 580-584. https://doi.org/10.1109/ICRITO.2017.8342494
[12] Aguilar, J., Valdiviezo, P., Cordero, J., Sanchez, M. (2015). Conceptual design of a smart classroom based on multiagent systems. In Proceedings on the International Conference on Artificial Intelligence (ICAI), p. 471.
[13] Singh, S.P., Kumar, A., Singh, A., Jain, K. (2015). Smart and intelligent next generation classrooms over cloud. 2015 IEEE 3rd International Conference on MOOCs, Innovation and Technology in Education (MITE), pp. 273-277. https://doi.org/10.1109/MITE.2015.7375329
[14] Huang, L.S., Su, J.Y., Pao, T.L. (2019). A context-aware smart classroom architecture for smart campuses. Applied Sciences, 9(9): 1837. https://doi.org/10.3390/app9091837
[15] Kim, Y., Soyata, T., Behnagh, R.F. (2018) Towards emotionally aware AI smart classroom: Current issues and directions for engineering and education. IEEE Access, 6: 5308-5331. https://doi.org/10.1109/ACCESS.2018.2791861
[16] Pacheco, A., Cano, P., Flores, E., Trujillo, E., Marquez, P. (2018). A smart classroom based on deep learning and osmotic IoT computing. In 2018 Congreso Internacional de Innovación y Tendencias en Ingeniería (CONIITI), pp. 1-5. https://doi.org/10.1109/CONIITI.2018.8587095
[17] Tew, Y.Q., Tang, T.Y., Lee., Y.K. (2017). A study on enhanced educational platform with adaptive sensing devices using IoT features. In 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 375-379. https://doi.org/10.1109/APSIPA.2017.8282061
[18] Karnalim, O., Budi, S., Santoso, S., Handoyo, E.D., Toba, H., Nguyen, H., Malhotra, V. (2018). Face-face at classroom environment: Dataset and exploration. In 2018 Eighth International Conference on Image Processing Theory, Tools and Applications (IPTA), pp. 1-6. https://doi.org/10.1109/IPTA.2018.8608166
[19] Sarkar, P.R., Mishra, D., Subhramanyam, G.R.K.S. (2019). Automatic attendance system using deep learning framework. In: Tanveer M., Pachori R. (eds) Machine Intelligence and Signal Analysis. Advances in Intelligent Systems and Computing, vol 748. Springer, Singapore. https://doi.org/10.1007/978-981-13-0923-6_29
[20] Budi, S., Karnalim, O., Handoyo, E.D., Santoso, S., Toba, H., Nguyen, H., Malhotra, V. (2018). IBAtS-Image-based attendance system: A low cost solution to record student attendance in a classroom. In 2018 IEEE International Symposium on Multimedia (ISM), pp. 259-266. https://doi.org/10.1109/ISM.2018.00037
[21] Sutabri, T., Pamungkur, Kurniawan, A., Saragih, R.E. (2019). Automatic attendance system for university student using face recognition based on deep learning. International Journal of Machine Learning and Computing, 9(5): 668-674. https://doi.org/10.18178/ijmlc.2019.9.5.856
[22] Gupta, S.K., Ashwin, T.S., Guddeti, R.M.R. (2019) Students’ affective content analysis in smart classroom environment using deep learning techniques. Multimedia Tools and Applications, 78(18): 25321-25348. https://doi.org/10.1007/s11042-019-7651-z
[23] Ashwin, T.S., Guddeti, R.M.R. (2020). Affective database for e-learning and classroom environments using Indian students' faces, hand gestures and body postures. Future Generation Computer Systems, 108: 334-348. https://doi.org/10.1016/j.future.2020.02.075
[24] Briggs, N. (2018). Neural machine translation tools in the language learning classroom: Students' use, perceptions, and analyses. Jalt Call Journal, 14(1): 3-24. https://doi.org/10.29140/JALTCALL.V14N1.221
[25] Cosbey, R., Wusterbarth, A., Hutchinson, B. (2019). Deep learning for classroom activity detection from audio. 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3727-3731. IEEE, https://doi.org/10.1109/ICASSP.2019.8683365
[26] Lu, Y.Y., Li, H.B. (2019). Automatic lip-reading system based on deep convolutional neural network and attention-based long short-term memory. Applied Sciences, 9(8): 1599. https://doi.org/10.3390/app9081599
[27] Ngoc Anh, B., Tung Son, N., Truong Lam, P., Phuong Chi, L., Huu Tuan, N., Cong Dat, N., Huu Trung, N., Umar Aftab, M., Van Dinh, T.A. (2019). A computer-vision based application for student behavior monitoring in classroom. Applied Sciences, 9(22): 4729. https://doi.org/10.3390/app9224729
[28] Duggal, K., Gupta, L.R., Singh, P. (2021). Gamification and machine learning inspired approach for classroom engagement and learning. Mathematical Problems in Engineering, 2021: 9922775. https://doi.org/10.1155/2021/9922775
