Personalized Assessment Mechanisms for English Reading Comprehension Ability Based on Image Processing and Visual-Semantic Relationship Mining

The proposed fine-grained image-text matching method for inferring students' image-text relationship understanding is based on the idea of overcoming the limitations of traditional global alignment by constructing a hierarchical visual-semantic reasoning framework to simulate and evaluate the cognitive process of establishing image-text associations during reading comprehension. This method first extracts semantically significant regional features through an attention mechanism-based image representation module, simulating the students' ability to selectively attend to key image information. Then, a global alignment module based on visual-semantic reasoning is employed to capture the overall semantic correlation between images and text, corresponding to students' macro understanding ability in grasping the main idea of the image-text pairing. More importantly, an innovative adaptive alignment module based on local cross-modal attention mechanisms is introduced. This module calculates bidirectional soft alignment between image regions and text words, precisely modeling fine-grained semantic correspondences, thereby diagnosing students' ability to construct associations at the micro level, such as word-object, attribute-feature relationships. This ability is directly related to their deep comprehension of textual reference, implicit reasoning, and other aspects. Finally, a cross-modal similarity fusion strategy integrates both global and local alignment signals to generate a matching score that reflects the depth of students' image-text relationship understanding. This score will be used as a key feature input into the personalized assessment model, quantifying their performance across multiple cognitive skills, thus providing interpretable and fine-grained data support for achieving precise English reading comprehension ability diagnosis. A diagram of the method principle is shown in Figure 1.

Figure 1. Principle of fine-grained image-text matching method for inferring students' image-text relationship understanding

2.1 Fine-grained image-text matching method for inferring students' image-text relationship understanding

2.1.1 Image representation

In the proposed fine-grained image-text matching method, the attention mechanism-based image representation forms the visual perception foundation of the entire evaluation system. This study uses a bottom-up attention mechanism, combining a Faster Region-based Convolutional Neural Network (R-CNN) object detector with a ResNet-101 backbone network, to extract 36 semantically significant regional features, denoted as N={n₁…n_j}, from each input image. This choice of technology has profound cognitive simulation value: just as students instinctively focus on key objects and significant regions in an image during reading comprehension, this detection mechanism generates candidate regions through the Region Proposal Network (RPN) and filters them based on an IoU threshold of 0.7 and a category detection probability of 0.3. The 36 regions with the highest confidence are ultimately retained.

At the feature processing level, each selected region is first converted into a 2048-dimensional feature vector via mean pooling, then projected into a unified D-dimensional semantic space through a fully connected layer. The transformation formula is as follows:

$n_u=Q_u d_u+y_u$                      (1)

This hierarchical feature learning process ensures the richness and discriminative power of the image representation: it not only preserves the original visual information but also abstracts it into higher-level semantics via deep learning networks, encoding the discrete region features into semantic vectors suitable for cross-modal matching. Notably, the pretraining of this detector on the VisualGenome dataset enables it to recognize a wide range of instance categories and attributes. When an image contains multiple semantic entities, this fine-grained regional representation can accurately capture the various visual cues that students may focus on, thus supporting the precise diagnosis of students' micro-level understanding abilities such as word-object correspondence and attribute-feature recognition.

2.1.2 Global alignment module

The fundamental reason for setting up the global alignment module in the proposed model is to simulate the cognitive process of students comprehensively understanding the image-text materials at the macro level and grasping the main idea. This is achieved by constructing a shared semantic space and performing high-order semantic alignment between the globally represented image and the globally represented text after deep relationship reasoning. For the image modality, on the basis of extracting salient region features, a graph convolutional network (GCN) is innovatively used to infer the potential semantic relationships between regions, constructing a structured visual scene. Then, a gated recurrent unit (GRU) network is applied in a sequential fusion manner to generate an image representation IG containing global semantic relationships, simulating the cognitive process of students integrating discrete visual information to form a coherent visual context. For the text modality, a bidirectional GRU network is employed to encode the bidirectional contextual semantics of the text, obtaining a global feature representation that comprehensively reflects the main idea of the text. Finally, the deep representations of both modalities are projected into the same semantic space, and their matching degree is calculated.

Specifically, the basic principle of region relationship inference is to break through the superficial perception of isolated image regions and instead simulate the deep relationship reasoning ability that humans rely on when understanding complex visual scenes. Specifically, the method first constructs a fully connected visual relationship graph $H=(N,R)$ from the $J$ region features extracted by the bottom-up attention mechanism, where each node represents a salient region, and each edge quantifies the semantic dependency strength between nodes through an affinity matrix $E$. The construction of this graph structure essentially reproduces the cognitive integration process of students when observing images—students do not isolate the recognition of "boy," "football," and "grass," but spontaneously establish semantic relations such as "the boy is playing football on the grass" in their minds. The affinity matrix $E$ is calculated by mapping pairs of nodes into a common embedding space, ensuring the semantic validity of the relationship measurement and providing a structured data foundation for subsequent deep reasoning. Assuming the two embeddings in the common embedding space are denoted as $Q_x \cdot n_u$ and $Q_y \cdot n_k$, the affinity matrix $E$ expression is as follows:

$E\left(n_u, n_k\right)=\left(Q_x \cdot n_u\right)^T\left(Q_y \cdot n_k\right)$                     (2)

On the basis of constructing the fully connected relationship graph, the model uses a three-layer graph convolution network with residual connections for iterative relationship reasoning. The GCN’s message-passing mechanism allows each node's feature to be dynamically updated based on its neighboring nodes' semantic relationships, capturing multi-hop dependency relationships through multi-layer propagation. This process aligns well with students' cognitive pattern of continuous reasoning in reading comprehension: when handling complex image-text materials, advanced understanding often requires linking multiple visual concepts. Adding residual connections not only alleviates the vanishing gradient problem in deep GCNs but also ensures that the model retains the discriminative information of the original region features while advancing relationship reasoning, achieving a balanced fusion of low-level visual features and high-level semantic relationships. Assuming the weight parameter matrix of the GCN layer is denoted as $Q_h$ and the weight matrix of the residual structure is denoted as $Q_e$, the specific reasoning formula is given as:

$N^*=Q_e\left(E N Q_h\right)+N$                  (3)

The relationship-enhanced features $N^*=\left\{n_1^* \ldots n_j^*\right\}$ output after GCN reasoning have been transformed from the initial discrete region representations into structured representations containing rich contextual semantics. These relationship-enhanced features are crucial for achieving accurate personalized assessments: in the subsequent global alignment stage, when these features are matched with the global text representations in the shared semantic space, the system can effectively distinguish whether the student has performed deep relationship reasoning or has only stayed at the superficial feature matching level. If a student struggles to understand a text that depends on complex visual relationships, the model can precisely pinpoint the cognitive weaknesses by analyzing the matching differences between the student's responses and the relationship-enhanced features.

Correspondingly, the global semantic reasoning module plays the key role of integrating the relationship-enhanced local region features into a coherent overall image semantics. The basic principle of this module is to simulate the serial, hierarchical cognitive processing strategy that humans adopt when understanding complex visual scenes: first focusing on the most salient core elements, and then gradually reasoning out more subtle semantic content based on the already constructed context. Specifically, the model sorts the relationship-enhanced features $N^*=\left\{n_1^* \ldots n_j^*\right\}$ output by the GCN in descending order according to region detection confidence, thereby setting the input sequence order for the GRU network. This priority design accurately reproduces the typical attention allocation pattern of students when processing image-text materials: Their attention is first drawn to the most prominent and easily recognizable objects in the image, and then gradually expands to secondary elements related to them, ultimately forming a complete and logically coherent visual context in their minds.

At the technical implementation level, the GRU network's sequential processing mechanism provides an ideal computational framework for this cognitive process. At each time step $u$, the update gate $c_u$ dynamically weighs the current input region feature $n^*_u$ against the visual scene description from the previous time step $l_{u-1}$ to determine the extent to which the memory unit is updated. Assuming the sigmoid activation function is denoted as $\delta_c$, and the weights are denoted as $Q_c$ and $I_c$, and the bias is $y_c$, the update gate formula is:

$c_u=\delta_c\left(Q_c n_u^*+I_c l_{u-1}+y_c\right)$                      (4)

This gating mechanism allows the model to intelligently filter and integrate information: core region features with high confidence are prioritized in the early part of the sequence, establishing solid semantic anchors for overall scene understanding.

Then, the model contextualizes and interprets regions with lower confidence or more complex semantics based on the already constructed context. Through this progressive reasoning, the hidden state $l_u$ acts as a dynamically updating "visual working memory," gradually refining the semantic representation of the entire image, and ultimately taking the hidden state at the end of the sequence $l_j$ as the global semantic representation of the image $U_H$. Specifically, assuming the tanh activation function is denoted as $\delta_l$, and the weights and biases are denoted as $Q_l, I_l$, and $y_l$, the element-wise multiplication is denoted by the symbol $\circ$, the reset gate is denoted by $e_u$, the new content added to each hidden state can be obtained through the following equation:

$\tilde{l}_u=\delta_l\left(Q_l n_u^*+I_l\left(e_u \circ l_{u-1}\right)+y_l\right)$                    (5)

Similar to the update gate, assuming the sigmoid activation function is denoted as $\delta_e$, the weights are denoted as $Q_e$ and $I_e$, and the bias is $y_e$, the calculation formula for $e_u$ is:

$e_u=\delta_e\left(Q_e n_u^*+I_e l_{u-1}+y_e\right)$                   (6)

The entire visual scene description at the current time step $l_u$ is expressed as:

$l_u=\left(1-c_u\right) \circ l_{u-1}+c_u \circ \tilde{l}_u$                      (7)

When the hidden state $l_j$ at the end of the sequence, representing the image $U_H$, is aligned and matched with the global text features in the shared semantic space, the system can effectively evaluate the student's ability level to integrate visual information at the macro level. Specifically, if the student performs poorly when handling image-text materials requiring multi-level reasoning, the difference between their response and the global representation generated by the model can be analyzed to precisely diagnose their cognitive bottleneck: whether they failed to capture the core elements of the image, lacked the ability to infer secondary information based on context, or had difficulty integrating multiple visual concepts into a coherent semantic structure.

In the proposed model, the design goal of the text encoder is to transform the linear sequence of text information into a structured representation that can deeply align with the global image representation in terms of semantics. This transformation process begins with learning the distributed representation of each word in the input text $S=\left\{q_1, \ldots, q_v\right\}$, by embedding the words into 300-dimensional vectors $a_u$. The model not only captures the surface semantics of the vocabulary but also injects the syntactic and semantic prior knowledge learned from large corpora. Assuming the embedding matrix of word vectors is denoted as $Q_r$, the expression for the word vector $a_u$ is:

$a_u=Q_r \times q_u$                   (8)

To enable effective interaction between the visual and language modalities, the model further employs a bidirectional gated recurrent unit (Bi-GRU) network to deeply encode the embedded word vector sequence. The Bi-GRU network captures the contextual dependencies of the text in both directions through independent forward and backward GRU encoders: the forward GRU encodes the sentence in its normal order, capturing the progressively unfolding semantic flow, while the backward GRU processes the text in reverse, enhancing sensitivity to subsequent context. The structure diagram is shown in Figure 2. Assuming the forward hidden state at time step $u$ and the backward hidden state at time step $u$ are denoted as $\overrightarrow{g_u}$ and $\overleftarrow{g_u}$, the expressions are:

$\overrightarrow{g_u}=\overrightarrow{G R U}\left(a_u, \overrightarrow{g_{u-1}}\right), u \in[1, v]$                     (9)

$\overleftarrow{g_u}=\overleftarrow{G R U}\left(a_u, \overleftarrow{g_{u-1}}\right), u \in[1, v]$                  (10)

2.png

Figure 2. Bidirectional GRU encoder structure diagram

This bidirectional architecture accurately simulates the cognitive process of proficient readers in understanding text—they not only predict future information based on what they have already read but also reinterpret previous content based on subsequent text, forming dynamic semantic constructions. The hidden states at the last time step from both directions are concatenated and projected through a fully connected layer, ultimately generating a global text representation $S_H$ that matches the dimensions of the image representation, ensuring that the two modalities can be meaningfully compared in the unified semantic space. Assuming the weight parameters in the text fully connected layer are denoted as $Q_H$, the bias parameters in the text fully connected layer are denoted as $y_H$, and the dimension of the final global representation feature vector of the text is denoted as $F$, the concatenation of the two hidden state vectors is denoted as [.,.], and the expression for $S_H$ is:

$S_H=Q_H \times\left[\overrightarrow{g_D}, \overleftarrow{g_D}\right]+y_H$                      (11)

The final output $S_H$ of the text encoder, as a condensed representation of the text's semantics, directly reflects the student's understanding of the main idea of the text when calculated for similarity with the global image representation $U_H$. Assuming the cosine similarity function between the image global feature U and the text global feature S is denoted as $SIM(.,.)$, the calculation formula is:

$T_H(U, S)={SIM}\left(U_H, S_H\right)$                       (12)

In the context of educational assessment, this global alignment mechanism can effectively diagnose the student's reading comprehension ability at the macro level: when the student accurately grasps the core idea of the text, their mental representation aligns highly with the $S_H-U_H$ matching mode of the model; and when there is a comprehension deviation, this inconsistency can be precisely captured through the quantitative analysis of the similarity score S_H(U, S). More importantly, by analyzing the matching patterns between the semantic features captured during the text encoding process and the image representation, the evaluation system can further distinguish whether the student's comprehension difficulty originates from insufficient vocabulary knowledge, complex syntactic structures, or weak logical reasoning skills.

2.1.3 Adaptive alignment module

The fine-grained image-text matching model needs to simulate the refined cognitive operations of proficient readers when understanding image-text materials: when facing a complex sentence and its corresponding image, they will unconsciously establish instantaneous associations between specific vocabulary and corresponding regions in the image. This momentary, local semantic matching is the foundation of deep understanding and is the key to evaluating the subtle differences in individual reading comprehension. To this end, this paper designs an adaptive alignment module based on a local cross-modal attention mechanism. The core principle is to break through the limitations of global alignment’s macro-semantics and, through constructing bidirectional fine-grained interactions between image regions and text words, precisely model and evaluate students' ability to establish image-text semantic associations at the micro level. This module does not operate in isolation; instead, under the macro framework of global alignment, it uses two complementary forms of attention—image-to-text attention and text-to-image attention—to deeply explore the local semantic correspondences between modalities.

Specifically, in terms of technical implementation, the module first constructs a feature foundation suitable for fine-grained alignment: for the image modality, the 36 region features directly extracted by the bottom-up attention mechanism are used, ensuring that each feature vector encodes a visual concept with potential semantic value; for the text modality, each word is contextually encoded using a Bi-GRU network, generating word features that fuse both the forward and backward semantic contexts. Specifically, the forward GRU network encodes the text $S$ from word vector $a_1$ to $a_u$ as follows:

$g_d(u)=\overrightarrow{G R U}\left(a_u\right), u \in[1, v]$                     (13)

The backward GRU network encodes the text $S$ from word vector $a_u$ to $a_1$ as follows:

$g_y(u)=\overleftarrow{G R U}\left(a_u\right), u \in[1, v]$                       (14)

The final feature representation of the $u$-th word, $r_u$, is calculated as:

$r_u=\frac{g_d(u)+g_y(u)}{2}, u \in[1, v]$                      (15)

Based on this, the image-text cross-modal attention mechanism calculates the soft attention weight of each image region for all text words, essentially evaluating the semantic relevance between visual regions and different vocabulary. On the other hand, the text-image cross-modal attention mechanism computes the attention distribution for each word over all image regions, revealing the specific referents of vocabulary in the visual scene. These two attention mechanisms together form a closed-loop, mutually verifying reasoning system. The fine-grained alignment signal output by this system can directly translate into quantifiable metrics that evaluate whether the student has successfully established key image-text associations, such as whether they can correctly match specific adjectives in the text with object attributes in the image or whether they can accurately associate action descriptions with dynamic regions in the image. This provides an indispensable micro-level evidence chain for achieving truly personalized ability diagnosis.

Specifically, the image-text cross-modal attention mechanism’s basic principle is to simulate the cognitive process by which students locate key information in the text through visual cues, enabling precise quantification of word-object level semantic associations. This mechanism takes as input the previously acquired image region feature set $N=\left\{n_1, \ldots, n_j\right\}$ and the text word feature set $R=\left\{r_1, \ldots, r_v\right\}$, and constructs a complete cross-modal association matrix by calculating the cosine similarity between all possible region-word pairs. The similarity $t_{uk}$ between the $u$-th region and the $k$-th word is calculated as:

$t_{u k}=n_u^T r_k, u \in[1, j], k \in[1, v]$               (16)

Apply a zero threshold as the lower bound to t_uk and perform L2 normalization, then we have:

$\hat{t}_{u k}=\frac{\operatorname{MAX}\left(t_{u k}, 0\right)}{\sqrt{\sum_{u=1}^j\left[\operatorname{MAX}\left(t_{u k}, 0\right)\right]^2}}$                   (17)

After applying zero-threshold and L2 normalization, the similarity matrix not only eliminates the interference of negative correlations but also makes the distribution of attention weights clearer.

Once the normalized similarity matrix is obtained, this mechanism further uses a softmax function to compute the attention distribution for each image region $n_u$ over all text words, achieving selective focusing on text semantics. This means that each visual region can "see" the entire text sequence but will focus with varying intensity on specific vocabulary related to its own semantics. After weighted aggregation, the resulting text feature vector $e_u^s$ is essentially a semantic context customized for region $n_u$, which integrates all the words related to that region in the text. Assuming the weight factor for the softmax function is denoted as $\eta$, the weighted text vector for the $u$-th image region is denoted as $\mu_{uk}$, then we have the expression:

$\mu_{u k}=\operatorname{softmax}\left(\eta \hat{t}_{u k}\right)$                     (18)

$e_u^s=\sum_{k=1}^v \mu_{u k} r_k$                    (19)

Next, by calculating the cosine similarity between each region feature $n_u$ and its corresponding text vector $e_u^s$, and applying average pooling to the similarities across all regions, we obtain the fine-grained image-text alignment similarity T_M (U, S). Let the obtained local similarity be $E_u^s$, and then the expression becomes:

$E_u^s=n_u^T e_u^s$                 (20)

$T_M(U, S)=\frac{\sum_{u=1}^j \operatorname{NORM}\left(E_u^s\right)}{j}$                    (21)

The above calculation process has clear cognitive interpretative value: it quantifies the extent to which each salient region in the image finds semantic support in the text and reflects the student’s ability to establish "visual concept-language concept" mappings.

This fine-grained alignment mechanism plays an irreplaceable role in achieving precise personalized assessments. By analyzing the attention weight distribution of each region-word pair, the evaluation system can trace the student’s semantic association path and diagnose specific weaknesses in their comprehension process. For instance, when the student is processing a complex scene with multiple objects, if the system detects that their attention weight on a key region is significantly low or that they establish a strong association between the region and unrelated text words, it can precisely pinpoint the source of their comprehension deviation—whether it is due to object recognition errors, attribute understanding biases, or confusion in relationship reasoning. These cross-modal attention-based diagnostic results complement the evaluations from the global alignment module, together constructing a multi-dimensional profile of the student's image-text understanding ability, providing strong data support and theoretical foundation for developing truly personalized reading ability improvement plans.

Correspondingly, the core principle of the text-image cross-modal attention mechanism is to simulate the student’s reverse cognitive process of tracing visual information through linguistic cues, that is, using each word in the text as the query subject to find its corresponding visual referent in the image regions. This mechanism first computes the attention distribution of each word feature $r_k$ over all image regions using the softmax local attention mechanism, thereby generating a weighted and focused image region feature vector $e_k^n$ for each word, which essentially represents the "semantic anchor" of the text word in the visual space. Then, by calculating the cosine similarity $E_u^s(r_k, e_k^n)$ between the word feature r_k and its corresponding visual anchor $e_k^n$, and applying average pooling to the similarities across all words, the final fine-grained text-image alignment similarity is obtained.

2.1.4 Model learning strategy

The basic principle of model learning in the fine-grained image-text matching model proposed in this paper is to organically combine the two complementary cognitive dimensions of global semantic alignment and local fine-grained correspondence through a carefully designed joint training framework. This framework constructs a multidimensional indicator system capable of comprehensively evaluating students' image-text understanding ability. The core innovation of this framework lies in adopting a dynamic balance strategy. Instead of simply linearly weighting the global similarity $T_H(U,S)$ and local similarity $T_M(U,S)$, the model autonomously learns to determine the optimal contribution ratio of the two in the final fused similarity $T(U,S)$. Let the weight hyperparameter for fusion be denoted as $\omega$, and the fused similarity is:

$T(U, S)=T_H(U, S)+\omega T_M(U, S)$                   (22)

This weighted operation enables a multifaceted representation of reading comprehension ability. Some students excel at grasping the macro theme, while others are skilled in analyzing detailed correspondences. A single-dimensional evaluation would clearly be biased. The fusion strategy ensures that the evaluation model can adaptively adjust the focus of the assessment based on the specific image-text content, thereby forming a more comprehensive and fair judgment of the student's ability.

In terms of loss function design, the model adopts the triplet hinge loss (M_TR), which emphasizes the hardest negative samples. This is crucial for providing discriminative power in personalized assessment. The loss function constructs training triplets by actively mining the most similar negative samples to the positive ones, forcing the model to learn to distinguish image-text pairs that are easily confused. Let the correctly matched image-text pair be represented as $(U,S)$, and the mismatched image-text pairs be represented as $(U, \hat{S})$ and $(\hat{U}, S)$, where $\hat{U}$=argmax_k≠UT(k,S) and $\hat{S}$=argmax_f≠S T(U, f) represent the mismatched pairs in which $\hat{U}$ and $\hat{S}$ are found. The fused similarity score is denoted as $T(.,.)$, and the margin parameter between the matching and mismatching pairs is denoted as $\beta$. The specific expression is:

$L_{I R}=M A X[0, \beta-T(U, S)+S(U, \hat{S})] +M A X[0, \beta-T(U, S)+T(\hat{U}, S)]$                      (23)

In the educational evaluation scenario, this mechanism enables the model to accurately capture students' "seemingly correct but subtly erroneous" understanding mistakes. For example, a student may have correctly understood the overall scene in the image but made a mistake in interpreting a key detail. This complex ability performance needs to be trained with difficult negative samples for the model to distinguish.

To further enhance the model's generalization ability and semantic transparency, this paper innovatively introduces a text generation task as an auxiliary training objective. This task requires the model to generate descriptive text based on the learned global image representation $U_H$. This process serves as a powerful regularization constraint, forcing the features learned by the visual encoder to contain sufficiently rich and structured semantic information to support reasonable language generation. The corresponding loss function expression is:

$L_{G E}=-\sum_{s=1}^m \log o\left(b_s \mid b_{s-1}, N^* ; \varphi\right)$                     (24)

From the perspective of cognitive diagnosis, the design of this loss function greatly enhances the model's interpretability: if the semantic representation of an image can be decoded into a coherent textual description, it indicates that the representation has indeed captured the core semantic content of the image. Therefore, the image-text matching score calculated based on this representation will also provide a more persuasive evaluation of the student's understanding level.

Finally, by jointly optimizing the matching and generation tasks, the model not only learns how to judge whether the image and text match but also gains a deeper understanding of "why" they match. This deeper causal understanding is key to achieving precise and interpretable personalized ability assessment. Let the length of the output text sequence $B=\left(b_1, \ldots, b_u\right)$ be denoted as $m$, and the hyperparameter of this sequence-to-sequence model be denoted as $\phi$. The final loss function is defined as:

$L=L_{T R}+L_{G E}$                (25)

2.2 Personalized assessment of English reading comprehension ability

The core mechanism for achieving personalized assessment of English reading comprehension ability in this paper is based on fine-grained image-text matching results, which constructs a diagnostic feedback loop that deduces the student’s cognitive process from their response. When a student completes a reading comprehension task that combines text and images, their provided answer is considered an external manifestation of their internal cognitive state. Figure 3 presents the personalized assessment framework based on fine-grained image-text matching. The system first takes the student's answer text as a query and inputs it into the pre-trained fine-grained image-text matching model to calculate its similarity scores with the standard answer and the image in the question, under both the global alignment module and the local cross-attention module. A key step in this process is performing a difference analysis: by comparing the student’s answer with the standard answer in terms of image-text matching patterns, the system can precisely infer the image-text relationship model established by the student. For example, if the student’s answer is close to the standard answer in terms of global similarity but deviates significantly in the local similarity involving specific region-word pairs, it indicates that the student may have correctly grasped the overall theme but made an error in associating a particular detail. For example, the student might incorrectly associate the phrase "gardener watering the plants" with an image region showing a person holding a stick. This analysis enables the model to go beyond simple right-or-wrong judgment and directly diagnose the potentially erroneous semantic connections that the student has made.

Figure 3. Personalized assessment framework for reading comprehension based on fine-grained image-text matching

After successfully inferring the image-text relationships established by the student, the system uses a pre-constructed "ability-indicator mapping matrix" to map these relationship patterns to specific reading comprehension ability dimensions. This mapping matrix is based on cognitive diagnostic theory and decomposes complex reading comprehension abilities into a series of observable and quantifiable fine-grained skills. For example:

• Macro-theme integration ability: Quantified by the global alignment similarity of the student's answer, assessing their ability to synthesize image and text information to grasp the core idea.
• Detail correspondence ability: Evaluated by the correlation between the student’s attention weights on key object-word pairs and the standard weights in the local alignment module.
• Implied relationship reasoning ability: Determined by analyzing whether the student’s answer successfully activates combinations of image regions that can only be associated through GCN-based relationship reasoning.
• Reference and semantic accuracy: Assessed through the text-image attention mechanism, evaluating whether the student can accurately anchor abstract vocabulary or pronouns to specific entities in the image.

Finally, the system generates a dynamic, personalized ability assessment report by integrating multimodal matching signals with the cognitive diagnostic model. This report not only contains the student’s proficiency probabilities across various ability dimensions but also provides interpretable cognitive diagnostic evidence. For example, the system might clearly indicate: "The student encountered difficulties with spatial relationship reasoning. Their answer failed to establish an effective link between the text description 'behind the tree' and the corresponding area in the image," and automatically recommend practice materials for "understanding spatial prepositions" and "analyzing complex scenes." This entire mechanism transforms advanced computer vision and natural language processing techniques into deep educational insights, realizing a shift from "grading answers" to "diagnosing thought processes." Figure 4 illustrates a typical cognitive bias in students' image-text relationship understanding and the diagnostic correction mechanism.

4.png