DOI:10.32604/cmc.2022.019256
| Computers, Materials & Continua DOI:10.32604/cmc.2022.019256 | |
| Article |
Isomorphic 2D/3D Objects and Saccadic Characteristics in Mental Rotation
1Department of Biosciences and Biomedical Engineering, Indian Institute of Technology Indore, Indore, 453552, India
2Department of Electrical Engineering, Indian Institute of Technology Indore, Indore, 453552, India
3Discipline of Psychology, Indian Institute of Technology Indore, Indore, 453552, India
4Center for Electric Vehicles and Intelligent Transport Systems, Indian Institute of Technology Indore, Indore, 453552, India
*Corresponding Author: Premjit Khanganba Sanjram. Email: sanjrampk@iiti.ac.in
Received: 08 April 2021; Accepted: 15 May 2021
Abstract: Mental rotation (MR) is an important aspect of cognitive processing in gaming since transformation and manipulation of visuospatial information are necessary in order to execute a gaming task. This study provides insights on saccadic characteristics in gaming task performance that involves 2D and 3D isomorphic objects with varying angular disparity. Healthy participants (N = 60) performed MR gaming task. Each participant was tested individually in an acoustic treated lab environment. Gaze behavior data of all participants were recorded during task execution and analyzed to find the changes in spatiotemporal characteristics of saccades associated with the variation in angular disparity and dimensionality. There were four groups with unique combination of angular disparity and dimensionality, each with fifteen participants randomly assigned. Results indicate that the spatial characteristics of the object affect the temporal aspect of saccade (duration), whereas the spatial aspect of the saccade (amplitude) is influenced by the objects’ dimension. A longer saccade duration indicates a prolonged suppression of spatial information processing during the MR tasks with objects at convex range angular disparities. Therefore, the MR tasks with convex angular disparity become more complex to process compared to the tasks with reflex angular disparity. MR process is faster and more accurate with 3D objects compared to the 2D objects. There is an interaction between angular disparity and dimensionality in terms of mental demand, such that the MR processing with 2D objects in reflex angular disparity was more mental demanding than that of convex angular disparity; however, this trend was absent in case of 3D objects. Hence, during the MR task, the longer saccade duration implies that the tasks with convex angular disparities become comparatively more challenging. Also, the lower saccadic amplitude for 2D objects indicates difficulties in processing due to deficient visual features. The findings could help in framing the computer-based game (or videogame) concerning MR abilities for training or rehabilitation purposes.
Keywords: Mental rotation; computer-game; angular disparity; dimensionality; gaze behavior
Visuospatial information processing involves transformation and manipulation of information. One of its essential aspects is MR, a process through which rotation of mental representations of two-dimensional (2D) and three-dimensional (3D) objects/images is achieved [1]. In general, it is established that individuals are able to allocate that two 2D images portray objects of the same 3D shape even when objects are represented in different orientations [2]. However, what remains relevant from the perceptive of eye-tracking research is to examine the saccadic activities involving angular disparity aspect of dimensionality in MR. Considering saccades are rapid eye-movements involved in shifting the foveal orientation to objects of visual interest [3], understanding saccadic activities pertaining to visuospatial relationship among the chunks of stimuli is important in order to gain better insight about MR. In this effort, the study experiment reported in this paper was designed to explore saccadic characteristics of gaze behavior by developing an MR gaming task. Such visually guided task provides an ideal platform to perform a scientific inquiry in a controlled environment (see Fig. 1) as well as the prospect of future remote gaming tasks facilitated by internet of things (IoT). This has potential for application in developing cognitive training modules for individuals who need visual attention training, and the training effects can be assessed in real-time through neurophysiological measures via the internet of medical things (IoMT) [4]. Further, for investigating the long-term training effects, the heavy-sized measurements can be transferred to the analysis server followed by encrypting and compressing them via various advanced techniques such as chaos-based encryption and compression [5].
Figure 1: MR process of finding a match of the target from objects A and B
Performing an MR task requires moving visual representation of an object to understand what they are (appearance) and where they belong (space). In a typical MR task, the successive processes followed by a participant are—creating a mental image of the presented object, rotating the mental image until it is comparable to imagery or physical target, comparing, and finally making the decision [6,7]. Fig. 1, illustrates a typical MR process where a participant tries to find the match of the target from the given 3D objects A and B. The participant first selects the object A and mentally rotate it via successive rotations for checking if it matches with the target object and makes the decision. The same is then repeated for object B.
Issues concerning angular disparity and dimensionality in MR have gained attention of various lines of research. Seminal work of [8] reported a study that investigated response time (RT) while the subjects were judging whether the multiple pairs of 3D, cubed, or asymmetrical lined objects were matching with each other. The results revealed that the RT linearly increases from 1 s at 0° to 4–6 s at 180° of angular disparity (orientation difference between two objects). When the angular disparity exceeded 180°, it was reported that the RT did not continue to increase monotonically but started decreasing up to 360°. The authors interpreted the result as a process of mental simulation of the physical world. Physically rotating an object 20 degrees takes less time than rotating it 120 degrees, and the same is true for the MR process. In 1978, [9] also conducted an experiment using similar objects and showed significant sex-based differences in the MR ability, with better performance in the male group. In last two decades, neuroimaging studies have shown an association between specific brain regions with MR. It has been observed that there is rotation-dependent neural activity in dorsal areas, which indicates the spatial nature of the MR process [10,11]. Harris et al. [12], using positron emission tomography (PET) imaging technique, found the right posterior parietal lobe activation during the MR task. Gogos et al. [13] in a functional magnetic resonance imaging (fMRI) study investigated differential activation of regions within the parietal region and their functional sub-specialization by examining the effects of varying the angular disparity.
What is interesting is the growing understanding of the mechanisms underlying MR through the diverse approaches driven by the interest the topic of MR has generated. Among several theories of how a cognitive system deals with a visuospatial stimulus, the piecemeal strategy [14] has been able to attract attention of various researchers and is widely discussed. The emphasis of piecemeal strategy lies in the way a person visually perceives the individual segments of the stimuli. The piecemeal approach in MR process involves segmentation of the mental image into pieces (e.g., the objects’ arms shown in Figs. 3c and 3d), then mentally rotating one piece to align the target object, and then apply the same rotation to the remaining pieces of the objects to see whether they match [14,7]. In the context of eye-tracking approach of understanding MR, eye fixation sequences are considered to suggest a piecemeal strategy during MR tasks [15,16]. As individuals’ fixations maintain the gaze on a single location, MR is closely related to our ability to visually encode spatially distributed information [17,18]. While there are studies [7,16,19–22], that have reported the changes in gaze metrics during the MR tasks, saccadic characteristics require attention of researchers. Irwin et al. [20] suggested that there is suppression of the MR process during the saccadic eye movements. According to Larsen [22] the length of the remaining trials rose linearly as a function of angular disparity until the first saccade switched between the stimulus object, while the original processing time was approximately unchanged until the first saccade switched between the stimulus object. Balaj [22] showed that the number of fixations increases, and their frequency decreases with the increase in rotation angles of the object in an MR task. It suggested that an increase in rotation angle links with an increase in cognitive engagement and increasing task difficulty. Xue et al. [7], using machine learning, showed that the gaze metrics possess the necessary information for determining the processing strategy during an MR task.
With the increase in accessibility to computers, computer-games have become a common engagement for entertainment in the society. This is also being facilitated by the rapidly growing accessibility to gaming apps for the users through handheld devices. A distinctive feature of computer-based gaming task is that it is comprised of complex visuospatial information and a speedy response is required. Therefore, spatial abilities such as MR plays a key role while executing the task efficiently. Increasingly computer-games are also gaining popularity as useful platforms for learning and training including therapeutic purposes aimed at enhancing cognitive skills e.g., 2D and 3D computer-based MR tests [23]. Oei et al. [24] suggested that frequent training of a specific cognitive ability in a video game improves performance in the tasks that share common underlying demands. Meneghetti et al. [25] also found the transfer effects of MR training in the task requiring spatial skill.
This paper emphasizes on the variations of object characteristics i.e., angular disparity and dimensionality. The investigators consider that in an MR task, angular disparity and dimensionality are crucial for identifying the similarities between two objects [26,22]. Angular disparity is a measure of orientation and it represents the spatial aspect of an object, whereas dimensionality (whether 2D or 3D), represents its appearance. Since changes in angular disparity and dimensionality will bring variation in the MR processing strategies, which in turn, is expected to affect task performance. With respect to gaze behavior, the point is that the saccades interfere selectively with cognitive processes that rely on the dorsal stream (also known as where-pathway), such as judgments of object orientation [27] and the image or object size shows the correlation with saccadic amplitude during the scene perception [28]. In general, while individuals perform MR task faster with 3D objects as they hold additional information as compared to the 2D objects, this study makes an attempt to examine how the spatial characteristics (angular disparity) of the object and dimensionality (2D/3D) affect temporal aspect of the saccade and spatial feature of the saccade respectively. The matter of focus is that this study expects— (a) longer saccadic duration involving convex angular disparity as compared to reflex angular disparity and (b) higher saccadic amplitude for 3D objects as compared to 2D objects.
In this study, investigators have used objects with multi-arms block-and-line similar to Shepard et al. [2] and Vandenberg et al. [9] by redrawing them in Blender's Python API (https://www.blender.org). The isomorphic pairs of 2D and 3D objects with two categories of angular disparity ranges, i.e., convex angle: 0°–180°, and reflex angle: 180°–360° were utilized in the tasks. The gaming tasks were designed using the Unity 3D game engine (https://unity.com/) to identify the best matching object from a given group of similar-looking objects as soon as possible. The gaze metrics—saccade duration, saccade amplitude, saccade velocity, and pupil size were acquired while participants were performing the gaming tasks using a desktop mounted eye-tracker.
RT was recorded for each trial during the experimental task. Simultaneously, the gaming scores were also recorded to find variation in task performance. National Aeronautics and Space Administration-Task Load Index (NASA-TLX) (Android version 1.0.1. BackGateEducation, https://play.google.com/store/apps/details?id=org.texoft.nasa_tlx) was used to measure the variation in the cognitive workload (CWL) of the participants.
For the gaming task, 2D and 3D objects with multi-arms were designed using Blender's Python API. In this study, 2D and 3D objects were isomorphic in nature. The 2D objects were obtained by removing the depth of their 3D counterparts. The game was developed for the Windows platform and presented on a 24 inches display with a 1920X1080 screen resolution (Tobii TX300). The participant had to fill the personal information at the beginning of the game that included personalized user log files. In each trial of the game, a player had to identify a multi-arms object presented on the left side of the monitor and from a group of five look-alike objects (distractors) on the right side. The distractors were rotated on a fixed angle. Based on the angle of rotation and dimensionality of the objects presented, the complete game was divided into four sections:
(i) 2D objects with convex angular disparities,
(ii) 3D objects with convex angular disparities,
(iii) 2D objects with reflex angular disparities,
(iv) 3D objects with reflex angular disparities.
The four convex type angular disparity conditions for sections (i) and (ii) were AD1CA (=40°), AD2CA (=80°), AD3CA (=120°), and AD4CA (=160°) whereas, the four reflex type angular disparity conditions for sections (iii) and (iv), were AD1RA (=320°), AD2RA (=280°), AD3RA (=240°), and AD4RA (=200°). The angles for AD#RA were the reflection of their convex counterparts from AD#CA across the 0° line in Fig. 2. The objects were rotated in XY-plane (the plane of the monitor screen), considering Z-axis normal to the plane.
Note: AD= angular disparity, CA= convex angle, RA= reflex angle, ‘#’ represents the index of the rotation.
Figure 2: Angular disparity conditions for convex angle (CA) and reflex angle (RA) ranges in the experiment
The location of the correct answer was randomized, and the player had to identify the match and click by the mouse button as soon as possible to proceed to the next trial. The trials were not time-limited so that a player could take time to respond as per his/her convenience, and the time was recorded as RT. One score for each correct answer was accumulated and displayed on the right upper corner of the screen throughout the gaming task. Figs. 3a and 3b show the setting window, and login window of the game while c–d show the snapshots of the task at any arbitrarily chosen time point during the game for a 2D and a 3D section. We also integrated event-markers at the start and endpoints of each session to identify the synchronized eye-tracking data for the corresponding sessions.
This study is a 2(AD: Convex Angle vs. Reflex Angle) × 2(D: 2D vs. 3D) between-groups design. There were four groups and, in each group, objects from four categories of the varying number of arms (4, 5, 6, and 7 arms) were presented twice for each of the four angles of rotation. Therefore, each of the four groups included 32 (4: arms × 2: repetitions × 4: angular disparities) trials altogether. Tobii TX300 (Tobii Tech, Sweden) eye tracker monitor was used as a gaming console, and participants’ gaze data during the gameplay was recorded simultaneously.
Figure 3: a) Game monitor setting window, b) Participants log in window, c) 2D representation of objects in the tasks and d) 3D representation of objects in the tasks
2.3 Participants and Experimental Setup
After the Institute Human Ethics Committee of the Indian Institute of Technology Indore approved the study, 72 healthy adults were recruited for their voluntary participation in the experiment. Due to issues related to eye-tracker calibration, few participants were not included in the study. Finally, 60 participants (51 males and 9 females; all right-handed; M = 21.6 years; SD = 3.13; range = 18–29 years; with no history of mental disorders) were included in the study.
Each participant had individual experimental session. Informed consent was obtained from each of the participants before starting the experiment, and they were randomly divided into four groups, 15 participants in each group. Each of these four groups was assigned one of the four sections of the gaming tasks.
All the participants received a token of thanks with a T-shirt for their participation in the study. During the experiment, Tobii TX300 eye-tracking system was used for acquiring participants’ gaze behavior data at a sampling rate of 300 Hz. The participant was seated on a comfortable chair facing towards the monitor and ∼65 cm apart from the eye-tracker (Fig. 4b). The eye-tracker calibration with every participant was done at the beginning of the gaming session. To synchronize the eye tracker system with the trials of the gaming task, a Stim tracker system (Cedrus Corporation, USA) was used. The experiment was conducted in an isolated sound proof experimental room which was acoustically treated and specially designed for performing research involving human participants. The room had a proper lightening and ventilation to provide a comfortable environment during the experiment. The room as well as all the utilized instruments were properly sanitized before each session.
Figure 4: a) Experimental workflow; b) Schematic diagram of the experimental setup (A: Tobii TX 300 Eye-tracker; B: Data acquisition computer; C: Task monitor)
2.4 Task Execution and Data Acquisition
The experimental task was presented using Tobii Studio (version 3.3.2). Prior to the commencement of the experiment, the experimenter explained the instructions for the task execution in details to each participant. After the calibration of the eye-tracker, a practice session with four trials was provided to the participant to get familiar with the task. The task used for the practice sessions was similar but not part of the actual experimental task. After the practice session, each participant started to execute the actual gaming-task assigned to the group. A photodiode attached on the monitor and Stim tracker was used for marking each trial to get the task-synchronized eye-tracker data. The gaming scores, response accuracy, and RT were saved as an output log file of the game. After the gaming- task session ended, the CWL experienced by the participants during the experimental task was measured by administering NASA-TLX. Fig 4a shows the experimental workflow and Fig. 4b shows the experimental set-up.
2.5.1 Computation of Parameters of Gaze Behavior
Saccadic duration: It is the time taken to complete a saccade and lies within a few tens of milliseconds. The visual information takes ∼75 ms to reach the visual cortex, which is usually faster than the saccade duration; therefore, the saccades cannot be modified during the transmission of visual information. The duration of saccade has been suggested as a period with no visual intake [29,30] and cognitive processing suppresses [19] during saccades. The investigators computed average saccadic duration (in millisecond) for each of the four groups.
Saccadic amplitude: Saccadic amplitude (or saccadic length) is the size of the saccade; it is the visual angle between two consecutive fixations usually measured in degrees or minutes of arc. A higher saccadic amplitude means the eye has travelled a greater distance during the saccade. In this study, average saccadic amplitudes were computed for each group in degree.
Saccadic velocity: Saccadic velocity is the ratio of saccadic amplitude and duration, and it is usually reported in degree per second. Velocity level can change along with eye movement; therefore, the average of the velocity is usually computed. Saccadic velocity has been used as a measure of cognitive activation level, often called arousal level. Low vigilance decreases saccadic velocity [31], and so does tiredness [32]. However, McGregor and Stern [32] suggested that the saccadic velocity change as an index of ‘fatigue’. The decrease in saccadic velocity indicates mental fatigue (tiredness) [33].
Pupil size: Another important and widely used measure of gaze behavior is pupil size, usually measured as pupil diameter (in mm). The pupil dilates with the increasing complexity of the task [22] and its size instantly responds to the changes in the CWL [34].
2.5.2 Computation of Parameters of Game Performance
RT: To measure the participant performance during the MR task, the RT is an important parameter. It is the elapsed time between the task/trial presented, and the response provided. Mean RT for the four groups were calculated.
Gaming task score: The MR task was presented in the form of a computer-based game, and the participants were rewarded by one mark for each correct answer. Accumulated score at any point of time was displayed on the right upper corner of the monitor throughout the session. After completion of a task, the total marks scored were used as another measurement of performance in the study. The gaming scores, response accuracy, and time were saved as an output log file of the game.
CWL: CWL of each participant was measured using the android-based application of NASA-TLX. The test was done on all participants when they completed the computer-based gaming task. The participants answered the questionnaires of NASA-TLX by rating each of the six subscales (mental demand, physical demand, temporal demand, performance, effort, and frustration level) according to the CWL they experienced during the task execution. The rating scale ranged from very low to very high for all dimensions except for the performance, which varies from perfect to failure. After rating, the CWL for each subscale and the overall CWL were automatically calculated by the application.
The data were analyzed offline using MATLAB®, Version 2010b (MathWorks, USA) and SPSS® Version 22 (IBM Corporation, USA). Tobii Studio provided eye-tracking data for each participant. The investigators first extracted the saccadic parameters and pupil size for each of the 60 participants and averaged these metrics within the four groups for further statistical analysis. Using the distribution within each group and their group average, we further conducted the statistical analysis.
To infer the statistical significance, the investigators used alpha level 0.05 throughout this paper. Fig. 5 shows the major steps in the study for analyzing the gaze data from the eye-tracker, task performance measures from the MR gaming-tasks, and CWL scores from the NASA-TLX test (Tab. 3).
Figure 5: Framework of different sets of data
Analysis revealed a significant main effect of angular disparity on saccade durations i.e., convex angular disparity had longer time duration than reflex angular disparity see Fig. 6a and Tab. 1. There was no significant main effect of dimensionality on saccade durations; see Tab. 1. There was no significant interaction between angular disparity and dimensionality for their interactive effect on saccade durations; F(1, 56)=0.496, p = .484. The two-way ANOVA test on saccadic amplitude for 2(AD: Convex Angle vs. Reflex Angle) × 2(D: 2D vs. 3D) between-groups design showed a significant main effect of dimensionality i.e., there was higher saccadic amplitude for 3D objects as compared to 2D objects. see Fig. 6b and Tab. 1. The analysis revealed no significant main effect of angular disparity on saccadic amplitude, although convex angular disparity range has higher saccadic amplitude than the reflex range; see Tab. 1. There was no significant interaction between angular disparity and dimensionality for their interactive effect on saccadic amplitude, F(1, 56) = 0.215, p = .645. Further, a two-way ANOVA on saccadic velocity for 2(AD: Convex Angle vs. Reflex Angle) × 2(D: 2D vs. 3D) between-groups design showed a significant main effect of dimensionality. 3D objects had higher saccadic velocity in contrast to 2D see Fig 6c Tab. 1. There was no significant interaction between angular disparity and dimensionality concerning their interactive effect on saccadic amplitude, F(1, 56) = 0.005, p = .941 see Fig. 6d and Tab. 1. The boldface in Tab. 1 represents a significant p-value.
A two-way ANOVA for 2(AD: Convex Angle vs. Reflex Angle) × 2(D: 2D vs. 3D) between-groups design was performed in order to examine variation in RT. Neither AD nor D showed a significant main effect on RT. Also, there was lower RT for 3D as compared to 2D; see Tab. 2 and Fig. 7c. There was no significant interaction between angular disparity and dimensionality concerning RT.
The correlation test was performed to investigate the correlation between task RT and gaze behavior. The correlation test was performed to investigate the correlation between task RT and gaze behavior. The result showed a significant negative correlation between RT and saccade duration r(60) = –.330, p< .05, 2-tailed. Similar trend was observed between RT and saccadic amplitude as well r(60) = –.376, p <. 05, 2-tailed. Fig. 7a shows the negative correlation between RT and saccadic duration while Fig. 7b demonstrates the negative correlation between RT and amplitude of the saccades.
Figure 6: Overlapped box-violin plots a) Saccade duration for convex and reflex angular disparities, b) Saccade amplitude for 2D and 3D object dimensionalities, c) Saccade velocity between 2D and 3D object dimensionalities, and d) Pupil size between 2D and 3D object dimensionalities
The analysis showed a significant main effect of dimensionality on gaming task-scores with lesser value for 2D than the 3D, but no significant main effect of angular disparity was found for the same; see Fig. 7d, and Tab. 2. The test revealed no significant interaction between angular disparity and dimensionality concerning their interactive effect on the score of the task, F(1, 56) = 0.106, p = .746. A significant positive correlation between the score of the task and the pupil size was found r(60) = .280, p <. 05). Fig. 8a shows the correlation plot between the task-score and pupil size.
A two-way ANOVA for 2(AD: Convex Angle vs. Reflex Angle) × 2(D: 2D vs. 3D) between-groups design was performed in order to examine CWL. The results showed no significant effect of angular disparity, F(1, 56) = 0.531, p = .469 as well as dimensionality, F(1, 56) = 0.108, p = .744 on the overall CWL. We also did not find a significant interaction between the angular disparity and dimensionality, F(1, 56) = 0.383, p = .539. However, there was a significant interaction between angular disparity and dimensionality, F(1, 56) = 4.601, p = .036, (see Fig. 8d) in terms of mental demand subscale, in such a way that the MR processing with 2D objects in reflex angular disparity was more mental demanding than that of convex angular disparity; however, in the case of 3D objects, this pattern did not exist. Also, there was a significant negative correlation between mental demand and pupil size r(60) = –.278, p <. 05); see Fig. 8b.
Figure 7: Correlation between a) Saccadic duration and average RT, b) Saccadic amplitude and average RT, c) Average RT for two angular disparity and the two dimensionality conditions, and d) Mean gaming scores for two angular disparity and the two dimensionality conditions
Figure 8: a) Correlation between pupil size and task score, b) Correlation between pupil size and mental demand index, d) Overall CWL index for all the four groups, d) Interactive effect of angular disparity, and dimensionality on mental demand
Visuospatial information processing in MR gaming task relies on perceiving, analyzing, manipulating, and transforming visual patterns. While looking at objects in order to perform the gaming task, human eyes are bombarded with various aspects of visuospatial information eye-tracking system is a useful tool to analyze gaze behavior of the users.
This study demonstrates that the spatial characteristic (angular rotation) of an object in the MR task influences the temporal characteristics (saccade duration). The significant effect of angular disparity on saccade duration implies a prolonged suppression of spatial processing [19,20]. The longer saccade duration indicates an elongated suppression of spatial information processing during the MR tasks when the object's angular disparity was in convex range. It is important to note that RT increases monotonically from 0° to 180° of angular disparity but decrease from 180° to 360° by identifying a shorter path using structural properties independent from the object's orientation [35]. Object view similarity often varies in response to increased angular disparity. Images/objects differing by a large angular disparity might be more similar to those with a smaller rotation difference [36]. When object rotates with a reflex angular disparity, it starts turning towards the reference object's orientation which eases the object recognition. Thus, with increasing angular disparity the comparison gets easier and RT decreases in reflex angular disparity range in contrast to the increasing RT with rotation in convex angular disparity. Thus, the complexity and time-consuming nature of convex range rotation are higher than the reflex which leads a longer saccade duration in convex range [37,38]. Increasing saccade duration shows the decreasing processing capacity in convex range than reflex range [39]. Interestingly, however, it is possible only in the case a participant chooses the shortest path to rotate an object from 180° to 360° range in the opposite direction because saccade duration would be higher in terms of the travelled path for rotation in convex range than the reflex. A negative correlation was found between saccade duration and fixation count that means higher fixation count in reflex angle range than convex. Since the number of saccades between stimuli decreases with experience, participants encode more complete representations, requiring fewer encode–rotate–compare iterations [40,21]. The cognitive emphasis on the angles or major feature of the visual stimulus has significant contribution in the creation of an internal representation of the stimulus, and when an original stimulus is perceived and interpreted by matching, similar fixation pattern is used on the matched stimuli. Subjects fixate on the same points, edges, or angles of the paired object in the same order as they did when encoding the previous stimulus [41]. In this case, the RT is possibly also affected by object features, e.g., the number of arms and dimensionality, rather than being influenced by angular disparity only. It can provide an external cue to anticipate the object, which might help the participant to make a correct decision. Thus, the RT can also be affected by anticipation and reinforcement learning. The participants might have applied different strategies apart from MR and found this strategy right for finding correct answers. If this strategy succeeds in the coming trials, it gives confidence to the participant to continue the same approach, which worked in each group, and that is why there is no significant difference in terms of RT either in case of angular disparity or dimensionality. According to Van Duren et al. [42] Van Duren and Sanders, target classification and response selection may also occur during a saccade. They hypothesized that during saccades, perceptual processes like stimuli encoding could be suppressed, but post-perceptual processes like target classification and response selection might not. Though the previous studies [2,35,27] expected to rotate the object in the same direction for both angular disparity ranges, we are not sure about the path participants followed to rotate and match the objects. Since the statistical test did not show a significant effect of angular disparity on RT, we can assume that the participants matched the arms of the target with the options instead of mentally rotating them.
In the current study, investigators did not find a significant effect of dimensionality on saccadic duration. The saccades interfere with the visual information which is processed by dorsal pathways (where pathways), such as judgments of object orientation [27]. The spatial characteristics of the object get suppressed during saccade in an MR task. Here, in this study, the angle is the parameter, which is affecting objects’ spatial orientation or location, whereas dimensionality reflects objects’ physical appearance. This could be the reason that saccade duration was unaffected by dimensionality. It is well known that during saccadic eye movements, exposure to visual stimuli is decreased, a phenomenon known as “saccadic repression” [43,44]. The suppression of visual sensitivity during saccades is primary because of visual masking [45]. The intake of objects’ visuospatial features is masked while planning to make the saccade, during which the brain sends signal to suppress the visual pathways [46]. Though the subject focuses on the target during saccades, the suppression in perceiving angular disparity affects individual's performance in MR. Suppression of cognitive processes during saccades might be expected to occur only when limited processing resources must be shared [19]. According to the dual-task theory, cognitive processing can be suppressed during saccades only when shared processing structure are used. While a diverse network of brain structures is involved in the generation of saccadic eye movements, the frontal (and supplementary) eye fields and the posterior parietal cortex tend to be the most important cortical regions [47]. Thus, dual-task hypothesis suggests that during saccades, cognitive processes that include these same brain regions will be suppressed. As suggested by several neurophysiological studies (e.g., [48–50], MR is one such process that involve parietal region. It was also thought that perceptual memory gets updated during saccades so that information gathered across fixations can be processed.
There was significant effect of dimensionality on saccadic amplitude and velocity. The lower saccadic amplitude and velocity were found for 2D objects, indicating that the tasks were more challenging and attention demanding than the 3D objects. The reason behind higher amplitude and velocity of saccades in the case of 3D objects can be understood in terms of how the brain anticipates the three-dimensional consequences of eye movements because of the depth perception of the objects [51]. Saccadic amplitude has been found directly proportional to the stimulus (image) size and dimension [28]. As the saccadic amplitude increases, it means that a greater portion of the object is being interpreted and a more complete representation of the object is being formed [40]. Surface depth gradients are followed by saccadic eye movements during random exploration of visual images/objects. Thus, surface orientation has a major impact on eye movements regardless of the task, when viewing stimuli in three dimensions [52]. The type of stimuli influences even simple eye movement properties. Furthermore, saccade dynamics vary when viewing 3D images vs. 2D images as the saccade velocity increases with the addition of 3D depth cues. More comprehensive images are explored faster i.e., the duration of fixations is decreased for images containing more information [53]. Angular disparity did not significantly affect saccadic amplitude and saccadic velocity. The possible reason behind this could be the reflexive nature of these saccades and dealing with the ventral stream, also known as what stream. A surprising and swift route connects visual processing in the ventral pathway (stream) with saccadic eye movement programming [54]. In this case, the object's appearance (dimensionality) influences on the effects of the spatial characteristics (angular disparity). The saccade velocity is mathematically closely connected with the amplitude, so the effect of dimensionality on saccade velocity was observed similar to the saccade amplitude. For rising saccade amplitude, not just the mean but also the maximum velocity increases. The amplitude of the saccade is a regulated quantity in the saccadic movement of the eyes on which the clear reception of the required visual information depends. Each saccadic eye movement is performed at maximum velocity, allowing for the necessary precision to be achieved. Therefore, when a participant scrutinizes an object, the saccadic amplitude gets shorter than during overview scans, which often happens if the task difficulty and complexity increases.
It is interesting to see a significant effect of dimensionality on the gaming task score. The lower scores under the 2D objects category reflect that the participant faced more challenges while mentally rotating the 2D objects as compared to the 3D (i.e., speed-accuracy trade-off). This trend was present in case of saccadic amplitude as well. The findings of the current study are in agreement with reports of Neubauer et al. [55] that claimed higher RT and a lower score of tasks for 2D presentation mode in MR. Though few previous studies suggested that 3D objects are more challenging in terms of correct responses as well as RT. However, those studies compared different non-isomorphic 2D and 3D objects. In the current study, the 2D objects were isomorphic to the corresponding 3D object obtained by removing the depth (Z-axis). 3D objects have comparatively more detailed information, so the brain may start anticipating the consequences of their rotation before the saccadic eye movement starts [51]. This phenomenon makes the MR process faster and reduces RT in an active task. Though, there was insignificant effect of dimensionality on RT, however, the mean RT was found lower for 3D objects as compared to 2D objects. The lower mean RT and high gaming task score collectively support the point that 3D condition can provide an external cue for anticipation, which may help the participant to make correct decisions and execute the task more efficiently in contrast to 2D condition [56].
Angular disparity as well as dimensionality did not significantly affect the overall CWL. Result showed a significant interaction of angular disparity and dimensionality for mental demand sub-scale of NASA-TLX in such a way that the MR processing with 2D objects in reflex angular disparity was more mental demanding than that of convex angular disparity; however, this trend was absent in case of 3D objects. Interestingly, lesser pupil size has been found in the case of the 2D object than 3D. Here, pupil size is getting affected by depth perception of the object. The basic act of recollecting the information will cause the pupil to dilate.; however, when the brain has to store and process the information at a faster pace than it can handle, the pupil contract [57]. Depth perception is helping the participant to match the target object with the given option. That is why we have found a negative correlation between pupil size and mental demand. The lesser pupil size in 2D means higher mental demand in 2D compared to 3D.
The study analyzed gaze behavior and task performance during a computer-based MR gaming task and investigated the effect of the angular disparity and dimensionality on user performance while executing the task. The isomorphic 2D and 3D images were used to minimize the other confounding factors than the dimension differences. The findings reveal correlations among objects’ spatial and appearance characteristics, gaze metrics, and task performances. The study showed that the spatial characteristic of the object, i.e., angular disparity, affects the temporal aspect of the saccade, i.e., saccade duration. In contrast, the spatial aspect of the saccade, i.e., the saccade amplitude and velocity, gets influenced by the objects’ dimension. More suppression of MR processing during convex angular disparity is making it more effort seeker, than the reflex one. The study also reveals that MR may occur faster and more accurately with 3D objects as they hold additional information, such as depth. As the 2D objects include less object information, a prolonged search for object features may delay the response. The interaction between angular disparity and dimensionality for mental demand in the gaming task indicated that they together could turn the task more challenging, which may increase the RT. Though the study has a few limitations, the findings enlighten the association of dependent and independent parameters in the MR task, which can be considered for making the strategies for improving the performance in a task involving spatial manipulation. Future studies will benefit if 2D objects are completely free from possible cue of depth. In the current study 2D objects were generated by removing the depth effect from the 3D objects. The results may also help in planning the structure of a computer-based game (or videogame) to train MR abilities for training or rehabilitation purposes. However, encountering the limitations would undoubtedly provide more confidence and possibly other significant effects of variations in angular disparity and dimensionality.
Acknowledgement: The authors thank the participants who extended their support in performing the study. The authors would also like to thank Mr. Mrigank Krishan for his help in designing the MR gaming-task used in the study. Authors share equal contribution as co-first authors.
Funding Statement: Akanksha Tiwari was supported by DST-INSPIRE fellowship from the Department of Science and Technology, Government of India.
Conflicts of Interest: The authors declare that they have no conflicts of interest to report regarding the present study.
1. M. I. Núñez-Peña and J. A. Aznar-Casanova, “Mental rotation of mirrored letters: Evidence from event-related brain potentials,” Brain and Cognition, vol. 69, no. 1, pp. 180–187, 2009. [Google Scholar]
2. S. Shepard and D. Metzler, “Mental rotation: Effects of dimensionality of objects and type of task,” Journal of Experimental Psychology: Human Perception and Performance, vol. 14, no. 1, pp. 3–11, 1988. [Google Scholar]
3. P. Termsarasab, T. Thammongkolchai, J. C. Rucker and S. J. Frucht, “The diagnostic value of saccades in movement disorder patients: A practical guide and review,” Journal of Clinical Movement Disorders, vol. 2, no. 14, pp. 1–10, 2015. [Google Scholar]
4. J. Chen, S. Sun, N. Bao, Z. Zhu and L. B. Zhang, “Improved reconstruction for CS based ECG acquisition in internet of medical things,” IEEE Sensors Journal, In press, 2021. https://doi.org/10.1109/JSEN.2021.3055635. [Google Scholar]
5. J. Chen, Y. Zhang, L. Qi, C. Fu and L. Xu, “Exploiting chaos-based compressed sensing and cryptographic algorithm for image encryption and compression,” Optics & Laser Technology, vol. 99, pp. 238–248, 2018. [Google Scholar]
6. A. M. Johnson, “Speed of mental rotation as a function of problem-solving strategies,” Perceptual and Motor Skills, vol. 71, no. 3, pp. 803–806, 1990. [Google Scholar]
7. J. Xue, C. Li, C. Quan, Y. Lu, J. Yue et al., “Uncovering the cognitive processes underlying mental rotation: An eye-movement study,” Scientific Reports, vol. 7, no. 1, no. 10076, pp. 1–12, 2017. [Google Scholar]
8. R. N. Shepard and J. Metzler, “Mental rotation of three-dimensional objects,” Science, vol. 171, no. 3972, pp. 701–703, 1971. [Google Scholar]
9. S. G. Vandenberg and A. R. Kuse, “Mental rotations, a group test of three-dimensional spatial visualization,” Perceptual and Motor Skills, vol. 47, no. 2, pp. 599–604, 1978. [Google Scholar]
10. I. Gauthier, W. G. Hayward, M. J. Tarr, A. W. Anderson, P. Skudlarski et al., “BOLD activity during mental rotation and viewpoint-dependent object recognition,” Neuron, vol. 34, no. 1, pp. 161–171, 2002. [Google Scholar]
11. J. M. Zacks, “Neuroimaging studies of mental rotation: A meta-analysis and review,” Journal of Cognitive Neuroscience, vol. 20, no. 1, pp. 1–19, 2008. [Google Scholar]
12. I. M. Harris, G. F. Egan, C. Sonkkila, H. J. Tochon-Danguy, G. Paxinos et al., “Selective right parietal lobe activation during mental rotation: A parametric PET study,” Brain, vol. 123, no. 1, pp. 65–73, 2000. [Google Scholar]
13. A. Gogos, M. Gavrilescu, S. Davison, K. Searle, J. Adams et al., “Greater superior than inferior parietal lobule activation with increasing rotation angle during mental rotation: An fMRI study,” Neuropsychologia, vol. 48, no. 2, pp. 529–535, 2010. [Google Scholar]
14. P. Khooshabeh, M. Hegarty and T. F. Shipley, “Individual differences in mental rotation: Piecemeal vs. holistic processing,” Experimental Psychology, vol. 60, no. 3, pp. 164–171, 2013. [Google Scholar]
15. M. A. Just and P. A. Carpenter, “The intensity dimension of thought: Pupillometric indices of sentence processing,” Canadian Journal of Experimental Psychology, vol. 47, no. 2, pp. 310–339, 1993. [Google Scholar]
16. M. A. Just and P. A. Carpenter, “Cognitive coordinate systems: Accounts of mental rotation and individual differences in spatial ability,” Psychological Review, vol. 92, no. 2, pp. 137–172, 1985. [Google Scholar]
17. R. N. Shepard and L. A. Cooper, “Mental Images and Their Transformations,” Cambridge, MA, USA: The MIT Press, 1986. [Google Scholar]
18. R. H. S. Carpenter, “Movements of the Eyes,” 2nd ed., London, UK: Pion Limited, 1988. [Google Scholar]
19. D. E. Irwin and L. A. Carlson-Radvansky, “Cognitive suppression during saccadic eye movements,” Psychological Science, vol. 7, no. 2, pp. 83–88, 1996. [Google Scholar]
20. D. E. Irwin and J. R. Brockmole, “Mental rotation is suppressed during saccadic eye movements,” Psychonomic Bulletin & Review, vol. 7, no. 4, pp. 654–661, 2000. [Google Scholar]
21. A. Larsen, “Deconstructing mental rotation,” Journal of Experimental Psychology: Human Perception and Performance, vol. 40, no. 3, pp. 1072–1091, 2014. [Google Scholar]
22. B. Bałaj, “The influence of object complexity and rotation angle on eye movements during mental rotation,” Roczniki Psychologiczne, vol. 18, no. 4, pp. 485–503, 2015. [Google Scholar]
23. I. D. Cherney, “Mom, let me play more computer games: They improve my mental rotation skills,” Sex Roles, vol. 59, no. 11–12, pp. 776–786, 2008. [Google Scholar]
24. A. C. Oei and M. D. Patterson, “Enhancing cognition with video games: A multiple game training study,” PLOS One, vol. 8, no. 3, pp. 1–16, 2013. [Google Scholar]
25. C. Meneghetti, E. Borella and F. Pazzaglia, “Mental rotation training: Transfer and maintenance effects on spatial abilities,” Psychological Research, vol. 80, no. 1, pp. 113–127, 2016. [Google Scholar]
26. G. Ganis and R. Kievit, “A New Set of three-dimensional shapes for investigating mental rotation processes: Validation data and stimulus Set,” Journal of Open Psychology Data, vol. 3, no. e3, pp. 1–7, 2015. [Google Scholar]
27. D. E. Irwin and L. E. Thomas, “The effect of saccades on number processing,” Perception & Psychophysics, vol. 69, no. 3, pp. 450–458, 2007. [Google Scholar]
28. R. von Wartburg, P. Wurtz, T. Pflugshaupt, T. Nyffeler, M. Luethi et al., “Size matters: Saccades during scene perception,” Perception, vol. 36, no. 3, pp. 355–365, 2007. [Google Scholar]
29. R. Dodge, “Visual perception during eye movement,” Psychological Review, vol. 7, no. 5, pp. 454–465, 1900. [Google Scholar]
30. F. C. Volkmann, “Human visual suppression,” Vision Research, vol. 26, no. 9, pp. 1401–1416, 1986. [Google Scholar]
31. N. Galley, “Saccadic eye movement velocity as an indicator of (de)activation: A review and some speculations. Journal of Psychophysiology, vol. 3, no. 3, pp. 229–244, 1989. [Google Scholar]
32. D. K. McGregor and J. A. Stern, “Time on task and blink effects on saccade duration,” Ergonomics, vol. 39, no. 4, pp. 649–660, 1996. [Google Scholar]
33. D. Schmidt, L. A. Abel, L. F. DellOsso and R. B. Daroff, “Saccadic velocity characteristics- intrinsic variability and fatigue,” Aviation, Space, and Environmental Medicine, vol. 50, no. 4, pp. 393–395, 1979. [Google Scholar]
34. M. Pomplun and S. Sunkara, “Pupil dilation as an indicator of cognitive workload in human-computer in-teraction,” in Proc. 10th Int. Conf. on HCI, Crete, Greece, pp. 542–546, 2003. [Google Scholar]
35. Y. Takano and M. Okubo, “Mental rotation,” in Encyclopedia of Cognitive Science, Tokyo: John Wiley & Sons, pp. 1–4, 2006. [Google Scholar]
36. W. G. Hayward, “Effects of outline shape in object recognition,” Journal of Experimental Psychology: Human Perception and Performance, vol. 24, no. 2 , pp. 427–440, 1998. [Google Scholar]
37. A. C. Smit, J. A. M. Van Gisbergen and A. R. Cools, “A parametric analysis of human saccades in different experimental paradigms,” Vision Research, vol. 27, no. 10, pp. 1745–1762, 1987. [Google Scholar]
38. T. Vuori, M. Olkkonen, M. Pölönen, A. Siren and J. Häkkinen, “Can eye movements be quantitatively applied to image quality studies?,” in Proc. of the Third Nordic Conf. on Human-Computer Interaction, Tampere, Finland, pp. 335–338, 2004. [Google Scholar]
39. P. E. G. Bestelmeyer, B. W. Tatler, L. H. Phillips, L,G Fraser, P. J. Benson et al., “Global visual scanning abnormalities in schizophrenia and bipolar disorder,” Schizophrenia Research, vol. 87, no. 1–3, pp. 212–222, 2006. [Google Scholar]
40. K. C. Moen, M. R. Beck, S. M., Saltzmann, T. M. Cowan, L. M. Burleigh et al., “Strengthening spatial reasoning: Elucidating the attentional and neural mechanisms associated with mental rotation skill development,” Cognitive Research: Principles and Implications, vol. 5, pp. 1–23, 2020. [Google Scholar]
41. D. Noton and L. Stark, “Scanpaths in saccadic eye movements while viewing and recognizing patterns,” Vision Research, vol. 11, no. 9, pp. 929–IN8, 1971. [Google Scholar]
42. L. L. Van Duren and A. F. Sanders, “Signal processing during and across saccades,” Acta Psychologica, vol. 89, no. 2, pp. 121–147, 1995. [Google Scholar]
43. E. Matin, “Saccadic suppression: A review and an analysis,” Psychological Bulletin, vol. 81, no. 12, pp. 899–917, 1974. [Google Scholar]
44. B. L. Zuber and L. Stark, “Saccadic suppression: Elevation of visual threshold associated with saccadic eye movements,” Experimental Neurology, vol. 16, no. 1, pp. 65–79, 1966. [Google Scholar]
45. F. W. Campbell and R. H. Wurtz, “Saccadic omission: Why we do not see a grey-out during a saccadic eye movement,” Vision Research, vol. 18, no. 10, pp. 1297–1303, 1978. [Google Scholar]
46. J. Ross, M. C. Morrone, M. E. Goldberg and D. C. Burr, “Changes in visual perception at the time of saccades,” Trends in Neurosciences, vol. 24, no. 2, pp. 113–121, 2001. [Google Scholar]
47. J. D. Schall, “Neural basis of saccade target selection,” Reviews in the Neurosciences, vol. 6, no. 1, pp. 63–85, 1995. [Google Scholar]
48. B. Alivisatos and M. Petrides, “Functional activation of the human brain during mental rotation,” Neuropsychologia, vol. 15, no. 2, pp. 111–118, 1997. [Google Scholar]
49. S. M. Kosslyn, G. J. DiGirolamo, W. L. Thompson and N. M. Alpert, “Mental rotation of objects vs. hands: Neural mechanisms revealed by positron emission tomography,” Psychophysiology, vol. 35, no. 2, pp. 151–161, 1998. [Google Scholar]
50. F. Peronnet and M. J. Farah, “Mental rotation: An event-related potential study with a validated mental rotation task,” Brain and Cognition, vol. 9, no. 2, pp. 279–288, 1989. [Google Scholar]
51. M. Wexler, “Anticipating the three-dimensional consequences of eye movements,” in Proc. National Academy of Sciences, vol. 102, no. 4, pp. 1246–1251, 2005. [Google Scholar]
52. M. Wexler and N. Ouarti, “Depth affects where we look,” Current Biology, vol. 18, pp. 1872–1876, 2008. [Google Scholar]
53. L. Jansen, S. Onat and P. König, “Influence of disparity on fixation and saccades in free viewing of natural scenes,” Journal of Vision, vol. 9, no. 1, pp. 29, 2009. [Google Scholar]
54. H. Kirchner and S. J. Thorpe, “Ultra-rapid object detection with saccadic eye movements: Visual processing speed revisited,” Vision Research, vol. 46, no. 11, pp. 1762–1776, 2006. [Google Scholar]
55. A. C. Neubauer, S. Bergner and M. Schatz, “Two-vs. three-dimensional presentation of mental rotation tasks: Sex differences and effects of training on performance and brain activation,” Intelligence, vol. 38, no. 5, pp. 529–539, 2010. [Google Scholar]
56. O. Chelnokova and B. Laeng, “Three-dimensional information in face recognition: An eye-tracking study,” Journal of Vision, vol. 11, no. 13, pp. 27, 2011. [Google Scholar]
57. G. K. Poock, “Information processing vs. pupil diameter,” Perceptual and Motor Skills, vol. 37, no. 3, pp. 1000–1002, 1973. [Google Scholar]
 | This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. |