Novel Approach to the Simulator Sickness Questionnaire
Article information
Abstract
Background and Objectives
Virtual reality (VR) applications change the perception of reality, resulting in a feeling of being in a natural environment. The occurrence of cybersickness (CS) when using VR applications is a well-documented side effect, and the Simulator Sickness Questionnaire (SSQ) has been used to assess CS. Considering the speed of VR technology development, CS will likely become a frequently researched and discussed topics in the near future. Therefore, the aim of this study was to conduct a Turkish validity and reliability study and introduce the SSQ to Turkish medical literature.
Materials and Methods
A total of 160 healthy individuals (80 females and 80 males) aged >18 years (28.4±7.2 years) were included in our study. The SSQ was provided to the participants through Google Forms before and after the VR provocation experience and within the scope of the test and retest protocol.
Results
The reliability and internal consistency of the questionnaire were observed at a high level (Cronbach’s alpha=0.854, Spearman-Brown coefficient r=0.871). Factor analysis was performed and the questionnaire was divided into three subfactors, consistent with the original questionnaire. In the responses obtained from the participants before and after VR provocation, statistically significant differences were observed in 13 of the 16 items in the questionnaire that are related to VR provocation (p<0.05). Statistically, the differences in fatigue, dizziness, and vertigo were greater in females than in males (p<0.05).
Conclusions
The Turkish version of the SSQ is an effective tool for measuring the side effects in VR environments. The inclusion of the SSQ in the Turkish literature enables the inclusion of non-English-speaking participants in research, especially for disciplines that consider peripheral and central vestibular disorders.
Introduction
Virtual reality (VR) systems are computer-based applications that change the person’s perception of reality, resulting in a feeling of being in a natural environment. VR systems and simulators mainly use systems that allow retinal changes. These visual stimuli may result in sensory conflict during the person’s adaptation to the new reality. As a result of altered perception of reality and sensory dissonance, motion sickness (MS) symptoms are expected to occur during VR experiences. McCauley called this phenomenon cybersickness (CS), also known as VR sickness [1]. CS, like MS is characterized by symptoms such as nausea, vomiting, dizziness, and vertigo.
CS is a subjective condition, and there are no standardized diagnostic tests. Questionnaires have been utilized as predominantly subjective methods to investigate the impacts of CS on participants, their experiences, and the accompanying symptoms [2]. The most frequently preferred questionnaires in the diagnosis of CS are the Motion Sickness Susceptibility Questionnaire (MSSQ), the Fast Motion Sickness Questionnaire, the Pensacola Motion Sickness Questionnaire, and the Simulator Sickness Questionnaire (SSQ) [3-6].
The SSQ is the most preferred among these questionnaires in the academic literature [7]. SSQ was first introduced to the literature by Kennedy, et al. [6] in 1993 by proving its reliability and validity. Considering technological developments authors revised the questionnaire in 2009. SSQ is still the most preferred questionnaire by many disciplines, especially engineering and medical disciplines study on VR. However, it has limited use in the Turkish literature due to its lack of Turkish validity and reliability [8-10]. This study aims to adapt the SSQ into Turkish and to evaluate the validity and reliability of the Turkish version of the SSQ.
Materials and Methods
Ethical consideration
Ethical clearance for conducting the study was obtained at the meeting numbered 2021/24 of the Medical Research Ethics Committee of Acibadem Mehmet Ali Aydinlar University on 17.12.2021 (case no 2021-24/33). All participants provided written informed consent.
Measurement tool
The SSQ is a 16-item, 4-point Likert-type questionnaire in English that measures individuals’ susceptibility to CS at a symptomatic level. Since this study is a questionnaire adaptation and validity-reliability study, it was planned in two stages (Fig. 1).
In the first stage, the questionnaire was translated into Turkish by preserving all the features of the questionnaire. Expert opinions were obtained by consulting two experts in the field of audiology who are competent in Turkish and English to confirm the accuracy of the translation. In the second stage of the study, the test-retest technique was applied to determine the validity and reliability of the Turkish version of questionnaire. Firstly, the SSQ, adapted into Turkish, was uploaded to the Google Forms online form system for application to the target group. The participants answered the SSQ adapted to Turkish via Google Forms online. Then, the participants were given a VR application in a quiet room environment. Finally, the participants were asked to answer the SSQ adapted to Turkish again. For the validity and reliability analyses of the Turkish form, the study was conducted with 160 individuals, which is ten times (16×10) the number of safe items [11,12]. Participants were selected within the age range of 18 to 45. As inclusion criteria, participants must not have a diagnosis of peripheral or central vestibular disorders, and they must have no previous otological history. Individuals who did not meet these inclusion criteria were excluded from the study. For the test phase, the questionnaire was administered to the participants (Pre-VR SSQ). Then, Samsung Gear VR glasses (South Korea) were placed on the participant’s head, and a 160-second VR stimulus was presented during the Rilix VR Roller Coaster (Rilix LLC) simulation. In this experimental phase, CS was tried to be created. The questionnaire was reapplied to all participants immediately after the experiment for the retest administered (post-VR SSQ) (Fig. 1).
Statistical analysis
The data obtained from the two questionnaire administrations were compared regarding test-retest reliability. Reliability and validity studies of the Turkish form were conducted. Additionally, the correlation between the results obtained from the Turkish version and the original study was examined.
Descriptive statistics (mean, standard deviation, minimum, median, maximum) were used to define continuous variables. The Mann-Whitney U test was used to compare two independent variables that did not follow a normal distribution. While studying the reliability of the survey; Cronbach alpha coefficient (Cronbach’s alpha=0.854) was used to evaluate internal consistency, and Split half method Spearman Brown correlation coefficient (Spearman-Brown coefficient r=0.871) was used to evaluate inter-class consistency. For the validity of the questionnaire; explanatory factor analysis was performed. Before the factor analysis, Kaiser-Meyer-Olkin (KMO) criterion was used for sample adequacy and the Bartlett test was used to show the suitability of the data for multiple normal distribution. Structural equation modeling (SEM) was used for confirmatory factor analysis (CFA). The level of statistical significance was set at 0.05. Analyses were performed using SPSS version 24 (IBM Corp.).
Results
Sex equality was observed in the study, which was conducted with 160 participants, 80 (50%) of whom were male and 80 (50%) of whom were female. The average age of the participants was 28.4±7.2 years.
Test-retest comparisons
A statistically significant difference was found between pre and post-VR exposure in terms of the distribution of questions 1, 2, 4, 5, 6, 7, 7, 8, 8, 9, 10, 11, 12, 13, 14 (p<0.05) (Table 1).
There is a statistically significant difference (p<0.05) in questions 2, 12, and 14 post- and pre-VR exposure according to sex. The symptom magnitude was higher in females than in males.
There is a statistically significant, weak positive correlation between age and the questions asking the disorientation subfactors (Questions 12, 13, and 14) before and after VR exposure (p<0.05). Additionally, we found a statistically significant weak negative correlation between age and eye strain (question 4) when comparing pre-VR and post-VR exposure (p<0.05) (Table 1).
Validity and reliability study
Participants’ responses to the SSQ pre- and post-VR exposure were averaged for reliability analyses. It was observed that the internal consistency (Cronbach’s alpha=0.854) and interclass consistency (Spearman-Brown coefficient r=0.871) were high, and the Turkish version was a reliable measurement tool. Based on the KMO criterion (0.804), our study had a sufficient sample size.
In addition, since Bartlett’s Test was calculated at a 0.05 significance level, p<0.001, the population correlation matrix was not a unit matrix, and the questionnaire was suitable for factor analysis. In the factor analysis, principal component analysis was used to determine the structure of the factors. Accordingly, the questionnaire consists of 3 factors, and the 3rd factor explains 52.6% of the total variance (Table 2 and Fig. 2).
According to the explanatory factor analysis results, factor 1 was associated with disorientation, factor 2 exhibited clustering with oculomotor items, and factor 3 showed clustering with nausea-related items.
If items have high loadings on a single factor and low loadings on other factors, it is recommended that the difference between the two high loadings should be at least 10 [13]. Accordingly, it was concluded that questions 1 and 7 should be removed from the questionnaire (Tables 3 and 4).
Confirmatory factor analysis
SEM was used. In the model, rectangles indicate observed variables (questionnaire items), ovals indicate latent variables (sub-factor), and the letter e shows an error or unexplained variance (Fig. 3) [13].
Since the distribution of the questions fit the normal distribution, Maximum Likelihood (MLE) was used as the estimation method. The fit index values for this measurement model were found as the degrees of freedom (DF)=73, χ2=137,409, p<0.000, CMIN/DF=1.882, RMSEA=0, 074, NFI=0.820, CFI=0.880 and TLI=0.980. Accordingly, the model output predicted by the most likelihood estimation method is presented in the path diagram. The model was found to be statistically significant (p<0.05) (Fig. 3 and Table 5).
Since all of the goodness-of-fit criteria were met, the model obtained with CFA can be interpreted. The factors obtained as a result of this model were determined as follows.
As all the goodness-of-fit criteria were met, the CFA model can be interpreted. The resulting factors were determined as follows.
Comparison of pre-VR and post-VR SSQ subfactors
There is a statistically significant difference between pre-VR and post-VR exposure in terms of factor 1 and factor 2 distributions (p<0.05). Average post-VR SSQ scores are higher than pre-VR SSQ scores. Additionally, this difference is more significant in the female population than in males (p<0.05). Furthermore, there is a statistically significant, positive, and weak correlation between age and the difference of the 1st factor pre-VR and post-VR (p<0.05) (Table 6).
Scoring of simulator sickness susceptibility
The original and Turkish versions of SSQ are 16-item questionnaires. Therefore, the minimum susceptibility score is 0, and the maximum score is 48. Considering the reliability and validity of the results, items 1 and 7 should be removed from the questionnaire. On the other hand, the SSQ is the most preferred questionnaire in the literature. For this reason, instead of removing the items from the questionnaire, keeping them in the questionnaire but accepting the item scores as “0” would be a more accurate approach regarding the literature. In this case, the maximum score that can be obtained from the SSQ can be 42.
In our study, participants did not report any symptoms during the testing process (pre-VR) but reported significant symptoms in the retest phase after provocation (post-VR). Therefore, the participants’ retest responses were considered when calculating the scores obtained from the questionnaire. According to the distribution of the participant’s responses to the questionnaire, the percentage distribution was calculated based on the simulator sickness susceptibility and the total score they reported on the questionnaire. A total score of 4 points or less indicates that the individual is not disturbed, 5–10 points indicate moderate discomfort, and 11 points and above indicate that the individual reports severe susceptibility to the VR. Considering that the MS is a set of symptoms and that the symptoms exacerbate each other, it was concluded that participants who scored 11 points and above were susceptible. In this context, the susceptibility assessment is as follows: below 50%, no susceptibility, 50%–80%, moderate susceptibility, and over 80% susceptibility.
Discussion
Questionnaires are data collection tools carefully prepared to determine the degree of the feature to be measured in scientific studies. In addition to the objective evaluation of disorders, they also enable the subjective evaluation of the person’s subjective expressions.
The SSQ was first developed in the 1990s to measure the quality of military flight simulators and their impact on users [6]. The development and widespread use of virtual reality systems following simulators has led to the need for more frequent questioning of CS symptoms. The symptomatic feature of the questionnaire has enabled it to be used in different research areas to measure the equipment’s quality and evaluate its impact on individuals. As a result, the SSQ has been widely adopted in world literature and used in over ten thousand studies.
The limited number of studies using the SSQ questionnaire belong to computer and software engineering fields. The common feature of these studies is that the entire sample group included in the study is English-speaking. Therefore, there was no need to translate the questionnaire into Turkish in these studies [8-10]. Adapting the questionnaire to the Turkish language can only ensure its applicability to the general public and expand the research areas.
In audiology, the MSSQ is preferred in studies on vestibular disorders such as vestibular migraine, phobic postural vertigo, MS, and Mal de Debarquement Syndrome (MdDS) [14-17]. The SSQ is a vehicle experience-based questionnaire and does not allow for symptomatic analysis. Therefore, the Turkish adaptation of the SSQ can be used for different purposes, such as determining the severity of the disease and the effectiveness of rehabilitation in vestibular disorders.
Construct validity analysis was conducted in our study. As in the original questionnaire, the items were divided into three sub-factors. The items that did not cluster under any factor were one (general discomfort) and seven (sweating). There might be a deviation related to the sample group. Kennedy developed the questionnaire in 1993. Since construct validity was not used in questionnaire standardization then, there was no construct validity in the original study [17]. According to Kennedy’s factor analysis, the factor loadings of these two items were compatible with our study. Since the purpose of the study was completely different, some factors were analyzed in several sub-factors simultaneously, even if the factor loadings were low. Considering the symptomatic nature of these items and that Kennedy’s SSQ is a study with over ten thousand citations, it was decided that it would be more appropriate to include these items in the questionnaire.
There are 16 items in the questionnaire. The original study used a factor loading of 0.3 as a basis for the factor analysis result. The items were distributed under the oculomotor, nausea, and disorientation headings, with seven items each according to the factor loadings. Some items were included in multiple sub-factors because their factor loadings were above 0.3. Items 1, 5, 8, and 9 were evaluated in two separate subfactors. The total score of the questionnaire was formulated and calculated according to the factor analysis results. For these reasons, the factor load distribution, construct validity results, and total calculation of the questionnaire differ from the original questionnaire.
Kennedy, et al. [6] focused on the development of the SSQ but did not define specific cut-off values. Balk used specific SSQ score ranges to determine symptom severity. A framework categorizing symptoms as low, moderate and high was also presented [7]. In our study analyzed the change in individual susceptibility depending on provocation using the test-retest technique. Considering the exacerbation of the participants’ symptoms after the provocation, the sensitivity percentages were calculated based on their score on the questionnaire after the provocation. In this context, the cut-off values we applied in our study are consistent with Balk’s study. We believe that this new approach will contribute to the literature in terms of questioning CS predisposition and measuring the severity of symptoms.
The change in sensitivity before and after provocation was found to be higher in the female population, consistent with the literature. In the literature, MS is higher in the female population regardless of vehicle [15,18]. Therefore, there is a sexrelated female dominance in the subgroups of MS. This finding is consistent with the literature on other vestibular disorders with similar symptomatic features to MS, such as vestibular migraine, MdDS, phobic postural vertigo, and persistent perceptual postural dizziness [19].
Conclusion
This study aimed to adapt the SSQ to Turkish and evaluate the validity and reliability of the Turkish version of the SSQ. According to the results, the Turkish version of the questionnaire is reliable and valid. Since the original study did not have construct validity, construct validity was investigated. However, the individual susceptibility formula has been recalculated.
Limitation of study
The limitations of the study are we used only Samsung Gear VR glasses and the Rilix VR Roller Coaster simulation, but different VR headsets and simulations may produce different results. In future studies different VR glasses and simulations can be used and they have to compare.
Notes
Conflicts of Interest
The authors have no financial conflicts of interest.
Author Contributions
Conceptualization: all authors. Data curation: Emel Uğur, Asime Kurter, Çağla Aydın. Investigation: Emel Uğur, Asime Kurter. Methodology: Emel Uğur, Asime Kurter, Çağla Aydın. Project administration: Emel Uğur, Bahriye Özlem Konukseven. Supervision: Emel Uğur, Bahriye Özlem Konukseven. Validation: all authors. Visualization: Emel Uğur, Asime Kurter, Çağla Aydın. Writing—original draft: Emel Uğur, Asime Kurter, Çağla Aydın. Writing—review & editing: all authors. Approval of final Manuscript: all authors.
Funding Statement
None
Acknowledgements
We would like to express our sincere gratitude to Busra Uludag from Istanbul Aydin University Institute of Graduate Study - Health Sciences Audiology Department for technical support.