Research in Academics

In June 2022, IDAT began a to explore the accuracy of test results. The project was being conducted by Heidi Reid, CEO of IDAT.

The purpose of the project was to analyse how ‘accurate’ school staff deemed scores of the IDAT had been for students who took the test and had since commenced studies with the school. As international admissions can be a lengthy process, it was crucial to find students who had not only taken the test but had begun their studies with the receiving institutions.  

IDAT selected 40 students who had taken the IDAT at four schools. The students were from K-12, across all stages and genders.  The students were from different language and cultural backgrounds including Kuwaiti, Korean, Chinese, Japanese, Iranian, Iraqi, Australian, British, and Emirati. Schools provided varying amounts of student data. Some included nationality, while others did not. The students were from Stage 1-5 as defined below. 

• Stage 1:  Year 1-2 UK, K- Gr 1 (USA)
• Stage 2:  Year 3-4 UK, Gr 2-3 (USA)
• Stage 3:  Year 5-6 UK, Gr 4-5 (USA)
• Stage 4:  Year 7-8 UK, Gr 6-7 (USA)
• Stage 5:  Year 9-10 UK, Gr 8-9 (USA)

As school applications and visa processing for students often takes more than a year, the students were selected based on the following criteria:

1. The IDAT was used by the school to assess the student for admissions.
2. The student had commenced studies with the school and studied with the school for a minimum of 4 months.

Once students were selected, surveys were sent to each student’s respective teacher to assess how accurate they found the IDAT test in the following areas for the student selected:

• Listening
• Speaking
• Writing
• Grammar
• Vocabulary
• Reading
• English Overall
• Mathematics

The mixed methods survey first included a Likert scale survey where the teachers were asked to review the results of the IDAT to ascertain how ‘accurate’ they found the results according to the following scale:

• 0 – I did not use this score
• 1 – Completely accurate
• 2 – Very accurate but not exactly
• 3 – Not very accurate
• 4 – Not at all accurate

Following completion of the Likert scale survey, teachers were asked to include comments related to the accuracy of the IDAT scores for each respective student. This qualitative data was collected to help better understand and illustrate the quantitative results from the Likert scale survey.

A particular limitation of this research was the wording which indicated there may be “an inaccurate test,” should there have been discrepancies in what practicing teachers found and what students were able to do. This is misworded, as the discrepancies may easily be explained by the length of time, circumstances of test, or non-measurable external factors.  All IDAT tests are considered accurate on the test day as a snapshot of student performance at that moment in time on the IDAT. However, even with this limitation, there is some numerical data which can be presented and used for quality improvement purposes.

Likert Scale Survey Results

The Likert scale survey results were analysed descriptively, and results showed teachers deemed the IDAT results to be accurate reflection of student ability in each of the areas surveyed: Listening, Speaking, Writing, Grammar, Vocabulary, Reading, English Overall, and Mathematics.

Table 1 illustrates results specifically related to the accuracy of English sections. In the area of Vocabulary, two scores were rated as “completely inaccurate.” Further inquiry found that the students for whom these scores were found to be “completely inaccurate” were in Stage 4. Outside of this area, many of the scores deemed to be “not so accurate” were in Stage 1 (younger learners). 

Table 1 – Accuracy of English Sections of the IDAT

 

 

 

 

 

 

 

 

 

In comparing the how teachers rated the accuracy of the English and Mathematics sections of the IDAT, the Mathematics and English Overall scores had far less “completely accurate” scores than other individual sections. Results showed that both sections were found to be “very accurate,” closely followed by “completely accurate” (See Table 2).

The “not so accurate” for English was much higher on the “overall” than on the individual sections. All the “not so accurate” scores for mathematics were from one school that uses an American-based curriculum and has anecdotally commented previously on lower-than-expected maths test.

The not-so-accurate for overall English as compared to English sections individually will result in further investigation of the weighting of percentages for the overall score. 

Table 2 –Accuracy of English and Mathematics Sections of IDAT

 

 

 

 

 

 

 

 

 

Qualitative Responses

After completing the Likert scale survey, teachers were asked to include comments related to the accuracy of the IDAT scores to help IDAT better understand the quantitative results from the Likert scale survey. The comments section did not rely on the same assumptions as the Likert scale survey nor was it limited by wording, as the quantitative survey. Due to this broader scope, IDAT finds more value from the qualitative data provided and found the comments trended toward the test being accurate or difficult, but not generally too easy for the test takers.

The perceived accurate comments were very positive and reaffirming to the academic team at IDAT and included phrases such as:

“The comments and support offered from the IDAT were excellent. This student has improved so well in her English, largely in part to the diagnostics provided by the IDAT.”

“The comments were very accurate.  This student was ready for the classroom as described but struggles with effort in mathematics.”

In the grouping for perceived somewhat accurate, even though there were some areas that were not deemed entirely accurate, the respondents recognised accuracy in many other areas as included in these comments:

“This student had much higher English than the test indicated for some scores. However, we were able to provide support he needed for math.”

“The biggest inconsistency was in the reading, which indicated a very low level.  This student was consistent and need some support for reading.  All other comments were very helpful, and her vocabulary continued to need support for some time.”

Of the inaccuracies described, there was only one comment which indicated a student who tested higher than abilities, and other comments indicating discrepancies showed that the IDAT indicated lower abilities than ascertained by teachers. In the case of the student with lower abilities than the test, it was indicated that this student took the test twice, which is not advised by IDAT. 

The qualitative feedback from teachers also included language such as “support” and “improve(d)” to indicate that IDAT results and comments had been used for support for students. One respondent remarked, “The comments and support offered from the IDAT were excellent.” While another highlighted that “…IDAT scores were quite accurate, and we were able to provide support which saw the student improve well. And another noted a student’s improvement “largely in part to the diagnostics provided by the IDAT.”

Table 3 includes a comprehensive list of comments given by the respondents regarding the perceived accuracy of the results based upon teacher experience with the student. The comments have been grouped into perceived accurate, perceived mostly accurate and perceived inaccurate but explanations given. To protect both student and teacher confidentiality and anonymity, all student and school names have been removed.

Table 3 – Comprehensive Comments from Teachers

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Implications and Future Research

This results from this small-scale study have given IDAT a preliminary understanding of how teachers perceive the accuracy of test results for students taking the IDAT test. Specifically, results from the Likert scale survey point to the need for IDAT to review the mathematics testing for students coming from American-based curriculum studies to determine if it is an appropriate and fair test for students from this background. Regarding stages, IDAT will further investigate the content and method of questioning Vocabulary for Stages 1 and 4 to determine if there are improvements that can be made to these stages. The team at IDAT recognizes the need to review and consider altering the weighting of percentages for the ‘overall’ scores given. Based on the results from this survey, IDAT has begun coding all questions in the system and will soon be able to extract data to see exactly which questions all students are getting correct or incorrect. IDAT recognizes that to assess well there should be no extreme variations among test results and is working adjust the method of questioning to ensure accuracy. In addition, IDAT will review the Stage 1 test in its entirety to determine whether an online test is a suitable platform for young students to demonstrate skills and knowledge.

Feedback gleaned from the qualitative data is encouraging and suggests that teachers are appreciative of and rely on the comments from IDAT team members that are included along with test results. IDAT will continue to include comments and support for teachers regarding test results and staging. In addition, the comments have brought to light for IDAT that inaccuracies are not necessarily related to test questions and scores but often have more to do with situations beyond the control of IDAT and the school such as outside support systems, native language, and simply being a good test taker. It is important that IDAT continue to monitor these inaccuracies and make adjustments or offer additional support where possible.  

In 2023, IDAT has commissioned a concordance study to measure the English language levels indicated to The Council of Europe (CEFR). This will be conducted by the National ELT Accreditation Scheme Pty Ltd (NEAS) who have expertise in English and English language quality assurance processes and procedures. In addition, IDAT will seek to commission a concordance study for mathematics as compared to the National Assessment Program for Literacy and Numeracy (NAPLAN) levels or other international benchmarking standards in 2023.

As IDAT continues to grow and is used more widely, a large-scale study looking at cross-sections of students based on native language, age, and stage placement will further help IDAT to validate the accuracy of test results for both English and Mathematics.

IDAT would like to thank C. Amelia Davis, Ph.D., an Adjunct Professor at the University of Louisiana Shreveport, who agreed to review and assist presentation of data.

The UK National Curriculum, the US Common Core, the Australia National Curriculum and the Ontario Curriculum, all contribute to the expected outcomes assessed by the IDAT as no one curriculum can cover it all. For IDAT to fulfil its requirements in terms of aims and outcomes, including benchmark statements, graduate attributes, national qualifications, or professional accreditation needs, test developers mapped commonalities across these curricula to the IDAT outcomes in mathematics and English. The similarities across curricula were then used to guide the creation of an IDAT curriculum, which assesses students from around the globe, no matter where they have studied in the past.