skip to main content

Research Developments from ACER

Subscribe
Higher Education
{rd-image-caption}

Image © Shutterstock/GaudiLab

A more accurate measure of the off-campus student experience

ACER has helped to revise a key measure of learner engagement to more accurately survey the experiences of both on-campus and off-campus students. Darren Matthews reports.

The Commonwealth Department of Education and Training commissioned the Australian Council for Educational Research (ACER) to review the Learner Engagement Scale (LES), a key measure of student experience. It is administered as part of the Student Experience Survey, an annual survey of a sample of the 400 000 first and final year undergraduates studying at Australian Higher Education institutions – 80 per cent on-campus and around 8 per cent off-campus.

Over the years, LES results have consistently shown substantially lower engagement levels in off-campus students – far in excess of the variation between teaching modes that might be expected – giving rise to concern that the survey itself was biased in favour of on-campus students. The review, which reported in late 2017, made several recommendations, including:

  • The inclusion of five new items specifically targeting the off-campus student experience
  • The revision of existing LES items to make them more relevant to off-campus students
  • The development of a new scale to more accurately measure learner engagement.

These recommendations were reviewed by the Department and the revision of existing items were accepted are now in use for the 2018 survey. Other recommendations are still under consideration by the Department.

The review process

The literature review, stakeholder consultation and psychometric analysis concluded that the existing LES primarily measured interactions and was therefore prejudiced in favour of on-campus students. It also found that often time-poor off-campus students valued different aspects of engagement and tended to privilege the relationship with their teacher, for example, over connections with other students. A redeveloped scale was piloted to test whether it more accurately measured student engagement.

Analysis of the pilot survey results showed that it worked well for both on- and off-campus students; however, it elicited more positive responses from off-campus students than the original, proving that these students are indeed engaged with their learning when questioned about factors they consider important.

Options for further redevelopment

Taken individually, the new and redeveloped survey items performed strongly for both on- and off-campus students, but psychometric analysis revealed that the survey results emphasised teaching quality rather than learner engagement. This suggests that separate scales – one measuring interactions with other students and the other measuring engagement with teachers and course delivery – may be appropriate in future iterations of the tool. The recommendations for new items to be added to the scale is being considered and reviewed further by the Department because questions around whether they accurately measure student engagement still remain.

Read the full report. ■

RD

About the author

Darren Matthews is a Research Fellow in ACER’s Tertiary Education research program.

More [rd] articles by Darren Matthews

Related articles

Higher Education
University completions and equity | RD

University completions and equity

06 May 2015

University students from disadvantaged groups have a lower completion rate than their more advantaged peers, but most disadvantaged students do complete their degrees, research reveals.

Evaluation, Quality & Standards, Higher Education, Indigenous, Featured higher education

Education & Development
Teacher absenteeism in Indonesia | RD

Teacher absenteeism in Indonesia

16 March 2015

A comprehensive new study reveals that teacher absenteeism in Indonesia is declining, and provides evidence for policy makers focused on improving teaching and learning, as Phil McKenzie explains.

Evaluation, Quality & Standards, Survey, School, Education & Development

School
A new tool to monitor whole-school mental health promotion | RD

A new tool to monitor whole-school mental health promotion

16 July 2018

Research shows that the best student mental health outcomes happen when wellbeing programs are applied school-wide, yet there has been no simple, brief measure of implementation until now, reports ACER's Katherine Dix.

Student Engagement, Survey, School, Featured school