Population definitions for comparative surveys in education

Abstract

This paper provides an overview of population definitions for large-scale comparative educational surveys. It has been prepared to help inform the development of a population definition and sampling framework that will be used in the British Council Global English research project. This paper examines a number of large-scale surveys including the Trends in International Mathematics and Science Study (TIMSS), which is conducted by the International Association for the Evaluation of Educational Achievement (IEA), as well as the Programme for International Student Assessment (PISA), and the Assessment of Higher Education Learning Outcomes (AHELO), both of which the Organisation for Economic Co-operation and Development (OECD) conduct. All of the surveys discussed in this paper are assessments of students. However, comparisons are not made between individual students’ results. Rather, data collected from students sampled to participate in the assessment are used to make inferences to a clearly defined population. By doing this, the results can be used to make comparisons between different populations. These comparisons can help identify factors such as teaching practices that may lead to better outcomes for a particular population compared to others. These comparisons can also help inform governments and policymakers about survey participants as well as more broadly about potential areas for improvement. The paper will examine how populations are defined in these large-scale international comparative educational surveys, examples of how some of these have evolved over time, and the implications of these definitions and evolutions on the interpretation of outcomes. It will also examine the implications of decisions about population definitions on the way in which the survey is conducted as well as the impact on data analysis. Finally the paper will provide some examples of how findings from these surveys are reported.