Research
Touchstone Institute regularly engages in research projects internally and with external academics and researchers. We are committed to a research-driven approach, which contributes to our continuous improvement and alignment with best practices. On this page, you will find recent research-related articles. If you have questions or are interested in collaborating with us, please send us an email at
celban@tsin.ca.
Sustaining an Occupation-Specific Language Assessment for the Canadian Healthcare Field
Since its implementation in 2004, the Canadian English Language Benchmark Assessment for Nurses (CELBAN) has been accepted as evidence of language ability for licensure of internationally educated nurses (IENs) in Canada. This article focuses on the complexities of sustaining an occupation-specific assessment over time. The authors reference the seminal work of Epp and Lewis, who developed the original CELBAN test forms and aligned the test results with the Canadian Language Benchmarks (CLB), and then go on to describe a research and development project that was carried out under the direction of Touchstone Institute and overseen by the Centre for Canadian Language Benchmarks (CCLB) to renew the test model and develop additional content. This is followed by a discussion of the maintenance strategies required to sustain a secure assessment within the evolving Canadian context.
.jpg.aspx "Picture9-(7).jpg")
Sustaining an Occupation-Specific Language Assessment for the Canadian Healthcare FieldExamining Rater Performance on the CELBAN Speaking: A Many Facets Rasch Measurement Analysis
Internationally educated nurses’ (IENs) English language proficiency is critical to professional licensure as communication is a key competency for safe practice. The Canadian English Language Benchmark Assessment for Nurses (CELBAN) is Canada’s only Canadian Language Benchmarks (CLB) referenced examination used in the context of healthcare regulation. This high-stakes assessment claims proof of proficiency for IENs seeking licensure in Canada and a measure of public safety for nursing regulators. Understanding the quality of rater performance when examination results are used for high-stakes decisions is crucial to maintaining speaking test quality as it involves judgement, and thus requires strong reliability evidence (Koizumi et al., 2017). This study examined rater performance on the CELBAN Speaking component using a Many-Facets Rasch Measurement (MFRM). Specifically, this study identified CELBAN rater reliability in terms of consistency and severity, rating bias, and use of rating scale. The study was based on a sample of 115 raters across eight test sites in Canada and results on 2698 examinations across four parallel versions. Findings demonstrated relatively high inter-rater reliability and intrarater reliability, and that CLB-based speaking descriptors (CLB 6-9) provided sufficient information for raters to discriminate examinees’ oral proficiency. There was no influence of test site or test version, offering validity evidence to support test use for high-stakes purposes. Grammar, among the eight speaking criteria, was identified as the most difficult criterion on the scale, and the one demonstrating most rater bias. This study highlights the value of MFRM analysis in rater performance research with implications for rater training. This study is one of the first research studies using MFRM with a CLB-referenced high-stakes assessment within the Canadian context.
.jpg.aspx "Picture9-(4).jpg")
Examining Rater Performance on the CELBAN Speaking: A Many Facets Rasch Measurement Analysis
Internationally Educated Nurses and the Canadian English Language Benchmark Assessment for Nurses: A Qualitative Test Validation Study of Test-Taker Accounts
This qualitative validation study examines sixteen Internationally Educated Nurses’ (IENs’) accounts of the Canadian English Language Benchmark Assessment for Nurses (CELBAN) at two testing centres (Toronto and Hamilton). This study adopts both focus groups and one on-one interviews to investigate the inferences drawn from the test, and its consequences. Focus groups and interviews were conducted using an adapted interview guide utilized in the TOEFL iBT investigation of test-taker accounts of construct representation and construct irrelevant variance (DeLuca et al., 2013). While construct representation describes the degree of authenticity in the presentation of Canadian English language nursing tasks, construct irrelevant variance refers to potential factors impacting the test-taking experience which might contribute to a score variance that was not reflective of test-taker knowledge of the testing constructs (Messick, 1989, 1991, 1996). In this study, test-taker accounts of construct representation and construct irrelevant variance constituted the data which were coded and analyzed abductively via the sensitizing concepts derived from DeLuca et al., and Cheng and DeLuca (2011) on examining test-takers’ experience and their contribution to validity. Seven themes emerged, answering four research questions: How do IENs characterize their test experience? How do IENs describe the assessment constructs? What, if any, sources of Construct Irrelevant Variance (CIV) do IENs describe? Do IENs feel the language tasks are authentic? Overall, participants reported positive experiences with the CELBAN, while identifying some possible sources of CIV. Given the CELBAN’s widespread use for high-stakes decisions (a component of nursing certification and licensure), further research of IEN-test-taker responses to construct representation and construct irrelevant variance will remain critical to our understanding of the role of language competency testing for IENs.
.jpg.aspx "Picture9-(5).jpg")
Internationally Educated Nurses and the Canadian English Language Benchmark Assessment for Nurses: A Qualitative Test Validation Study of Test-Taker AccountsCELBAN™: A 10-Year Retrospective
This article provides a 10-year review by the test developers of the Canadian English Language Benchmark Assessment for Nurses (CELBAN™). From 2004 to 2014, the development, implementation, national administration, and operations of CELBAN and CELBAN-related products and services were the responsibility of the test developers and team at the Canadian English Language Assessment Services (CELAS) Centre at Red River College, Winnipeg, Manitoba. The CELAS Centre team experienced both challenges and opportunities during this 10-year period. As CELBAN expands, and in light of its current profile as a high stakes language assessment tool, a time for reflection and review is warranted. This retrospective review of CELBAN provides an overview of its history, administration, operations, and growth, as well as challenges experienced and lessons learned by the CELAS Centre team. Further research and development ideas are also posited by the CELBAN test developers.
.jpg.aspx "Picture9-(6).jpg")
CELBAN™: A 10-Year Retrospective