Cognitive representations in raters' assessment of teacher portfolios

Marieke F. van der Schaaf; Karel M. Stokking; Nico Verloop

doi:10.1016/j.stueduc.2005.02.005

Cognitive representations in raters' assessment of teacher portfolios

Marieke F. van der Schaaf^*, Karel M. Stokking, Nico Verloop

^*Corresponding author for this work

Research output: Contribution to journal › Article › Academic › peer-review

18 Citations (Scopus)

Abstract

Portfolios are frequently used to assess teachers' competences. In portfolio assessment, the issue of rater reliability is a notorious problem. To improve the quality of assessments insight into raters' judgment processes is crucial. Using a mixed quantitative and qualitative approach we studied cognitive processes underlying raters' judgments and the reliability of these judgments. Six raters systematically assessed 18 portfolios. The interrater reliability of 12 portfolios was satisfactory. Variance analysis showed slight rater effects. We used the Correspondent Inference Theory (Jones & Davis, 1965) and the Associative Systems Theory (Carlston, 1992; 1994) to analyse judgment forms and retrospective verbal protocols. Raters' cognitive representations on the dimensions abstract-concrete and positive-negative were significantly related to the judgments given and to the reliability of these judgments.

Original language	English
Pages (from-to)	27-55
Number of pages	29
Journal	Studies in Educational Evaluation
Volume	31
Issue number	1
DOIs	https://doi.org/10.1016/j.stueduc.2005.02.005
Publication status	Published - 2005
Externally published	Yes

Access to Document

10.1016/j.stueduc.2005.02.005

Cite this

@article{84c65eb8f9c44b088b8fa7d01e9767d3,

title = "Cognitive representations in raters' assessment of teacher portfolios",

abstract = "Portfolios are frequently used to assess teachers' competences. In portfolio assessment, the issue of rater reliability is a notorious problem. To improve the quality of assessments insight into raters' judgment processes is crucial. Using a mixed quantitative and qualitative approach we studied cognitive processes underlying raters' judgments and the reliability of these judgments. Six raters systematically assessed 18 portfolios. The interrater reliability of 12 portfolios was satisfactory. Variance analysis showed slight rater effects. We used the Correspondent Inference Theory (Jones & Davis, 1965) and the Associative Systems Theory (Carlston, 1992; 1994) to analyse judgment forms and retrospective verbal protocols. Raters' cognitive representations on the dimensions abstract-concrete and positive-negative were significantly related to the judgments given and to the reliability of these judgments.",

author = "{van der Schaaf}, {Marieke F.} and Stokking, {Karel M.} and Nico Verloop",

year = "2005",

doi = "10.1016/j.stueduc.2005.02.005",

language = "English",

volume = "31",

pages = "27--55",

number = "1",

}

TY - JOUR

T1 - Cognitive representations in raters' assessment of teacher portfolios

AU - van der Schaaf, Marieke F.

AU - Stokking, Karel M.

AU - Verloop, Nico

PY - 2005

Y1 - 2005

N2 - Portfolios are frequently used to assess teachers' competences. In portfolio assessment, the issue of rater reliability is a notorious problem. To improve the quality of assessments insight into raters' judgment processes is crucial. Using a mixed quantitative and qualitative approach we studied cognitive processes underlying raters' judgments and the reliability of these judgments. Six raters systematically assessed 18 portfolios. The interrater reliability of 12 portfolios was satisfactory. Variance analysis showed slight rater effects. We used the Correspondent Inference Theory (Jones & Davis, 1965) and the Associative Systems Theory (Carlston, 1992; 1994) to analyse judgment forms and retrospective verbal protocols. Raters' cognitive representations on the dimensions abstract-concrete and positive-negative were significantly related to the judgments given and to the reliability of these judgments.

AB - Portfolios are frequently used to assess teachers' competences. In portfolio assessment, the issue of rater reliability is a notorious problem. To improve the quality of assessments insight into raters' judgment processes is crucial. Using a mixed quantitative and qualitative approach we studied cognitive processes underlying raters' judgments and the reliability of these judgments. Six raters systematically assessed 18 portfolios. The interrater reliability of 12 portfolios was satisfactory. Variance analysis showed slight rater effects. We used the Correspondent Inference Theory (Jones & Davis, 1965) and the Associative Systems Theory (Carlston, 1992; 1994) to analyse judgment forms and retrospective verbal protocols. Raters' cognitive representations on the dimensions abstract-concrete and positive-negative were significantly related to the judgments given and to the reliability of these judgments.

UR - http://www.scopus.com/inward/record.url?scp=19744362669&partnerID=8YFLogxK

U2 - 10.1016/j.stueduc.2005.02.005

DO - 10.1016/j.stueduc.2005.02.005

M3 - Article

AN - SCOPUS:19744362669

SN - 0191-491X

VL - 31

SP - 27

EP - 55

JO - Studies in Educational Evaluation

JF - Studies in Educational Evaluation

IS - 1

ER -

Cognitive representations in raters' assessment of teacher portfolios

Abstract

Access to Document

Other files and links

Fingerprint

Cite this