Generalizability Theory Analysis of Local Curriculum Validation Instruments: Item Effects and Rater Consistency

Abstract

Validating assessment instruments for local curricula presents unique challenges due to context-specific content and reliance on expert judgment. This study applies Generalizability Theory (GT) to evaluate the reliability of a curriculum validation instrument designed for the Social Studies Local Plants program in Fakfak, Indonesia. A fully crossed GT design was used involving 26 items evaluated by 3 expert validators, yielding 78 observations. Each expert rated all items using a 4-point Likert scale. Variance components were estimated using a linear mixed-effects model with REML, implemented via the lme4 package in R. The analysis revealed that item variance accounted for 98.2% of total score variance (σ² = 291.97), indicating strong item discriminability. Rater variance (0.2%) and item × rater interaction (1.1%) were minimal, demonstrating high inter-rater consistency. The Generalizability Coefficient (G = 0.995) and Dependability Coefficient (Φ = 0.986) exceeded the thresholds for both relative and absolute decision-making. A D-study showed that high reliability (Φ ≥ 0.90) could be maintained with as few as 2 raters and 15 items. The instrument demonstrated excellent reliability and is suitable for evaluating local curriculum validity. Minimal rater-related variance suggests that future improvements should focus on item refinement rather than rater training. These findings support the broader use of GT in educational instrument validation, particularly in context-rich, expert-judged settings.