How Do We Measure AI Literacy: Developing a Scale That Captures Both Knowledge and Confidence?
Building and psychometrically validating a dual-format AI literacy scale, then discovering a reversal of the Dunning-Kruger effect: people who are most confident about AI tend to overestimate what they actually know.
How do we measure AI literacy in a way that captures both what people know and what they think they know?
Two-stage psychometric scale development and validation across two independent samples (N = 476) using EFA, IRT, and CFA.
The scale works: but the real discovery is a calibration reversal: people who rate their AI knowledge highest show the largest gap between confidence and actual knowledge.
Overview
Measuring AI literacy sounds straightforward. Ask people how well they understand AI, and report the answers. The problem is that this approach assumes people know what they know. This project tests that assumption and finds it fails in a surprising direction.
I developed and psychometrically validated a dual-format AI literacy scale that pairs 7-point Likert self-assessment items with objective multiple-choice knowledge questions. Validated across two independent samples (n = 288 and n = 188), the scale revealed a consistent and striking pattern: people who rate themselves as highly AI literate tend to overestimate their actual knowledge, while those who rate themselves as less knowledgeable tend to underestimate it. A reversal of the classic Dunning-Kruger effect.
Generated all items grounded in Long and Magerko's AI literacy framework. Refined through expert think-aloud sessions with three domain experts and a five-person pilot study before main data collection.
Deployed two independent validation surveys via Prolific, collecting samples of n = 288 and n = 188. Managed data quality checks and participant compensation across both studies.
Ran the full validation pipeline: EFA, IRT modeling (Graded Response Model and 3PL), CFA, convergent and discriminant validity, and measurement invariance analysis across both samples.
Designed and ran the analysis that uncovered the reverse Dunning-Kruger pattern, computing bias index scores across the combined N = 476 sample and characterizing the systematic bidirectional mismatch.
The Problem
Most AI literacy instruments rely exclusively on self-report: asking people how confident they feel about AI-related topics. The assumption is that self-assessed confidence tracks with actual knowledge. But confidence and competence are not the same thing, and treating them as equivalent produces systematically misleading data about who is and isn't AI literate.
The Measurement Gap
A person who scores high on a knowledge test may rate their own literacy as "moderate." A person who scores poorly may rate themselves as "quite knowledgeable." Single-format scales cannot see this gap, and the gap turns out to be systematic, not random.
Research Questions
Can a dual-format scale combining self-assessment and factual knowledge items achieve strong psychometric properties across independent samples?
How does self-reported AI literacy relate to objectively measured knowledge, and what calibration patterns appear across the population?
Research Process
Designing the Scale
I grounded item development in Long and Magerko's AI literacy framework, which specifies 17 core abilities covering technical, ethical, and societal dimensions of AI understanding. The initial pool of 25 Likert items and 47 multiple-choice items was refined through expert think-aloud sessions with three domain experts, then through a five-person pilot study, arriving at 30 Likert and 35 multiple-choice candidates for the first validation study.
The Calibration Finding
After validating the scale's psychometric structure, I combined both samples (N = 476) to examine how self-assessed and factual understanding aligned. Computing calibration scores by subtracting standardized factual scores from standardized self-assessed scores, I found the pattern was not random but systematic and bidirectional, revealing a reversal of the classic Dunning-Kruger effect.
Scale Design
The scale captures AI literacy through two complementary formats, administered together. Items were grounded in Long and Magerko's framework, refined iteratively through expert review and IRT-based item reduction, and validated on an independent sample.
Two-component structure: Format A captures subjective confidence across two factors; Format B captures objective knowledge as a unidimensional construct.
Methods
Scale Development
Item generation grounded in Long and Magerko’s framework. Expert think-aloud sessions with 3 domain experts and a 5-person pilot study refined items before main data collection. Both samples recruited via Prolific.
Exploratory Factor Analysis + IRT
Applied to Sample 1 (n=288) using Principal Axis Factoring with oblique rotation. Graded Response Model IRT evaluated item discrimination. Parallel analysis, scree test, and MAP criteria determined factor count.
Confirmatory Factor Analysis
Applied to Sample 2 (n=188) to verify the two-factor Likert structure and evaluate convergent validity (CR, AVE) and discriminant validity (HTMT, Fornell-Larcker criterion).
Calibration Analysis
Computed bias index scores (z-scored self-assessment minus z-scored factual knowledge) across the combined sample (N=476) to examine alignment by self-assessed AI literacy level and prior AI training.
Key Insights
Strong psychometric properties across both samples
The Likert component demonstrated excellent internal consistency (Cronbach's alpha = 0.96 and 0.90 per factor) and a two-factor structure confirmed in both development and validation samples. The multiple-choice component showed acceptable reliability (alpha = 0.85 in validation) with good item discrimination across ability levels.
Self-assessed and factual understanding are only weakly correlated
Across the full combined sample (N=476), Spearman's correlation between self-assessed and factual scores was only rs = 0.17. Confidence and demonstrated knowledge are related but distinct constructs, and treating them as one produces misleading data.
A reversal of the Dunning-Kruger effect
Participants who self-assessed as highly AI literate tended to overestimate their actual knowledge (mean calibration score = +0.69). Those who self-assessed as less knowledgeable tended to underestimate theirs (mean = -0.73). This is a reversal of the classic pattern, where low performers typically overestimate.
More training predicts higher confidence, not higher knowledge
Formal AI training significantly increased self-assessed understanding across all measures, but factual knowledge scores did not differ significantly across training levels (p=.065). Those with formal training were the most overconfident group. Prior AI training is not a reliable proxy for actual AI literacy.
The Calibration Gap
Plotting self-assessed AI understanding against calibration score (self-assessed minus factual, both standardized) across the full sample reveals the reversal pattern clearly: higher confidence predicts greater overestimation, while lower confidence predicts underestimation.
Reverse Dunning-Kruger: higher self-assessed AI literacy predicts greater overestimation of actual knowledge. Self-assessment was the strongest predictor of calibration mismatch (b = 0.84, p<.001).
Design Implications
Core Insight
Any team that screens user AI literacy to recruit study participants, segment users, or target onboarding is likely getting distorted data if they use self-report alone. The most confident participants were the most miscalibrated. Using confidence as a proxy for knowledge produces systematic errors in both directions.
- Self-report AI literacy screening is unreliable. If your user research screener asks "how would you rate your AI knowledge?", high scorers are likely to overestimate and low scorers to underestimate. Segment by demonstrated knowledge, not reported confidence, when it matters.
- Calibration gap is a design target. Products that surface AI literacy feedback in onboarding flows, explainability features, and AI education tools should aim to close the gap between what users think they know and what they actually know, not just increase confidence scores.
- Formal AI training inflates confidence without proportionally improving knowledge. Users who have taken AI courses or training were the most overconfident group in this study. Onboarding that builds confidence without building accurate knowledge may be doing more harm than good.
Impact
For user researchers: any screener question that asks participants to self-rate their AI knowledge is producing a distorted sample. The most AI-confident participants in this study were also the most miscalibrated, overestimating their actual knowledge by the widest margin. This is not noise; it is a systematic pattern that replicates across two independent samples.
For product teams and AI educators: onboarding programs that build user confidence without improving actual knowledge may be actively widening the calibration gap. This validated dual-format scale provides a practical tool for measuring both dimensions, including identifying where the gap is largest. Manuscript under review.