How Much Detail Should a Training Data Explanation Include?
A factorial study that cracks open the depth-versus-accessibility tradeoff, finding the design pattern that resolves it.
Overview
Every explanation designer faces the same dilemma: give users enough information to reason critically, and you risk overwhelming them. Keep it simple, and you leave them without the tools to scrutinize what matters.
This project tackles that tension directly. Using a 2×2 factorial design, I examined how information depth (summary vs. detailed) and disclosure style (static vs. progressive) interact to shape user perceptions, cognitive load, and the quality of critiques users produce about AI systems. The findings reveal a clear, actionable design pattern, explaining why one specific combination outperforms the others.
I designed and prototyped all four explanation conditions for the 2×2 study — a substantial design workload, since each condition had to be internally consistent while differing only on the intended variables. I piloted the materials with a separate group before the main study, recruited and ran all 32 participants, and conducted the statistical analysis including interaction effect testing.
The Problem
Rich, detailed explanations provide the information needed for informed evaluation. But they also impose cognitive demands that can impede the very reasoning they aim to support. Sparse summaries are easier to process but consistently leave users without the context needed to think critically or identify problems.
The Tradeoff
More information = better reasoning potential, but higher cognitive cost. The question is not which side to pick: it is whether a disclosure strategy can let users have both.
Research Questions
How does information depth (summary vs. detailed) affect user perceptions of the AI system and the quality of their critiques?
Does progressive disclosure reduce cognitive load associated with detailed explanations without sacrificing critique quality?
Are there interaction effects between information depth and disclosure style?
Research Process
Designing the 2×2
Choosing which two variables to cross — and how to operationalize each level — defined the study's ability to answer the question.
- Information depth addresses the content question: how much should an explanation say?
- Disclosure style addresses the delivery question: should all of it arrive at once, or be structured?
- They are independently manipulable — depth and delivery are orthogonal design decisions practitioners face separately
- A factorial design reveals not just main effects but whether the variables interact — which turned out to be the most important finding
- Progressive disclosure had to feel like natural pacing, not artificial gating — getting the trigger mechanism right took multiple pilot iterations
- The detailed condition had to be genuinely detailed, not just longer — content was carefully structured to add analytical depth, not repetition
- All four conditions used the same AI system and underlying data so that only format varied
- A pilot study with a separate group confirmed that each condition was experienced as intended before the main study
Study Design
Participants were randomly assigned to one of four conditions created by crossing two independent variables. All conditions used the same AI system (a real-estate recommendation tool) and the same underlying training data content.
Summary · Static
High-level overview shown all at once. Lowest information load, lowest critique depth.
Summary · Progressive
Same overview, revealed in steps. Added structure with minimal benefit at low depth.
Detailed · Static
Full information shown simultaneously. High load, gains partially offset by overload.
Detailed · Progressive ✓ Best
Full information revealed progressively. Highest critique quality, cognitive load comparable to summary conditions.
The 2×2 design. The highlighted cell (detailed information with progressive disclosure) consistently outperformed all alternatives.
After reviewing their assigned explanation, participants completed a NASA Task Load Index (NASA-TLX) to measure cognitive load, then wrote an open-ended critique of the AI system. Critiques were coded for specificity, accuracy, and the identification of data-related concerns.
Methods
2×2 Factorial Design
Between-subjects design crossing information depth and disclosure style to test main effects and interaction.
NASA-TLX
Validated cognitive load measure capturing mental demand, effort, and frustration across the four explanation conditions.
Open-Ended Critique Task
Participants wrote free-form critiques of the AI system. Critiques were coded for depth, specificity, and data issue identification.
Likert-Scale Questionnaires
Measured perceived usefulness, satisfaction with the explanation, and trust in the AI system after each condition.
It felt like it was letting me go at my own pace. I didn't feel rushed or dumped on. I could just keep going deeper when I wanted to.Participant in the Detailed · Progressive condition
Key Insights
Progressive disclosure is cognitive scaffolding
Progressive disclosure did not reduce information; it restructured how information arrived. Participants in the detailed-progressive condition processed significantly more content while reporting cognitive load levels comparable to the summary conditions.
Depth only pays off with the right delivery
Detailed-static explanations did not consistently outperform summary conditions on critique quality. The information was there, but the simultaneous delivery created overload that prevented users from fully engaging with it.
Ease of use does not mean adequacy
Users preferred summary explanations for how comfortable they felt. But that preference did not translate into better evaluation of the AI system. Perceived accessibility and actual reasoning quality are different things, and conflating them leads to under-designed transparency.
Interaction effects matter
Progressive disclosure had a significantly stronger benefit for detailed explanations than for summary ones. The mechanism is scaffolding: it only matters when there is enough complexity to scaffold. Designers should not apply progressive disclosure indiscriminately.
Design Implications
- Use detailed + progressive as the target pattern for critical evaluation tasks. When the goal is to support users in thoroughly evaluating an AI system, detailed information delivered progressively is the optimal design. It outperforms all alternatives on critique quality without the cognitive cost of static delivery.
- Reserve summaries for orientation, not evaluation. Summary explanations are appropriate when users need a quick overview before engaging more deeply, not as standalone explanations for high-stakes AI contexts where critical reasoning is needed.
- Do not optimize explanations for how comfortable they feel. User preference surveys can be misleading: participants consistently rate simpler explanations as better, even when richer ones produce superior outcomes. Explanation design should be evaluated against behavioral and reasoning measures, not just satisfaction.