curiositech/applied-cognitive-task-analysis-acta-met

SKILL: Applied Cognitive Task Analysis (ACTA) Methodology

Name: ACTA — Extracting and Applying Expert Cognitive Knowledge Description: Systematic methodology for eliciting, representing, and operationalizing expert cognitive skills—pattern recognition, situation assessment, mental simulation—from practitioners without requiring research training. Bridges academic rigor and practical usability.

DECISION POINTS

Expert Response Pattern → Interview Pivot Strategy

| Expert Response Type | Detection Signal | Next Probe Action | |---------------------|-----------------|-------------------| | Vague Generalization | "You develop a feel for it" / "Experience teaches you" | → Incident Grounding: "Walk me through the last time you..." | | Procedural Recitation | "First I do X, then Y, then Z..." | → Cognitive Dimension Probe: "What tells you it's time for the next step?" | | Abstract Principles | "Always prioritize safety" / "Focus on the customer" | → Contrastive Example: "Show me two cases—one where that applies, one where it doesn't" | | Contradicts Other Expert | "Never do X" (when Expert 2 said "Always X") | → Conditional Exploration: "Under what circumstances is X appropriate?" | | Describes Outcomes | "Then the system works better" / "Quality improves" | → Cue Detection: "What do you notice that tells you it's working?" |

Knowledge Audit Dimension Selection

IF expert mentions "seeing patterns" or "noticing things"
  THEN probe Perceptual Skills
    → "Show me two X's—one normal, one problematic. What differs?"

IF expert mentions "figuring out what happened" or "predicting outcomes"  
  THEN probe Mental Simulation
    → "What must have happened for..." / "If you did X, what would result?"

IF expert mentions "keeping track of multiple factors" or "big picture"
  THEN probe Situational Awareness
    → "What relationships do you monitor?" / "What trends matter?"

IF expert mentions "when standard procedures don't work"
  THEN probe Improvisation
    → "Tell me about a time the SOP failed" / "How do you adapt?"

IF expert mentions "knowing when you're wrong" or "second-guessing"
  THEN probe Metacognition
    → "How do you catch your own mistakes?" / "When do you seek help?"

Cognitive Demands Table Population Strategy

Difficult Element Identification:
  IF multiple experts struggle to articulate the same aspect
    THEN high cognitive demand
  IF novices consistently fail here despite training
    THEN expertise-dependent element
  IF standard procedures break down in this area
    THEN cognitive skill required

Why Difficult Analysis:
  IF novices miss perceptual cues → "Lacks pattern recognition for..."
  IF novices apply wrong strategy → "Cannot distinguish situation types..."  
  IF novices tunnel vision → "Focuses on single factor instead of..."
  IF novices freeze up → "No mental model for..."

Cues/Strategies Extraction:
  IF expert says "I just know" → probe for specific observable indicators
  IF expert gives multiple approaches → identify situational triggers
  IF expert mentions "experience" → extract pattern exemplars

FAILURE MODES

Rubber Stamp Elicitation

Detection: Expert interviews produce only procedural descriptions or textbook answers. No cognitive insights emerge. Symptoms: Transcripts read like SOPs. Experts say "just follow the process." No mention of judgment, adaptation, or situational factors. Root Cause: Abstract questioning triggers rationalized responses, not actual expert reasoning. Fix: Switch to incident-based probing. "Walk me through the last time you encountered..." Ground all questions in specific scenarios.

Single Expert Oracle

Detection: Design decisions based on one expert's approach treated as universal truth. Symptoms: "The expert says always do X" without conditional reasoning. No variation documented across experts. Root Cause: Treating individual expertise as domain expertise. Missing situational dependencies. Fix: Interview 3-5 experts. When approaches conflict, probe conditionals: "Under what circumstances would you do it differently?"

Transcript Worship

Detection: Raw interview notes treated as final deliverable. No transformation to actionable representation. Symptoms: Hours of interview audio with no structured output. Analysis stops at "we captured their knowledge." Root Cause: Confusing data collection with knowledge extraction. Fix: Force transformation through Cognitive Demands Table. Cannot complete without identifying difficult aspects, explaining why difficult, extracting cues/strategies.

Comprehensiveness Paralysis

Detection: Project stalled waiting for "complete" expertise capture before building anything. Symptoms: Endless additional interviews. "We need to understand everything first." No deployment timeline. Root Cause: Perfectionism prevents pragmatic deployment. Academic standards applied to practical problems. Fix: Extract sufficient expertise for initial capabilities. Deploy with monitoring and iteration cycles. 70% coverage enables valuable applications.

Proceduralization Fantasy

Detection: Attempts to convert expert judgment into decision trees or rule sets. Symptoms: Complex branching logic that doesn't capture actual expert reasoning. Rules that break in novel situations. Root Cause: Treating expertise as refined procedure rather than pattern recognition. Fix: Build pattern classification and situation assessment capabilities. Focus on "What kind of situation is this?" not "What's the prescribed action?"

WORKED EXAMPLES

Example 1: Emergency Nurse Triage

Context: Hospital needs to improve triage accuracy. Nurses with 2+ years experience significantly outperform new graduates in patient priority assignment.

Task Diagram Phase: Initial interview reveals standard triage procedure: vital signs → protocol lookup → priority assignment. But experts mention "something doesn't look right" decisions that novices miss.

Knowledge Audit Phase: Probe: "Show me two patients with similar vital signs—one you'd fast-track, one you wouldn't." Expert Response: "This one [points to case] has normal BP and pulse, but look at the skin color and how she's positioning herself. See how she's leaning forward? That's respiratory distress compensation." Follow-up: "What would a new nurse miss?" Expert Response: "They'd see normal vitals and send her to regular queue. They don't recognize the positioning pattern or skin color changes."

Simulation Interview: Present scenario: 45-year-old male, chest pain, normal vitals, says "feels like heartburn." Expert Response: "I'd ask about radiation to jaw or left arm. Also look at diaphoresis—sweating pattern. Men often minimize cardiac symptoms. Even with normal vitals, the description pattern raises MI risk."

Cognitive Demands Table Output: | Difficult Element | Why Difficult | Cues/Strategies | |------------------|---------------|-----------------| | Detecting compensated respiratory distress | Vitals appear normal due to physiological compensation | Forward-leaning posture, pursed lips, skin color changes, accessory muscle use | | Recognizing atypical cardiac presentations | Standard symptom descriptions don't match textbook | Male minimization patterns, "heartburn" descriptions with diaphoresis, jaw/arm radiation |

Agent Capability Mapping:

• Pattern recognition system trained on posture/positioning data
• Skin color analysis in combination with vital trends
• Natural language processing for symptom description patterns
• Risk stratification that weights behavioral cues alongside physiological measures

Trade-off Analysis: Visual pattern recognition requires camera systems vs. privacy concerns. Behavioral cue detection needs training data vs. patient consent. Expert pattern recognition operates on multi-modal inputs that are technically challenging but clinically critical.

Example 2: Manufacturing Safety Inspection

Context: Chemical plant needs to improve pre-startup safety checks. Experienced inspectors catch hazards that procedural checklists miss.

Task Diagram Phase: Standard inspection covers 47 checklist items across systems. But senior inspectors mention "intuitive" hazard detection that prevents incidents.

Knowledge Audit Phase: Probe: "Walk me through the last time you found something not on the checklist." Expert Response: "Pump was running within specs, pressure normal, but the vibration pattern felt different. Not louder—different frequency. Turned out bearing was starting to fail. Would've gone catastrophic during the run." Follow-up: "How do you know normal vibration patterns?" Expert Response: "After 15 years, you feel how each pump runs. This one usually has a smooth hum. That day it had a slight roughness underneath."

Simulation Interview: Present scenario: All checklist items pass, but unusual smell in Unit 3. Expert Response: "I'd trace the smell. Chemical odors shouldn't penetrate the building envelope. Either we have a leak that isn't registering on sensors, or ventilation system failure. Both are serious even if readings look normal."

Cognitive Demands Table Output: | Difficult Element | Why Difficult | Cues/Strategies | |------------------|---------------|-----------------| | Detecting equipment degradation before sensor alerts | Requires learned baselines for normal operation patterns | Vibration frequency changes, sound pattern shifts, subtle performance variations | | Identifying containment failures through sensory cues | Chemical detection systems have lag time and blind spots | Odor tracing, airflow pattern assessment, correlation with environmental conditions |

Agent Capability Mapping:

• Vibration analysis with learned baselines for each equipment piece
• Chemical sensor networks with pattern recognition for anomalous readings
• Environmental monitoring that correlates multiple sensory inputs
• Predictive maintenance algorithms that weight subtle degradation indicators

Trade-off Analysis: Sensor density vs. cost considerations. Learned baselines require operational history vs. new equipment deployment. Human sensory integration (smell, vibration, sound) difficult to replicate technically but critical for early hazard detection.

QUALITY GATES

Cognitive Demands Table Completion:

• [ ] Each "Difficult Element" appears in multiple expert interviews (not individual quirks)
• [ ] "Why Difficult" explanations are observable/testable (not circular reasoning like "requires experience")
• [ ] "Cues/Strategies" are specific enough for training design (not vague like "pattern recognition")
• [ ] Expert consensus reached on major cognitive demands (3+ experts validate core elements)
• [ ] Agent can classify novel scenarios using identified cues (testable discrimination)
• [ ] Novice failure modes mapped to specific cognitive gaps (not just "lack of training")

Interview Quality Assessment:

• [ ] Expert provided specific incidents, not just general principles
• [ ] Cognitive dimensions probed systematically (perception, simulation, awareness, etc.)
• [ ] Conflicting expert responses resolved through conditional exploration
• [ ] Context-dependent reasoning captured (when to use different strategies)
• [ ] Tacit knowledge made explicit through structured probes

Practical Usability Validation:

• [ ] Non-experts can use table for training design without attending interviews
• [ ] Agent specifications derivable from cue/strategy descriptions
• [ ] Failure modes predictable from "why difficult" analysis
• [ ] Knowledge portable across similar domains/applications

NOT-FOR BOUNDARIES

Do NOT use ACTA for:

• Simple procedural tasks: If experts just follow procedures faster, use process improvement methods instead
• Well-documented domains: If expertise is already captured in accessible form, focus on transfer/training optimization
• Individual skill coaching: For developing one person's expertise, use mentoring or deliberate practice approaches
• Academic research requirements: If you need publication-quality rigor, use full Cognitive Task Analysis methods
• Real-time performance support: For in-the-moment assistance, use job aids or decision support systems

Delegate instead to:

• Process mapping → for procedural optimization
• Training design specialists → for curriculum development from ACTA outputs
• Domain modeling → for comprehensive knowledge representation
• Ethnographic methods → for cultural/organizational expertise factors
• Psychometric assessment → for individual aptitude measurement

ACTA specifically addresses: Extracting cognitive expertise that enables superior pattern recognition, situation assessment, and adaptive problem-solving for training design and agent specification purposes.