qa-aman/interview-scorecard

Overview

Based on Who: The A Method for Hiring by Geoff Smart and Randy Street, and Work Rules! by Laszlo Bock. Unstructured interviews are nearly random at predicting performance. Structured scorecards work because every interviewer evaluates the same competencies using the same questions and the same rubric, making calibration possible and reducing bias.

Workflow

Step 1: Define Role Outcomes (from the JD or intake meeting)

A scorecard starts with outcomes, not skills. List 3-5 things the person must accomplish to be considered a success. These come from the job description or intake meeting with [hiring manager].

Format:

Role Outcomes:
1. [Specific outcome in first 90 days]
2. [Outcome at 6 months]
3. [Strategic outcome at 12 months]

Step 2: Identify 4-6 Competencies

Competencies are the behaviors and capabilities that predict whether someone will hit the outcomes. Derive them from the outcomes - don't start with a generic competency list.

For each outcome, ask: "What capability does someone need to achieve this?"

Common competency categories to consider:

• Technical / Functional: Domain knowledge, craft skills, tool fluency
• Problem-solving: Analytical rigor, structured thinking, ambiguity tolerance
• Execution: Ownership, delivery track record, bias for action
• Collaboration: Stakeholder management, communication, influence without authority
• Leadership: Team development, strategic thinking, driving alignment (for senior roles)

Cap at 6 competencies. More than 6 makes calibration impractical.

Step 3: Write 2-3 Structured Questions per Competency

Use behavioral questions (past behavior predicts future performance) following the STAR format: Situation, Task, Action, Result.

The Mom Test principle applied to interviewing: ask about specific past events, not hypotheticals. "Tell me about a time you..." beats "What would you do if..."

Question templates by competency type:

Execution / Ownership:

• "Tell me about the project you're most proud of shipping. Walk me through what you owned end-to-end."
• "Describe a time you missed a deadline. What caused it and what did you do?"

Problem-solving:

• "Walk me through the hardest technical problem you've solved in the last 12 months. How did you structure your approach?"
• "Tell me about a decision you made with incomplete information. What was your process?"

Collaboration / Influence:

• "Describe a time you had to convince someone senior to change direction. What was the situation and how did you approach it?"
• "Tell me about a conflict with a peer or cross-functional partner. How did you resolve it?"

Leadership (senior roles):

• "Tell me about someone you developed who went on to do something bigger than you expected. What did you do specifically?"
• "Describe a time you changed the strategy of a team or function. What triggered it?"

Step 4: Build the Scoring Rubric

Use a 3-point scale (A/B/C) to keep calibration fast and consistent. Avoid 5-point scales - the middle becomes a gravity well.

For each competency, define what A, B, and C look like:

Competency: [Name]

A (Strong Yes):
- Clear ownership of outcomes in past roles
- Specific, detailed examples with measurable results
- Demonstrated in multiple contexts
- Proactively identified and solved problems beyond their scope

B (Leaning Yes / Needs Discussion):
- Examples exist but are thin on detail or recency
- Achieved outcomes but with significant guidance
- One clear data point, not a pattern

C (No Hire):
- Examples are vague, hypothetical, or borrowed from team credit
- Avoids accountability ("we" language throughout)
- Cannot describe the impact of their work
- Repeated pattern of blaming external factors

Step 5: Assign Interviewers to Competencies

Each interviewer covers 1-2 competencies, not all of them. This prevents overlap, reduces candidate fatigue, and gives each interviewer deep coverage of their area.

Scorecard assignment format:

Interviewer 1 (Recruiter - phone screen): Culture fit, communication, career narrative
Interviewer 2 (Hiring Manager): Execution, ownership, alignment to outcomes
Interviewer 3 (Peer / IC): Technical / functional competency, collaboration
Interviewer 4 (Cross-functional): Stakeholder management, communication

Step 6: Define the Final Scorecard Template

Deliver a scorecard document interviewers fill in after each conversation:

# Interview Scorecard: [Role Title]
Candidate: _______________
Interviewer: _______________
Date: _______________

## Competencies Assessed

### [Competency 1]
Question(s) asked:
Evidence collected (verbatim notes preferred):
Rating: A / B / C
Notes:

### [Competency 2]
[Same format]

## Overall Recommendation
[ ] Strong Yes - A hire. Would advocate strongly.
[ ] Yes - Meets bar. Would not block.
[ ] No - Does not meet bar. Would block.
[ ] Strong No - Clear concerns. Would actively recommend against.

Key reason for recommendation (1-2 sentences):

Anti-Patterns

1. Using the same generic question bank for every role Bad: "Tell me about yourself. What's your greatest weakness? Where do you see yourself in 5 years?" Good: Questions tied to the specific competencies that predict success in this role and at this company stage.

2. Treating scorecard as a formality filled out after the debrief Bad: Interviewers compare notes verbally, then fill in the scorecard to match the group consensus. Good: Every interviewer submits their scorecard independently before the debrief begins.

3. Evaluating everything with one score Bad: "I'd give her a 7/10 overall." Good: Separate scores per competency. A candidate can be A on execution and C on leadership - that distinction matters.

4. Hypothetical questions Bad: "What would you do if a stakeholder pushed back on your recommendation?" Good: "Tell me about the last time a stakeholder pushed back on your recommendation. What happened?"

5. Allowing halo/horn effect Bad: One impressive answer early in the interview colors all subsequent assessments. Good: Structured questions force coverage of all competencies regardless of first impression.

Quality Checklist

• [ ] Scorecard has 4-6 competencies (not more)
• [ ] Each competency derives from a role outcome, not a generic list
• [ ] Every question is behavioral ("Tell me about a time...") not hypothetical
• [ ] A/B/C rubric is written out explicitly for each competency
• [ ] Interviewers are assigned specific competencies, not the full list
• [ ] Scorecards are submitted independently before the debrief
• [ ] Overall recommendation uses a 4-point scale (Strong Yes, Yes, No, Strong No)