brycewang-stanford/survey-research-guide

Survey Research Guide

A practical skill for conducting rigorous survey research from instrument design through data analysis. Covers questionnaire construction, sampling strategies, administration methods, response bias mitigation, and analytical techniques commonly used in communication studies, anthropology, management, and sociology.

Survey Design Process

Phase 1: Conceptualization

Map your research questions to survey constructs:

def create_survey_blueprint(research_questions: list[dict]) -> dict:
    """
    Generate a survey blueprint mapping RQs to constructs and items.

    Args:
        research_questions: List of dicts with 'rq', 'constructs', 'hypothesized_relationship'
    """
    blueprint = {'sections': [], 'total_estimated_items': 0}

    for rq in research_questions:
        section_items = 0
        constructs = []
        for construct in rq['constructs']:
            n_items = construct.get('n_items', 4)  # default 4 items per construct
            constructs.append({
                'name': construct['name'],
                'type': construct.get('type', 'latent'),
                'scale': construct.get('scale', 'Likert 7-point'),
                'validated_instrument': construct.get('instrument', None),
                'items_needed': n_items
            })
            section_items += n_items

        blueprint['sections'].append({
            'research_question': rq['rq'],
            'constructs': constructs,
            'total_items': section_items
        })
        blueprint['total_estimated_items'] += section_items

    # Estimate completion time (3-4 items per minute)
    blueprint['estimated_minutes'] = round(blueprint['total_estimated_items'] / 3.5, 1)
    return blueprint

# Example
rqs = [
    {
        'rq': 'How does organizational culture affect employee innovation?',
        'constructs': [
            {'name': 'organizational_culture', 'instrument': 'OCAI (Cameron & Quinn)'},
            {'name': 'employee_innovation', 'instrument': 'Innovative Work Behavior Scale'}
        ],
        'hypothesized_relationship': 'positive'
    }
]
print(create_survey_blueprint(rqs))

Phase 2: Item Writing

Rules for writing effective survey items:

DO:
  - Use simple, unambiguous language (8th grade reading level)
  - Ask about one concept per item
  - Provide a reference period ("In the past 30 days...")
  - Include both positively and negatively worded items (reverse-coded)
  - Match response options to the question stem

DO NOT:
  - Use double negatives ("I do not disagree...")
  - Use absolutes ("always", "never")
  - Ask hypothetical questions when actual behavior data is available
  - Include two ideas in one question (double-barreled)
  - Assume knowledge or use jargon

Phase 3: Response Scale Design

| Scale Type | Use Case | Example | |-----------|----------|---------| | Likert (agreement) | Attitudes, beliefs | Strongly Disagree to Strongly Agree | | Frequency | Behavioral frequency | Never / Rarely / Sometimes / Often / Always | | Semantic differential | Perceptions | Cold ------- Warm | | Visual analog (VAS) | Continuous measurement | 0-100mm line | | Ranking | Relative preferences | Rank items 1 through N |

Survey Administration

Mode Selection

| Mode | Response Rate | Cost | Data Quality | Best For | |------|-------------|------|-------------|----------| | Online (Qualtrics/SurveyMonkey) | 10-30% | Low | Moderate | General population, students | | Telephone (CATI) | 15-40% | High | High | Older adults, nationally representative | | In-person (CAPI) | 50-70% | Very high | Highest | Sensitive topics, low-literacy populations | | Mail | 20-40% | Moderate | Moderate | Rural populations, older adults | | Mixed-mode | 30-60% | Moderate-high | High | Coverage optimization |

Response Bias Detection

def detect_response_patterns(responses: pd.DataFrame,
                              reverse_items: list[str]) -> dict:
    """
    Flag potential problematic response patterns.
    """
    flags = {}

    # 1. Straight-lining detection
    row_variance = responses.var(axis=1)
    flags['straight_liners'] = (row_variance < 0.1).sum()

    # 2. Speeding (if timing data available)
    if 'completion_seconds' in responses.columns:
        median_time = responses['completion_seconds'].median()
        flags['speeders'] = (responses['completion_seconds'] < median_time * 0.33).sum()

    # 3. Inconsistency (reverse-coded item pairs)
    if reverse_items:
        for rev_item in reverse_items:
            original = rev_item.replace('_R', '')
            if original in responses.columns and rev_item in responses.columns:
                max_scale = responses[original].max()
                expected = max_scale + 1 - responses[rev_item]
                diff = abs(responses[original] - expected)
                flags[f'inconsistent_{original}'] = (diff > 2).sum()

    # 4. Missing data pattern
    flags['pct_missing'] = responses.isnull().mean().mean() * 100

    return flags

Analysis Techniques

Structural Equation Modeling (SEM)

For testing hypothesized relationships between latent constructs:

# Using semopy for SEM in Python
# pip install semopy

model_spec = """
    # Measurement model
    org_culture =~ oc1 + oc2 + oc3 + oc4
    innovation =~ inn1 + inn2 + inn3 + inn4
    job_satisfaction =~ js1 + js2 + js3

    # Structural model
    innovation ~ org_culture + job_satisfaction
    job_satisfaction ~ org_culture
"""

# Fit indices to report:
# - Chi-square (p > 0.05)
# - CFI > 0.95
# - TLI > 0.95
# - RMSEA < 0.06
# - SRMR < 0.08

Report reliability (Cronbach's alpha, composite reliability), convergent validity (AVE > 0.50), and discriminant validity (Fornell-Larcker criterion) for all latent constructs.

Reporting Standards

Follow the AAPOR (American Association for Public Opinion Research) reporting guidelines: report response rate, sampling method, margin of error, field dates, mode of administration, and weighting procedures. For academic publication, include the full survey instrument as supplementary material.