brycewang-stanford/software-engineering-research

Software Engineering Research Guide

Navigate the landscape of software engineering research, including key subfields, methodologies, datasets, benchmarks, and top venues.

SE Research Subfields

| Subfield | Key Topics | Major Venues | |----------|-----------|-------------| | Software Testing | Test generation, fuzzing, mutation testing, flaky tests | ISSTA, ICST, ASE | | Program Analysis | Static analysis, abstract interpretation, symbolic execution | PLDI, POPL, OOPSLA | | Software Maintenance | Code refactoring, technical debt, code smells, evolution | ICSME, MSR, SANER | | SE for AI/ML | ML pipeline testing, data quality, model debugging | ICSE-SEIP, FSE | | AI for SE | Code generation, bug detection, program repair | ICSE, FSE, ASE | | Distributed Systems | Consensus, fault tolerance, scalability, microservices | SOSP, OSDI, EuroSys | | Cybersecurity | Vulnerability detection, malware analysis, privacy | IEEE S&P, CCS, USENIX Security | | HCI in SE | Developer tools, IDE usability, code comprehension | CHI, CSCW, VL/HCC | | Empirical SE | Mining repositories, developer surveys, controlled experiments | ESEM, MSR, TOSEM |

Research Methodologies in SE

Controlled Experiments

Testing a specific hypothesis with treatment and control groups:

Example: Does AI code completion improve developer productivity?

Design:
- Participants: 60 professional developers
- Treatment: IDE with AI code completion enabled
- Control: IDE with AI code completion disabled
- Task: Complete 5 programming tasks of varying difficulty
- Metrics: Task completion time, code correctness, lines of code
- Analysis: Mixed-effects linear model with participant as random effect

Threats to validity:
- Internal: Learning effect (counterbalance task order)
- External: Lab setting may not reflect real development
- Construct: "Productivity" operationalized as speed + correctness

Mining Software Repositories (MSR)

Analyzing data from version control, issue trackers, code review systems:

# Example: Analyze commit patterns using PyDriller
from pydriller import Repository

repo_url = "https://github.com/apache/kafka"

commit_data = []
for commit in Repository(repo_url, since=datetime(2023, 1, 1),
                          to=datetime(2023, 12, 31)).traverse_commits():
    commit_data.append({
        "hash": commit.hash[:8],
        "author": commit.author.name,
        "date": commit.committer_date,
        "files_changed": commit.files,
        "insertions": commit.insertions,
        "deletions": commit.deletions,
        "message": commit.msg[:100]
    })

df = pd.DataFrame(commit_data)
print(f"Total commits in 2023: {len(df)}")
print(f"Unique contributors: {df['author'].nunique()}")
print(f"Avg files per commit: {df['files_changed'].mean():.1f}")

Case Studies

In-depth investigation of a phenomenon in its real-world context:

Case Study Protocol (based on Yin, 2018):
1. Research questions: How do teams adopt microservices?
2. Unit of analysis: Development teams at 3 companies
3. Data sources:
   - Semi-structured interviews (8-12 per company)
   - Architecture documentation review
   - Commit history and deployment logs
   - Meeting observations
4. Analysis: Thematic analysis with cross-case comparison
5. Validity: Triangulation across data sources, member checking

Key Datasets and Benchmarks

Code Understanding and Generation

| Benchmark | Task | Languages | Size | |-----------|------|-----------|------| | HumanEval | Code generation from docstrings | Python | 164 problems | | MBPP | Code generation from descriptions | Python | 974 problems | | SWE-bench | Real-world GitHub issue resolution | Python | 2,294 instances | | CodeXGLUE | Multiple code tasks | 6 languages | Varies by task | | BigCloneBench | Clone detection | Java | 6M clone pairs | | Defects4J | Bug localization and repair | Java | 835 real bugs |

Software Engineering Process

| Dataset | Content | Use Cases | |---------|---------|-----------| | GHTorrent | GitHub event data (commits, issues, PRs) | MSR studies | | Software Heritage | Universal source code archive | Code evolution, provenance | | Stack Overflow Data Dump | Q&A posts, tags, votes | Developer knowledge, NLP | | CVE Database | Vulnerability records | Security research | | Chrome/Firefox Bug Trackers | Bug reports, patches | Bug triage, severity prediction |

Static Analysis Tools for Research

# Example: Using tree-sitter for AST-level code analysis
from tree_sitter import Language, Parser
import tree_sitter_python as tspython

PYTHON_LANGUAGE = Language(tspython.language())
parser = Parser(PYTHON_LANGUAGE)

source_code = b"""
def fibonacci(n):
    if n <= 1:
        return n
    return fibonacci(n-1) + fibonacci(n-2)
"""

tree = parser.parse(source_code)
root = tree.root_node

def count_nodes(node, node_type):
    """Count AST nodes of a given type."""
    count = 1 if node.type == node_type else 0
    for child in node.children:
        count += count_nodes(child, node_type)
    return count

print(f"Function definitions: {count_nodes(root, 'function_definition')}")
print(f"If statements: {count_nodes(root, 'if_statement')}")
print(f"Return statements: {count_nodes(root, 'return_statement')}")
print(f"Function calls: {count_nodes(root, 'call')}")

Code Metrics

# Common software metrics
metrics = {
    "Lines of Code (LOC)": "Total lines (including blanks and comments)",
    "Cyclomatic Complexity": "Number of independent paths (McCabe, 1976)",
    "Halstead Volume": "Based on operators and operands count",
    "Maintainability Index": "Composite of LOC, CC, and Halstead",
    "Coupling Between Objects": "Number of other classes referenced",
    "Depth of Inheritance": "Levels in class hierarchy",
    "Code Churn": "Lines added + modified + deleted per period",
    "Comment Density": "Ratio of comment lines to total lines"
}

# Calculate cyclomatic complexity using radon
# pip install radon
import subprocess
result = subprocess.run(
    ["radon", "cc", "my_module.py", "-s", "-j"],
    capture_output=True, text=True
)
print(result.stdout)

Top Venues and Impact

Tier-1 SE Venues

| Venue | Type | Acceptance Rate | Focus | |-------|------|-----------------|-------| | ICSE | Conference | ~22% | Broad SE | | FSE/ESEC | Conference | ~24% | Broad SE | | ASE | Conference | ~22% | Automated SE | | ISSTA | Conference | ~25% | Software testing | | MSR | Conference | ~30% | Mining repositories | | TOSEM | Journal | -- | Broad SE (ACM) | | TSE | Journal | -- | Broad SE (IEEE) | | EMSE | Journal | -- | Empirical SE (Springer) |

Systems and Security Venues

| Venue | Type | Focus | |-------|------|-------| | SOSP/OSDI | Conference | Operating systems, distributed systems | | EuroSys | Conference | Systems (Europe) | | NSDI | Conference | Networked systems design | | IEEE S&P (Oakland) | Conference | Security and privacy | | USENIX Security | Conference | Security | | CCS | Conference | Computer and communications security | | NDSS | Conference | Network and distributed systems security |

Research Tools Ecosystem

| Tool | Purpose | URL | |------|---------|-----| | PyDriller | Git repository mining (Python) | github.com/ishepard/pydriller | | Radon | Python code metrics | github.com/rubik/radon | | SonarQube | Multi-language static analysis | sonarqube.org | | Understand | Code analysis and metrics | scitools.com | | Joern | Code analysis platform (CPG) | joern.io | | CodeQL | Semantic code analysis | codeql.github.com | | tree-sitter | Incremental parsing library | tree-sitter.github.io |