brycewang-stanford/coding-guidelines

Research Project Coding Guidelines

Version: 1.0 Last Updated: January 2026 Purpose: Standardized coding practices for empirical research projects

---

Table of Contents

1. Project Structure
2. Python Guidelines
3. Stata Guidelines
4. General Best Practices
5. Quick Reference Templates

---

Project Structure

Directory Organization

ProjectName/
├── Code/                           # All analysis scripts
│   ├── [Number]_[Name].py         # Data processing (Python)
│   ├── AN_[Number]_[Name].do      # Analysis scripts (Stata)
│   ├── AN_[Number]_[Name].py      # Analysis scripts (Python)
│   ├── LogFiles/                  # Stata log files
│   └── README.md                  # Project documentation
├── Data/
│   ├── Raw/                       # Original data (never modify)
│   ├── Intermediate/              # Partial processing
│   └── Clean/                     # Analysis-ready data
└── Results/
    ├── Tables/                    # Regression tables
    └── Figures/                   # Visualizations

Script Numbering Convention

• 0: Initial data extraction
• 1a, 1b, 1c: Data cleaning and preparation
• 2a, 2b: Data merging and linking
• 3a, 3b: Feature extraction and engineering
• 4a, 4b: Final data preparation
• 5a, 5b: Descriptive analysis
• AN_1, AN_2: Formal analysis and regressions

Use letter suffixes (a, b, c) for parallel steps Use number suffixes (1, 2, 3) for sequential substeps

Tool Preferences by Task

| Task | Preferred Tool | Rationale | |------|---------------|-----------| | Figures/Visualizations | Python | Better control, publication-quality with matplotlib/seaborn | | Regression Tables | Stata | More efficient with outreg2/esttab, standard in economics | | Data Cleaning | Python | Better for large datasets, flexible transformations | | Panel Regressions | Stata | reghdfe package is gold standard |

---

Python Guidelines

1. Script Template

#!/usr/bin/env python3
"""
ScriptName.py

Brief description of what this script does.

Input files:
- Data/Raw/input1.csv
- Data/Intermediate/input2.csv

Output files:
- Data/Clean/output.csv (description)

Author: Your Name
Date: YYYY-MM-DD
"""

import pandas as pd
import numpy as np
from pathlib import Path
import matplotlib.pyplot as plt
import seaborn as sns

def get_project_root():
    """Automatically detect the project root directory."""
    return Path(__file__).parent.absolute()


def main():
    """Main processing function."""
    print("=" * 70)
    print("SCRIPT TITLE")
    print("=" * 70)

    # Setup paths
    base_dir = get_project_root()
    data_clean_dir = base_dir / ".." / "Data" / "Clean"

    # Your code here

    print("\n" + "=" * 70)
    print("PROCESSING COMPLETE")
    print("=" * 70)


if __name__ == "__main__":
    main()

2. Path Management (CRITICAL)

Always use this pattern for portability:

def get_project_root():
    """Automatically detect the project root directory."""
    return Path(__file__).parent.absolute()

# Then use relative paths
base_dir = get_project_root()
data_raw_dir = base_dir / ".." / "Data" / "Raw"
data_clean_dir = base_dir / ".." / "Data" / "Clean"
data_intermediate_dir = base_dir / ".." / "Data" / "Intermediate"
results_tables_dir = base_dir / ".." / "Results" / "Tables"
results_figures_dir = base_dir / ".." / "Results" / "Figures"

# Create directories if they don't exist
data_intermediate_dir.mkdir(parents=True, exist_ok=True)

3. Data Loading & Saving

# Loading with error handling
if not input_file.exists():
    raise FileNotFoundError(f"Input file not found: {input_file}")

try:
    df = pd.read_csv(input_file, low_memory=False)
    print(f"Loaded {len(df):,} records")
except Exception as e:
    print(f"Error reading file: {e}")
    return

# Saving with confirmation
df_sorted = df.sort_values(['company', 'year'])
df_sorted.to_csv(output_file, index=False)
print(f"Saved {len(df_sorted):,} records to: {output_file}")

4. Progress Reporting

Use consistent formatting for readability:

# Section headers
print("\n" + "=" * 70)
print("DATA PROCESSING")
print("=" * 70)

# Progress with comma formatting
print(f"\nLoaded {len(df):,} records")
print(f"  After filtering: {len(df_filtered):,} ({len(df_filtered)/len(df)*100:.1f}%)")

# Summary statistics
print("\n=== SUMMARY ===")
print(f"Total companies: {df['company'].nunique():,}")
print(f"Date range: {df['date'].min()} to {df['date'].max()}")
print(f"Match rate: {match_rate*100:.1f}%")

5. Function Documentation

def clean_company_name(name_str):
    """
    Clean and standardize company names for matching.

    Parameters:
    - name_str: Raw company name string

    Returns:
    - Cleaned company name (uppercase, no punctuation)
    """
    if pd.isna(name_str):
        return ""

    # Remove common suffixes
    name = str(name_str).upper()
    name = re.sub(r'\b(INC|CORP|LTD|LLC)\b', '', name)
    name = re.sub(r'[^\w\s]', '', name)  # Remove punctuation

    return name.strip()

6. Data Validation

# Check for required columns
required_cols = ['company', 'year', 'value']
missing_cols = [col for col in required_cols if col not in df.columns]
if missing_cols:
    raise ValueError(f"Missing required columns: {missing_cols}")

# Report data quality
print("\nData Quality Checks:")
print(f"  Missing values in key column: {df['key_col'].isna().sum():,}")
print(f"  Duplicate records: {df.duplicated().sum():,}")
print(f"  Unique companies: {df['company'].nunique():,}")

7. Merging Pattern

# Prepare keys
df1['merge_key'] = df1['company'].astype(str).str.strip().str.upper()
df2['merge_key'] = df2['company'].astype(str).str.strip().str.upper()

# Merge with reporting
print(f"\nMerging datasets:")
print(f"  Dataset 1: {len(df1):,} records")
print(f"  Dataset 2: {len(df2):,} records")

df_merged = df1.merge(df2, on='merge_key', how='inner', indicator=True)

print(f"  Merged: {len(df_merged):,} records")
print(f"  Match rate: {len(df_merged)/len(df1)*100:.1f}%")

# Check merge results
print("\nMerge indicator breakdown:")
print(df_merged['_merge'].value_counts())

8. Visualization Standards

# Setup (at top of script)
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style("whitegrid")
plt.rcParams['figure.figsize'] = (12, 7)

# Create publication-quality figures
fig, ax = plt.subplots(figsize=(14, 8))

ax.plot(x, y, marker='o', linewidth=2, markersize=8,
        color='#2E86AB', label='Series Name')

ax.set_xlabel('X-axis Label', fontsize=12, fontweight='bold')
ax.set_ylabel('Y-axis Label', fontsize=12, fontweight='bold')
ax.set_title('Figure Title', fontsize=14, fontweight='bold', pad=20)

# Format y-axis with commas
ax.yaxis.set_major_formatter(plt.FuncFormatter(lambda x, p: f'{int(x):,}'))

ax.legend(loc='best', frameon=True, fancybox=True, shadow=True)
ax.grid(True, alpha=0.3)

plt.tight_layout()
plt.savefig(output_path, dpi=300, bbox_inches='tight')
plt.close()

print(f"Saved figure to: {output_path}")

9. Variable Naming Conventions

| Type | Convention | Examples | |------|-----------|----------| | DataFrames | df_ prefix | df, df_filtered, df_merged, df_agg | | Paths | _dir or _file suffix | base_dir, input_file, output_path | | Functions | snake_case verbs | clean_data(), load_files(), calculate_returns() | | Variables | snake_case | company_name, year_founded, total_assets | | Constants | UPPER_CASE | START_YEAR, MIN_OBSERVATIONS |

---

Stata Guidelines

1. Script Template

/*
================================================================================
ScriptName.do

Description of the analysis performed in this script.

Inputs:
- ../Data/Clean/input_data.csv

Outputs:
- ../Results/Tables/Table1_MainResults.xls
- ../Code/LogFiles/ScriptName.log

Author: Your Name
Date: YYYY-MM-DD
================================================================================
*/

*** Set up paths
global repodir "/Users/zrsong/MIT Dropbox/Zirui Song/Research Projects/PROJECT_NAME"
global datadir "$repodir/Data"
global cleandir "$datadir/Clean"
global intdir "$datadir/Intermediate"
global tabdir "$repodir/Results/Tables"
global figdir "$repodir/Results/Figures"
global logdir "$repodir/Code/LogFiles"

*** Start log
log using "$logdir/ScriptName.log", text replace

/*==============================================================================
    Data Preparation
==============================================================================*/

import delimited "$cleandir/input_data.csv", clear

[Your code here]

*** Close log
log close

2. Global Path Setup (CRITICAL)

Always define these at the top:

global repodir "/Full/Path/To/Project"
global datadir "$repodir/Data"
global cleandir "$datadir/Clean"
global intdir "$datadir/Intermediate"
global rawdir "$datadir/Raw"
global tabdir "$repodir/Results/Tables"
global figdir "$repodir/Results/Figures"
global logdir "$repodir/Code/LogFiles"

Note: Update repodir for each user/computer

3. Regression Structure

/*==============================================================================
    Main Regressions - Table 1
==============================================================================*/

*** Define variable lists
local borr_controls "log_assets leverage tangibility profitability"
local loan_controls "log_amount maturity"
local all_controls "`borr_controls' `loan_controls'"

*** Column 1: No controls
reghdfe outcome treatment_var, ///
    absorb(industry year) ///
    vce(cluster firm_id)

outreg2 using "$tabdir/Table1_MainResults.xls", replace excel ///
    ctitle("(1) No Controls") label dec(3) ///
    addtext(Industry FE, YES, Year FE, YES) ///
    keep(treatment_var)

*** Column 2: With controls
reghdfe outcome treatment_var `all_controls', ///
    absorb(industry year) ///
    vce(cluster firm_id)

outreg2 using "$tabdir/Table1_MainResults.xls", append excel ///
    ctitle("(2) Full Controls") label dec(3) ///
    addtext(Industry FE, YES, Year FE, YES, Controls, YES) ///
    keep(treatment_var `all_controls')

4. Output Table Conventions

Two output workflows:

| Stage | Command | Output Format | Use Case | |-------|---------|---------------|----------| | Working/Exploratory | outreg2 | Excel (.xls) | Quick iteration, reviewing results | | Final Paper | esttab | LaTeX (.tex) | Publication-ready tables |

---

A. Working Tables: outreg2 with Excel

Use outreg2 with the excel option for exploratory analysis and quick iterations:

outreg2 using "$tabdir/TableName.xls", [replace/append] excel ///
    ctitle("Column Title") ///              // Column header
    label ///                                // Use variable labels
    dec(3) ///                              // 3 decimal places
    keep(vars_to_show) ///                  // Variables to display
    addtext(Industry FE, YES, ///           // Notes for fixed effects
            Year FE, YES, ///
            Controls, YES)

Working table naming:

• Table1_MainResults.xls
• Table2_Robustness.xls
• TableA1_DescriptiveStats.xls (appendix)

---

B. Final Paper Tables: esttab for LaTeX

Use esttab to generate publication-ready LaTeX tables:

*** Store regression results
eststo clear

eststo m1: reghdfe outcome treatment, absorb(industry year) vce(cluster firm_id)
eststo m2: reghdfe outcome treatment `controls', absorb(industry year) vce(cluster firm_id)
eststo m3: reghdfe outcome treatment `controls', absorb(firm_id year) vce(cluster firm_id)

*** Output LaTeX table
esttab m1 m2 m3 using "$tabdir/Table1_MainResults.tex", replace ///
    b(3) se(3) ///                          // 3 decimal places for coef and SE
    star(* 0.10 ** 0.05 *** 0.01) ///       // Significance stars
    label ///                                // Use variable labels
    booktabs ///                            // Professional table formatting
    nomtitles ///                           // No model titles (use column numbers)
    mgroups("Dependent Variable: Outcome", pattern(1 0 0) ///
            prefix(\multicolumn{@span}{c}{) suffix(}) span erepeat(\cmidrule(lr){@span})) ///
    keep(treatment `controls') ///           // Variables to display
    order(treatment `controls') ///          // Variable order
    stats(r2 N, fmt(3 0) labels("R-squared" "Observations")) ///
    indicate("Industry FE = *industry*" "Year FE = *year*" "Firm FE = *firm_id*") ///
    addnotes("Standard errors clustered by firm in parentheses." ///
             "* p<0.10, ** p<0.05, *** p<0.01")

Simplified esttab for quick LaTeX output:

esttab m1 m2 m3 using "$tabdir/Table1.tex", replace ///
    b(3) se(3) star(* 0.10 ** 0.05 *** 0.01) ///
    label booktabs ///
    keep(treatment) ///
    stats(r2 N, fmt(3 0) labels("R\$^2\$" "N")) ///
    addnotes("Clustered SEs in parentheses.")

Final table naming:

• Table1_MainResults.tex
• Table2_Robustness.tex
• TableA1_Appendix.tex

5. Fixed Effects Patterns

*** Firm and year fixed effects with clustering
reghdfe outcome treatment controls, ///
    absorb(firm_id year) ///
    vce(cluster firm_id)

*** Industry and year fixed effects
reghdfe outcome treatment controls, ///
    absorb(industry year) ///
    vce(cluster firm_id)

*** Fama-French 12 industry classification
sicff sic, ind(12) gen(ff12)
reghdfe outcome treatment controls, ///
    absorb(ff12 year) ///
    vce(cluster firm_id)

6. Subsample Analysis

/*==============================================================================
    Subsample Analysis - Large Firms
==============================================================================*/

preserve

*** Keep only observations meeting criteria
keep if total_assets > median_assets

*** Run regressions for subsample
reghdfe outcome treatment controls, ///
    absorb(industry year) ///
    vce(cluster firm_id)

outreg2 using "$tabdir/Table2_Subsamples.xls", append excel ///
    ctitle("Large Firms") label dec(3)

restore

7. Variable Generation

*** Generate dummy variables
gen high_leverage = (leverage > 0.3)
replace high_leverage = 0 if missing(high_leverage)
label variable high_leverage "Leverage > 30%"

*** Generate interaction terms
gen treat_x_post = treatment * post
label variable treat_x_post "Treatment × Post"

*** Generate time trends
gen year_trend = year - 2000
gen year_trend_sq = year_trend^2

8. Summary Statistics Table

/*==============================================================================
    Table 1: Descriptive Statistics
==============================================================================*/

*** Summary statistics
estpost tabstat outcome treatment control1 control2, ///
    statistics(count mean sd min p25 p50 p75 max) ///
    columns(statistics)

esttab using "$tabdir/Table1_Descriptives.csv", ///
    cells("count mean sd min p25 p50 p75 max") ///
    replace noobs nomtitle nonumber

*** Alternative: Use outreg2
outreg2 using "$tabdir/Table1_Descriptives.xls", replace sum(log) ///
    keep(outcome treatment control1 control2) ///
    eqkeep(N mean sd min max)

9. Section Headers

/*==============================================================================
    Section Title
==============================================================================*/

*** Subsection description
[Code for subsection]

*** Another subsection
[More code]

---

General Best Practices

1. File Naming Conventions

| Type | Convention | Examples | |------|-----------|----------| | Data cleaning | [N]_[Action][Dataset].py | 1a_CleanIPO.py, 2_MergeCompustat.py | | Analysis | AN_[N]_[Description].[ext] | AN_1_DescriptiveStats.py, AN_2_MainReg.do | | Data files | Descriptive names | comp_crspa_merged.csv, loan_panel_final.csv |

2. Workflow Checklist

Before running any script:

• [ ] Input files exist in specified locations
• [ ] Output directories are created
• [ ] Paths are correctly specified

After running any script:

• [ ] Check output file was created
• [ ] Verify record counts make sense
• [ ] Review summary statistics
• [ ] Check for unexpected missing values

3. Documentation Requirements

Every script must include:

• Docstring/header comment with purpose
• List of input files
• List of output files
• Date last modified

Every analysis must document:

• Sample selection criteria
• Variable construction
• Outlier treatment
• Missing data handling

4. Code Organization

Within a script, follow this order:

1. Imports/library loading
2. Path setup
3. Helper functions
4. Main processing function
5. Execution block (if __name__ == "__main__":)

5. Error Prevention

# Check before processing
assert df['key'].notna().all(), "Key column has missing values"
assert df['year'].between(1990, 2025).all(), "Year outside expected range"
assert not df.duplicated(subset=['key']).any(), "Duplicate keys found"

# Validate merge results
assert len(df_merged) > 0, "Merge produced no matches"
assert '_merge' in df_merged.columns, "Merge indicator missing"

6. Reproducibility

Always include:

• Random seeds when using randomization: np.random.seed(42)
• Package versions in README
• Data processing workflow diagram
• Clear execution order in README

Never:

• Hardcode absolute paths (except Stata global repodir)
• Modify raw data files
• Delete intermediate files until project complete
• Use undocumented manual data adjustments

7. Version Control

Git best practices:

• Commit after completing each script
• Use descriptive commit messages
• Don't commit large data files
• Include .gitignore for:

  __pycache__/
  *.pyc
  .DS_Store
  LogFiles/
  *.log
  .ipynb_checkpoints/
  

---

Quick Reference Templates

Python: Basic Data Cleaning

#!/usr/bin/env python3
"""Brief description."""

import pandas as pd
from pathlib import Path

def get_project_root():
    return Path(__file__).parent.absolute()

def main():
    # Paths
    base_dir = get_project_root()
    input_file = base_dir / ".." / "Data" / "Raw" / "input.csv"
    output_file = base_dir / ".." / "Data" / "Clean" / "output.csv"

    # Load
    df = pd.read_csv(input_file)
    print(f"Loaded {len(df):,} records")

    # Process
    df_clean = df.dropna(subset=['key_col'])
    df_clean = df_clean[df_clean['year'] >= 2000].copy()

    # Save
    df_clean.to_csv(output_file, index=False)
    print(f"Saved {len(df_clean):,} records")

if __name__ == "__main__":
    main()

Python: Data Merging

def merge_datasets(df1, df2, merge_key, how='inner'):
    """Merge with reporting."""
    print(f"\nMerging datasets:")
    print(f"  Left: {len(df1):,} records")
    print(f"  Right: {len(df2):,} records")

    merged = df1.merge(df2, on=merge_key, how=how, indicator=True)

    print(f"  Result: {len(merged):,} records")
    print(f"\nMerge breakdown:")
    print(merged['_merge'].value_counts())

    return merged

Stata: Standard Regression Table

*** Table X: Main Results

local controls "control1 control2 control3"

*** Column 1
reghdfe outcome treatment, absorb(fe1 fe2) vce(cluster id)
outreg2 using "$tabdir/TableX.xls", replace excel ///
    ctitle("(1)") label dec(3) ///
    addtext(FE1, YES, FE2, YES) keep(treatment)

*** Column 2
reghdfe outcome treatment `controls', absorb(fe1 fe2) vce(cluster id)
outreg2 using "$tabdir/TableX.xls", append excel ///
    ctitle("(2)") label dec(3) ///
    addtext(FE1, YES, FE2, YES, Controls, YES) keep(treatment `controls')

Stata: Summary Statistics

*** Generate summary statistics
estpost tabstat var1 var2 var3, ///
    statistics(count mean sd min max) columns(statistics)

esttab using "$tabdir/Summary.csv", ///
    cells("count mean sd min max") replace noobs

---

Common Pitfalls to Avoid

Python

1. SettingWithCopyWarning: Always use .copy() after filtering

   df_subset = df[df['year'] >= 2000].copy()  # Good
   df_subset = df[df['year'] >= 2000]         # Bad
   

2. Path issues: Use Path objects, not string concatenation

   file = base_dir / "Data" / "file.csv"  # Good
   file = base_dir + "/Data/file.csv"     # Bad
   

3. Memory issues: Use low_memory=False for large CSV files

   df = pd.read_csv(file, low_memory=False)
   

Stata

1. Path separators: Use forward slashes even on Windows

   global dir "C:/Users/Name/Project"  // Good
   global dir "C:\Users\Name\Project"  // Bad
   

2. Missing absorb(): Don't forget to include absorb when using reghdfe

   reghdfe y x, absorb(fe1 fe2) vce(cluster id)  // Good
   reghdfe y x, vce(cluster id)                   // Bad - will error
   

3. Log file conflicts: Always use replace option

   log using "$logdir/script.log", text replace  // Good
   log using "$logdir/script.log", text          // Bad - will error if exists
   

---

Checklist for New Projects

Initial Setup

• [ ] Create directory structure (Code/, Data/, Results/)
• [ ] Create subdirectories (Raw/, Intermediate/, Clean/, Tables/, Figures/)
• [ ] Initialize git repository
• [ ] Create README.md with project description
• [ ] Create .gitignore file
• [ ] Document data sources and access methods

For Each Script

• [ ] Include header docstring with inputs/outputs
• [ ] Use standardized path management
• [ ] Include progress reporting
• [ ] Validate input files exist
• [ ] Check output makes sense
• [ ] Document any manual decisions

Before Finalizing

• [ ] All scripts run without errors
• [ ] Results reproduce from raw data
• [ ] Tables and figures saved to Results/
• [ ] README documents full workflow
• [ ] Code is commented appropriately
• [ ] No hardcoded paths (except Stata repodir)