wenmin-wu/nlp-dropout-disabled-inference

Dropout-Disabled Inference

Overview

model.eval() disables dropout at the module level, but some transformer implementations read dropout rates from the config during forward passes (e.g., custom attention implementations, flash attention paths). Explicitly setting all dropout config values to 0.0 before loading weights guarantees deterministic inference regardless of implementation details.

Quick Start

from transformers import AutoConfig, AutoModel

config = AutoConfig.from_pretrained('model_path', output_hidden_states=True)
config.hidden_dropout = 0.0
config.hidden_dropout_prob = 0.0
config.attention_dropout = 0.0
config.attention_probs_dropout_prob = 0.0
model = AutoModel.from_pretrained('model_path', config=config)
model.eval()

Workflow

1. Load the model config with AutoConfig.from_pretrained()
2. Set all dropout fields to 0.0 (different model families use different field names)
3. Load the model with the modified config
4. Call model.eval() as usual

Key Decisions

• Which fields: BERT uses hidden_dropout_prob and attention_probs_dropout_prob; RoBERTa/DeBERTa may add hidden_dropout, attention_dropout; check your model's config.json
• When needed: always safe to apply; critical when averaging predictions across checkpoints or when reproducibility is required
• Training vs inference: only zero dropout at inference — training needs dropout for regularization
• Alternative: some use model.config.update({"hidden_dropout_prob": 0}) post-load, but pre-load is cleaner

References

• LECR-stsb_roberta_base
• 0.459 | Single Model Inference w/ PostProcessing