wenmin-wu/nlp-joint-span-answer-type-head

Joint Span + Answer Type Head

Overview

Open-domain QA requires both extracting an answer span and classifying the answer type (short answer, long answer, yes/no, or unanswerable). A joint model shares the transformer backbone and adds two heads: (1) a token-level Dense(2) split into start/end logits for span extraction, and (2) a Dense(N) on the CLS pooled output for answer type classification. Joint training improves both tasks through shared representations.

Quick Start

import torch
import torch.nn as nn
from transformers import AutoModel

class JointQAModel(nn.Module):
    def __init__(self, model_name, n_answer_types=5):
        super().__init__()
        self.backbone = AutoModel.from_pretrained(model_name)
        hidden = self.backbone.config.hidden_size
        self.span_head = nn.Linear(hidden, 2)       # start + end logits
        self.type_head = nn.Linear(hidden, n_answer_types)

    def forward(self, input_ids, attention_mask):
        outputs = self.backbone(input_ids, attention_mask=attention_mask)
        sequence_output = outputs.last_hidden_state  # (B, seq_len, H)
        pooled_output = outputs.last_hidden_state[:, 0]  # CLS token

        span_logits = self.span_head(sequence_output)  # (B, seq_len, 2)
        start_logits, end_logits = span_logits.split(1, dim=-1)
        start_logits = start_logits.squeeze(-1)  # (B, seq_len)
        end_logits = end_logits.squeeze(-1)

        type_logits = self.type_head(pooled_output)  # (B, n_types)
        return start_logits, end_logits, type_logits

Workflow

1. Pass input through shared transformer backbone
2. Apply Dense(2) to sequence output, split into start/end logits
3. Apply Dense(N) to CLS pooled output for answer type
4. Train with combined loss: span CE + type CE (weighted)
5. At inference, use type prediction to decide whether to emit a span

Key Decisions

• Answer types: Typically 5 classes — UNKNOWN, YES, NO, SHORT, LONG
• Loss weighting: Equal weights work; tune if one task dominates
• Pooled output: Use CLS token or mean pooling for type head
• Span constraints: Enforce start < end and max span length at decode time

References

• BERT Joint Baseline Notebook
• TensorFlow 2.0 - Bert YES/NO Answers