ADu2021/feedback-friction-llm-response

Feedback Friction: LLMs Struggle to Fully Incorporate External Feedback

Core Concept

Despite access to high-quality feedback from more capable models with ground-truth knowledge, large language models consistently show resistance to incorporating guidance to improve responses. This "Feedback Friction" dominates 62.8-100% of unsolved problems even under ideal conditions (frontier models, extended thinking, multiple feedback iterations). The study reveals that model confidence (measured via semantic entropy) predicts feedback receptiveness, and that feedback resistance—not feedback quality—is the limiting factor.

Architecture Overview

• Systematic Feedback Framework: Three mechanisms of increasing sophistication (binary correctness, self-generated reflection, strong-model reflection) across up to 10 iterations
• Monotonic Accuracy Measurement: Models retain correct answers while only modifying incorrect responses, ensuring accurate tracking
• Semantic Entropy Analysis: Measures true confidence at meaning level rather than surface tokens; high correlation with feedback receptiveness
• Error Categorization: Three failure types identified; feedback resistance dominates across diverse benchmarks
• Tested Models: Llama-3.3-70B, Llama-4, Claude 3.7, Claude 3.7 Thinking on AIME, MATH-500, GPQA, MMLU variants

Implementation

Step 1: Feedback Framework and Mechanisms

import torch
from typing import Dict, List, Tuple

class FeedbackMechanism:
    """
    Three levels of feedback sophistication for guiding LLM improvements.
    Enables systematic study of feedback receptiveness.
    """

    def __init__(self, model, ground_truth_answers):
        self.model = model
        self.ground_truth = ground_truth_answers
        self.feedback_history = []

    def feedback_mechanism_1(self, question: str, model_response: str) -> str:
        """
        F₁: Simple binary correctness signal.
        Returns: "Your answer is correct" or "Your answer is incorrect"
        """

        expected_answer = self.ground_truth.get(question)

        if self._extract_answer(model_response) == expected_answer:
            return "Your answer is correct. Stop here."
        else:
            return "Your answer is incorrect. Please reconsider."

    def feedback_mechanism_2(self, question: str, model_response: str) -> str:
        """
        F₂: Self-generated reflective feedback.
        Model generates its own analysis of why the response might be wrong.
        """

        reflection_prompt = f"""
        Question: {question}
        Your response: {model_response}

        Analyze your reasoning step-by-step:
        1. What assumption did you make?
        2. Could there be an error in your calculation?
        3. Did you consider all constraints?

        Provide honest assessment of correctness.
        """

        # Model generates self-reflection
        self_reflection = self.model.generate(reflection_prompt, max_length=300)

        return f"Self-analysis: {self_reflection}. Please revise your answer."

    def feedback_mechanism_3(self, question: str, model_response: str) -> str:
        """
        F₃: Strong-model reflective feedback.
        GPT-4.1 mini provides expert analysis with access to ground-truth.
        """

        expected_answer = self.ground_truth.get(question)

        feedback_prompt = f"""
        Question: {question}
        Model's response: {model_response}
        Expected answer: {expected_answer}

        Provide guidance on what went wrong and how to improve:
        1. Identify the error
        2. Suggest correct approach
        3. Point out missed considerations
        """

        # Strong model provides feedback
        expert_feedback = self.model.generate(feedback_prompt, max_length=300)

        return expert_feedback

    def apply_iterative_feedback(self, question: str, max_iterations: int = 10) -> Dict:
        """
        Apply feedback iteratively up to max_iterations.
        Track whether model incorporates guidance over time.
        """

        current_response = self.model.generate(question, max_length=512)
        iteration_results = []

        for iteration in range(max_iterations):
            # Evaluate current response
            expected_answer = self.ground_truth.get(question)
            is_correct = self._extract_answer(current_response) == expected_answer

            iteration_results.append({
                'iteration': iteration,
                'is_correct': is_correct,
                'response': current_response,
                'confidence': self._measure_confidence(current_response)
            })

            if is_correct:
                # Correct answer found; stop iteration
                break

            # Select feedback mechanism
            feedback_level = min(iteration // 3, 2)  # Escalate feedback
            if feedback_level == 0:
                feedback = self.feedback_mechanism_1(question, current_response)
            elif feedback_level == 1:
                feedback = self.feedback_mechanism_2(question, current_response)
            else:
                feedback = self.feedback_mechanism_3(question, current_response)

            # Apply feedback and generate new response
            guided_prompt = f"""
            Question: {question}

            Previous response: {current_response}

            Feedback: {feedback}

            Based on this feedback, please provide a new, improved response:
            """

            current_response = self.model.generate(guided_prompt, max_length=512)
            self.feedback_history.append({
                'iteration': iteration,
                'feedback_level': feedback_level,
                'feedback': feedback
            })

        return {
            'question': question,
            'iterations': iteration_results,
            'final_correct': iteration_results[-1]['is_correct'] if iteration_results else False
        }

    def _extract_answer(self, response: str) -> str:
        """Extract final answer from response."""
        # Simplified: would use answer extraction based on task type
        lines = response.strip().split('\n')
        return lines[-1] if lines else ""

    def _measure_confidence(self, response: str) -> float:
        """Measure model confidence in response (0-1)."""
        # Placeholder: actual implementation uses semantic entropy
        return 0.5

Step 2: Semantic Entropy for Confidence Measurement

import numpy as np
from collections import Counter

class SemanticEntropyAnalyzer:
    """
    Measures true model confidence using semantic entropy.
    Operates at meaning level (multiple samples) rather than surface tokens.
    Predicts feedback receptiveness.
    """

    def __init__(self, model, num_samples=50):
        self.model = model
        self.num_samples = num_samples

    def compute_semantic_entropy(self, question: str) -> float:
        """
        Compute semantic entropy: uncertainty about meaning, not tokens.

        Procedure:
        1. Generate multiple samples from model
        2. Group semantically equivalent responses
        3. Compute entropy over meaning clusters
        """

        # Generate multiple samples
        samples = []
        for _ in range(self.num_samples):
            sample = self.model.generate(question, max_length=512, temperature=0.7)
            samples.append(sample)

        # Cluster semantically equivalent responses
        clusters = self._cluster_semantic_groups(samples)

        # Compute entropy over clusters
        cluster_sizes = [len(c) for c in clusters]
        probabilities = np.array(cluster_sizes) / len(samples)

        # Shannon entropy
        entropy = -np.sum(probabilities * np.log(probabilities + 1e-10))

        return entropy

    def _cluster_semantic_groups(self, samples: List[str]) -> List[List[str]]:
        """
        Cluster samples into semantically equivalent groups.
        Uses answer extraction + embedding similarity.
        """

        # Extract answers
        answers = [self._extract_answer(s) for s in samples]

        # Simple clustering: group identical answers
        answer_groups = {}

        for answer, sample in zip(answers, samples):
            if answer not in answer_groups:
                answer_groups[answer] = []
            answer_groups[answer].append(sample)

        return list(answer_groups.values())

    def predict_feedback_receptiveness(self, semantic_entropy: float) -> float:
        """
        Predict probability model will accept feedback based on entropy.

        Key finding: Absolute improvement rate increases from ~0 at low entropy
        (high confidence, resistant to feedback) to 0.4-0.8 at high entropy
        (low confidence, receptive to feedback).
        """

        # Empirically derived relationship
        if semantic_entropy < 0.5:
            receptiveness = 0.05  # High confidence: very resistant
        elif semantic_entropy < 1.0:
            receptiveness = 0.15
        elif semantic_entropy < 1.5:
            receptiveness = 0.4  # Medium confidence
        else:
            receptiveness = 0.7  # Low confidence: receptive

        return receptiveness

    def _extract_answer(self, response: str) -> str:
        """Extract answer for semantic clustering."""
        return response.strip().split('\n')[-1] if response else ""

Step 3: Error Analysis and Categorization

class FeedbackResistanceAnalyzer:
    """
    Categorizes persistent failures into three error types.
    Identifies whether failures are due to feedback quality or feedback resistance.
    """

    def __init__(self, model):
        self.model = model

    def categorize_persistent_failure(self, question: str, feedback_history: List,
                                     max_iterations: int = 10) -> Dict:
        """
        Analyze why a problem remains unsolved after feedback iterations.

        Three error categories:
        1. Feedback Quality Issue: Feedback is incorrect or unhelpful
        2. Comprehension Issue: Model understands feedback but can't apply it
        3. Feedback Resistance: Model ignores feedback despite understanding
        """

        categories = {
            'feedback_quality': 0.0,
            'comprehension': 0.0,
            'resistance': 0.0
        }

        # Test 1: Verify feedback correctness
        feedback_quality_score = self._verify_feedback_correctness(question, feedback_history)
        categories['feedback_quality'] = feedback_quality_score

        # Test 2: Model comprehension of feedback
        comprehension_score = self._test_comprehension(question, feedback_history[-1])
        categories['comprehension'] = comprehension_score

        # Test 3: Feedback resistance (residual)
        resistance_score = 1.0 - feedback_quality_score - comprehension_score
        categories['resistance'] = max(0, resistance_score)

        # Normalize
        total = sum(categories.values())
        if total > 0:
            for key in categories:
                categories[key] /= total

        return categories

    def _verify_feedback_correctness(self, question: str, feedback: List) -> float:
        """
        Verify that provided feedback is actually correct.
        Returns confidence that feedback accurately guides to correct answer.
        """

        verification_prompt = f"""
        Question: {question}

        Feedback provided:
        {feedback[-1] if feedback else "None"}

        Is this feedback correct and helpful? (0-1)
        """

        # Simplified: assume feedback from GPT-4 is correct
        return 0.95

    def _test_comprehension(self, question: str, feedback: str) -> float:
        """
        Test whether model actually understands the feedback.
        Ask model to explain what it should do based on feedback.
        """

        comprehension_prompt = f"""
        Question: {question}

        Feedback: {feedback}

        Based on this feedback, what is the correct approach? Explain:
        """

        explanation = self.model.generate(comprehension_prompt, max_length=300)

        # Score explanation against feedback content
        comprehension_score = self._score_explanation(explanation, feedback)

        return comprehension_score

    def _score_explanation(self, explanation: str, feedback: str) -> float:
        """Score how well explanation reflects feedback understanding."""
        # Placeholder: would use semantic similarity
        return 0.5

Step 4: Mitigation Strategies

class FeedbackFrictionMitigation:
    """
    Explores strategies to reduce feedback resistance.
    Tests: temperature scaling, rejection sampling, combined approaches.
    """

    def __init__(self, model):
        self.model = model

    def mitigate_with_temperature_scaling(self, question: str, feedback: str,
                                         temperatures: List[float]) -> Tuple[bool, float]:
        """
        Strategy 1: Progressive temperature increase encourages reconsideration.
        Higher temperature = more exploration, potentially better feedback incorporation.
        """

        best_response = None
        best_accuracy = 0.0

        for temp in temperatures:
            guided_prompt = f"""
            Question: {question}
            Feedback: {feedback}
            Please revise your answer:
            """

            response = self.model.generate(
                guided_prompt,
                temperature=temp,
                max_length=512
            )

            # Evaluate response
            is_correct = self._evaluate(response, question)

            if is_correct:
                return True, temp

            if self._response_quality(response) > best_accuracy:
                best_response = response
                best_accuracy = self._response_quality(response)

        return False, None  # Mitigation unsuccessful

    def mitigate_with_rejection_sampling(self, question: str, feedback: str,
                                        num_samples: int = 10) -> Tuple[bool, float]:
        """
        Strategy 2: Rejection sampling - generate multiple responses, select best.
        Combines temperature variation with explicit selection.
        """

        candidates = []

        for temp in [0.3, 0.7, 1.0, 1.3]:
            for _ in range(num_samples // 4):
                guided_prompt = f"""
                Question: {question}
                Feedback: {feedback}
                Please provide a revised answer:
                """

                response = self.model.generate(
                    guided_prompt,
                    temperature=temp,
                    max_length=512
                )

                candidates.append({
                    'response': response,
                    'correct': self._evaluate(response, question),
                    'quality': self._response_quality(response)
                })

        # Select best candidate
        best_candidate = max(candidates, key=lambda x: x['quality'])

        if best_candidate['correct']:
            return True, best_candidate['quality']

        return False, best_candidate['quality']

    def mitigate_combined_approach(self, question: str, feedback: str,
                                  max_attempts: int = 5) -> Tuple[bool, float]:
        """
        Strategy 3: Combined temperature scaling + rejection sampling.
        Escalates intervention based on failure analysis.
        """

        for attempt in range(max_attempts):
            # Attempt 1-2: Temperature scaling
            if attempt < 2:
                success, score = self.mitigate_with_temperature_scaling(
                    question, feedback, [0.5 + 0.3 * attempt]
                )
                if success:
                    return True, score

            # Attempt 3-4: Rejection sampling
            else:
                success, score = self.mitigate_with_rejection_sampling(
                    question, feedback, num_samples=20
                )
                if success:
                    return True, score

        return False, 0.0

    def _evaluate(self, response: str, question: str) -> bool:
        """Check if response is correct."""
        # Placeholder implementation
        return False

    def _response_quality(self, response: str) -> float:
        """Score response quality 0-1."""
        # Placeholder implementation
        return 0.5

Practical Guidance

Feedback Framework Implementation:

• F₁ (binary): Quick, no model required; use first
• F₂ (self-reflection): Model-generated; captures own reasoning limitations
• F₃ (strong-model): GPT-4 mini; most informative but costly
• Escalate feedback mechanism over iterations (start F₁, progress to F₃)

Confidence Measurement:

• Semantic entropy threshold: entropy < 0.5 indicates high confidence (resistant)
• Entropy > 1.5 indicates low confidence (receptive to feedback)
• Sample 50 outputs per question for robust entropy estimate

Error Attribution:

• Feedback resistance dominates: 62.8-100% of unsolved problems
• Feedback quality: Usually good (95%+ from GPT-4)
• Comprehension: Mixed; models understand but resist applying

Mitigation Effectiveness:

• Temperature scaling: Modest improvements; max +5% accuracy
• Rejection sampling: Better; +10-15% improvement
• Combined approach: Best; +15-20% improvement, but still falls short of theoretical ceiling

When Feedback Friction Occurs:

• Reasoning tasks (AIME, MATH, GPQA): High friction
• Factual tasks (MMLU): Lower friction
• Confident predictions: Higher friction
• Extended thinking models: Still shows significant friction

Recommendations:

• Accept that LLMs won't achieve theoretical maximum via iterative feedback
• Budget resources for rejection sampling (most effective practical mitigation)
• Use semantic entropy to identify which problems are worth improving
• Focus feedback on low-confidence responses where resistance is lower

Reference

• Semantic entropy: Meaning-level uncertainty vs token-level entropy
• Feedback friction: Resistance to incorporating guidance despite understanding
• Rejection sampling: Select best from multiple samples; often more effective than resampling with guidance
• Iterative refinement limits: Fundamental limitation in current LLM architecture for feedback incorporation