Overview

User feedback is the most direct signal of response quality. Collect thumbs up/down, ratings, and comments to understand what works and what doesn't.

Feedback Types

Binary Feedback (Thumbs Up/Down)

Simple, low-friction feedback:

// User clicks thumbs up
await obs.feedback({
  traceId: trace.id,
  type: 'binary',
  value: 'positive',  // or 'negative'
});

Rating Scale

More granular feedback:

// User rates 1-5 stars
await obs.feedback({
  traceId: trace.id,
  type: 'rating',
  value: 4,
  scale: { min: 1, max: 5 },
});

Text Feedback

Detailed comments:

// User provides comment
await obs.feedback({
  traceId: trace.id,
  type: 'text',
  value: 'The response was helpful but too long',
});

Multi-Dimension Feedback

Multiple aspects:

await obs.feedback({
  traceId: trace.id,
  type: 'multi',
  values: {
    helpful: true,
    accurate: true,
    clear: false,  // User found it unclear
  },
});

Collecting Feedback

Frontend Integration

Add feedback UI to your chat:

// React example
function ChatMessage({ message, traceId }) {
  const [feedback, setFeedback] = useState(null);
 
  const handleFeedback = async (value: 'positive' | 'negative') => {
    await fetch('/api/feedback', {
      method: 'POST',
      body: JSON.stringify({ traceId, value }),
    });
    setFeedback(value);
  };
 
  return (
    <div className="message">
      <p>{message}</p>
      <div className="feedback">
        <button
          onClick={() => handleFeedback('positive')}
          className={feedback === 'positive' ? 'active' : ''}
        >
          👍
        </button>
        <button
          onClick={() => handleFeedback('negative')}
          className={feedback === 'negative' ? 'active' : ''}
        >
          👎
        </button>
      </div>
    </div>
  );
}

API Endpoint

Handle feedback on your backend:

// pages/api/feedback.ts
import { getObservability } from '@transactional/observability';
 
export async function POST(req: Request) {
  const { traceId, value, comment } = await req.json();
  const obs = getObservability();
 
  await obs.feedback({
    traceId,
    type: 'binary',
    value,
    comment,
  });
 
  return Response.json({ success: true });
}

Passing Trace ID to Frontend

Include trace ID in your response:

// Backend: Generate response with trace ID
async function chat(message: string) {
  const trace = obs.trace({ name: 'chat' });
  const response = await generateResponse(message);
  await trace.end({ output: response });
 
  return {
    response,
    traceId: trace.id,  // Send to frontend for feedback
  };
}

Feedback UI Patterns

Inline Feedback

Show feedback options with each message:

[AI]: Here's how to reset your password...

[👍] [👎] Was this helpful?

Follow-up Questions

Ask for details on negative feedback:

[👎 clicked]

Sorry to hear that. What was the issue?
[ ] Not relevant
[ ] Incorrect information
[ ] Too long/short
[ ] Other: [___________]

Delayed Feedback

Ask after task completion:

[After conversation ends]

How was your experience?
⭐⭐⭐⭐⭐

[Optional] Tell us more: [___________]

Analyzing Feedback

Dashboard Metrics

View feedback in the dashboard:

Feedback Rate: % of responses with feedback
Positive Rate: % positive feedback
NPS Score: Net Promoter Score equivalent
Feedback by Category: Breakdown by reason

Filtering by Feedback

Find problematic responses:

Go to Traces
Filter by Feedback: Negative
Review low-rated responses
Identify patterns

Correlation Analysis

Understand what drives feedback:

Positive feedback correlates with:
- Shorter responses (< 500 tokens)
- Specific model (gpt-4o)
- Certain query types

Negative feedback correlates with:
- Long responses (> 2000 tokens)
- Missing context (RAG failures)
- Complex questions

Feedback-Driven Improvement

Identify Issues

Common negative feedback patterns:

Issue	Feedback Signal	Action
Too verbose	"Too long" comments	Adjust prompt for conciseness
Missing info	"Incomplete" tags	Improve context retrieval
Wrong tone	Tone-related comments	Update system prompt
Factual errors	"Incorrect" feedback	Add fact-checking

A/B Testing

Test improvements with feedback:

// Tag traces with experiment variant
const trace = obs.trace({
  name: 'chat',
  metadata: {
    experiment: 'prompt-v2',
    variant: Math.random() > 0.5 ? 'control' : 'treatment',
  },
});
 
// Compare feedback between variants
// Control: 65% positive
// Treatment: 78% positive

Feedback Loop

Continuous improvement process:

Collect feedback from users
Analyze patterns and issues
Hypothesize improvements
Implement changes
Measure impact on feedback
Iterate

Best Practices

1. Make Feedback Easy

Minimal clicks
Non-intrusive
Optional detailed feedback

2. Act on Feedback

Review negative feedback regularly
Close the loop with users when possible
Track improvement over time

3. Combine with Other Signals

// Don't rely solely on explicit feedback
const signals = {
  explicitFeedback: userRating,
  implicitSignals: {
    copied: didUserCopy,
    retried: didUserRetry,
    followUp: didUserAskFollowUp,
    timeSpent: readingTime,
  },
};

4. Watch for Bias

Happy users don't always give feedback
Angry users are more likely to rate
Sample may not represent all users

Implicit Feedback Signals

Track behavior, not just explicit ratings:

Positive Signals

User copies response
User shares response
Conversation ends successfully
User returns to use again

Negative Signals

User retries with same question
User rephrases immediately
User abandons conversation
User contacts support

Tracking Implicit Signals

// Track when user copies
window.addEventListener('copy', () => {
  if (lastMessageElement.contains(window.getSelection()?.anchorNode)) {
    obs.feedback({
      traceId: lastTraceId,
      type: 'implicit',
      signal: 'copied',
    });
  }
});

Next Steps

LLM as Judge - Automated evaluation
Custom Scores - Add your own metrics
Dashboard - View feedback analytics

User Feedback