Thumbs Up/Down Feedback with Admin Review & Prompt Tracking

[paullizer]

Overview

Introduce a user feedback system allowing users to rate AI-generated responses with a thumbs up/down. When a response is rated thumbs down, users can optionally provide a reason before submitting.

Feedback will be stored in a new Cosmos DB container called feedback, where admins can review, analyze, and take action on reported issues. Admins will have the ability to acknowledge, analyze, respond, and document actions taken for each piece of feedback.

Additionally, the system will track the user’s prompt so that an admin can test the same prompt and report back with improvements or findings.

---

Feature Scope

1. User Feedback on AI Responses

• Thumbs Up/Down UI in Chat:

  • Each AI response will have 👍 | 👎 buttons for quick feedback.
  • If thumbs down is selected, a textbox appears to capture the reason (optional for thumbs up).
  • Clicking Submit logs the feedback.

• Feedback Data Stored in Cosmos DB (feedback container):

  • Stores:
    • User ID
    • User Prompt
    • AI Response
    • Feedback Type (positive or negative)
    • Reason (if provided)
    • Timestamp

---

2. Admin Feedback Review Page

• A new page (not linked from base UI, only accessible via User Profile for Admins).

• Allows admins to view and manage submitted feedback.

• Data displayed per feedback entry:

  • User’s Prompt (to test AI response quality)
  • Thumbs Up/Down rating
  • AI Response given at the time
  • User-provided reason for bad feedback
  • Admin Review Fields
    • Acknowledge (true/false) – Has an admin seen this feedback?
    • Analysis Notes – What was the issue with the AI response?
    • Response to User – Optional response to communicate findings.
    • Action Taken – Any corrective action (e.g., model update, prompt tuning, filtering rules).

• Admins Can Retest the Prompt:

  • Allows manually running the prompt again to compare AI responses over time.
  • Provides a way to improve AI behavior based on user-reported issues.

---

New Cosmos DB Container: feedback

• Partition Key: /id
• Schema Example:

  {
    "id": "feedback-12345",
    "userId": "user-789",
    "prompt": "What is the capital of France?",
    "aiResponse": "The capital of France is Paris.",
    "feedbackType": "negative",
    "reason": "AI response lacked context",
    "timestamp": "2025-02-04T12:30:00Z",
    "adminReview": {
      "acknowledged": false,
      "analyzedBy": null,
      "analysisNotes": null,
      "responseToUser": null,
      "actionTaken": null,
      "reviewTimestamp": null
    }
  }

---

How This Helps

1. Improves AI Performance

• Identifies weak AI responses and provides data for model refinement.
• Allows admins to test and refine prompts for better AI interactions.

2. Increases Transparency & Trust

• Users see that their feedback is stored and reviewed, improving trust in AI systems.
• AI model updates can be communicated based on feedback trends.

3. Business & Compliance Benefits

• Helps detect biases or incorrect information in AI responses.
• Ensures AI meets business and ethical guidelines over time.

---

Implementation Considerations

Frontend Updates

• Chat UI: Add thumbs up/down + optional reason field for thumbs down.
• User Profile Page: Add "View My Feedback" to see user-submitted feedback.
• Admin Review Page: Add UI elements for acknowledging, analyzing, responding, and documenting actions.

Backend Updates

• API to Store Feedback: /feedback/submit (POST request).
• Admin Feedback Page API: /feedback/review (GET request for admins).
• Admin Update API: /feedback/review/{feedbackId} (PATCH request to update review fields).
• Prompt Retest API: /feedback/retest/{feedbackId} (Admin-triggered AI rerun of prompt).

Security & Privacy

• Feedback should be stored securely and linked to the user ID for visibility.
• Only admins should have access to the full feedback review page.

---

Deployment & Scaling Considerations

• Admins should be notified of new negative feedback for review.
• Implement filters to prioritize feedback based on severity, trends, or volume.
• Provide feedback analytics (e.g., most common issues, improvement trends).

---

This feature enhances AI response quality, transparency, and user engagement by making feedback actionable and trackable while ensuring admins can review and improve AI performance over time. 🚀