Prevent AI from 'seeing' hidden context in blind product tests with ChatGPT
Job to be done: Refining AI-assisted blind product testing to prevent context leakage
🇳🇬 Ways to use this in Nigeria
Ideas to get you started, adapt to your situation.
- Entrepreneur
Get unbiased feedback on a new product design by telling ChatGPT to act as a fresh user, ignoring any prior project context.
- Student
Test if your assignment prompt is clear by asking ChatGPT to review it as a new student, without using any prior knowledge of the course.
- Small business
Evaluate customer service responses by having ChatGPT act as a new customer, ensuring it doesn't use internal business jargon.
What you’ll get
You will learn how to conduct truly ‘blind’ product tests using ChatGPT, ensuring the AI only sees what a real user would. This prevents the AI from accidentally using hidden information (like project-specific ‘skills’ or tools) that could bias its feedback, leading to more accurate and unbiased reviews. This method helps you get unbiased feedback by isolating the AI from any background context or tools it might otherwise access.
Tools you need
- ChatGPT (freemium): An AI chatbot for generating text and understanding images, used here to simulate a product tester.
Steps
-
Open a new temporary chat: Go to ChatGPT and start a new chat. The author refers to this as a ‘Temporary Chat’, which is a good way to think about starting fresh without previous conversation history. You should see a blank chat window, ready for your first message.
-
Instruct ChatGPT not to use skills: Before sharing any product details, tell ChatGPT explicitly not to use any extra tools or context. This is the key step to ensure a truly blind test.
Do not use any skills, additional tools, or any other context that might provide information a typical user would not have. Act as a completely fresh, unbiased reviewer.ChatGPT will acknowledge your instruction, or simply wait for your next input without mentioning any skills.
-
Provide the product information: Now, share the product details you want ChatGPT to review. This could be text, images, or other media, just as a real user would encounter it. The author tested images. If you are testing images, upload them. If you are testing text, paste it directly. ChatGPT will process the information and be ready to respond based only on what you provided.
-
Ask for the review: Prompt ChatGPT to give its feedback or perform the review task you have in mind, based solely on the information you just provided. The author doesn’t share their exact prompt; a starting point:
Based only on the information I just provided, please give me a first-look review. What are your initial impressions?ChatGPT will provide its review, which should now be free from any hidden context or skill usage.
Original source
This workflow is based on an insight shared by Reddit user /u/Asteristix on the r/ChatGPT subreddit. They discovered that even ‘temporary’ chats could sometimes access hidden project-related ‘skills’ and shared a simple precaution to prevent this, ensuring truly blind AI testing.
Notes & variations
This workflow works well on the free tier of ChatGPT. Other freemium AI chatbots like Claude.ai or Gemini.google.com can also be used, but you would need to adapt the prompt to their specific features regarding ‘skills’ or ‘extensions’.
A common mistake is assuming a new chat is automatically ‘blind’. Always explicitly instruct the AI to avoid using skills or external context, especially when testing something where unbiased feedback is critical.
To get the best results, be very specific in your initial prompt about the role you want the AI to play (e.g., ‘Act as a new user seeing this for the first time’) and what kind of feedback you need.