How I Fix Prompts for Flawless AI Responses Every Time

Rifx.Online
Natural Language Processing , Chatbots , Machine Learning
26 Dec, 2024

When ChatGPT first launched, prompt templates for nearly every industry and profession flooded the internet. You’ve probably come across posts like “The Best [N] ChatGPT Prompts for [Sector/Profession].”

These posts helped introduce many people to AI tools, making it easy to ask questions and get answers. But now, most of us have gotten the hang of that and want to push things further. Simple prompts lead to simple results, so we’ve started crafting more detailed and complex prompts.

However, this has brought its challenges: sometimes, the “AI” just doesn’t give us the response we’re expecting. Especially, when you have tasks that require specific formatting or have certain constraints, writing the prompts that work flawlessly can be difficult.

In this post, I’ll show you how to quickly identify problems in your prompts and improve them to get consistent and reproducible responses.

Problem Statement

To make sure we’re on the same page, let’s look at a real example where a Large Language Model (LLM) gives a sub-optimal response.

QUESTION:
The value "1" for the question "Which of the following best describes 
the nature of your impairment or health issue?" is not valid.

Please correct it to one or more of the following: 
['1 - Autism', '2 - Physical disability', 
'3 - Registered blind or visual impairments uncorrected by glasses', 
'4 - Deaf/sign language user'].

Provide your answer as JSON with a 'valid_value' key.
For multiple selections, use a comma-separated string.

Example: {"valid_value": "Option A, Option B"}
    
RESPONSE:
1 - Autism, 2 - Physical disability, 3 - Registered blind or visual impairments uncorrected by glasses, 4 - Deaf/sign language user

For context, imagine users are filling out a form without any validation, and your task is to correct their answers. In this case, you want to change a response like “1” to “1 — Autism.” However, the large language model struggles to make this fix. And just to clarify, I was using GPT-4o, OpenAI’s current flagship model, but still couldn’t get the right result.

Now, while this is a programming-related example, the solution I will share is not. You could easily face the same issue when working on a different use case, for example, writing a report in a specific format, but the LLM keeps missing a section.

The key takeaway here is that this is just an example — the solution applies to a wide variety of tasks.

First, Reread your prompt

I know it sounds simple, but trust me — I’m serious 😊. Complex prompts can be long, and sometimes the best solution is just to take a few minutes, clear your head and carefully read your prompt again.

When writing a lengthy prompt, you’ve likely made several changes along the way. In doing so, there’s a good chance you’ve unintentionally introduced inconsistencies or contradictions that confuse the AI model.

Sometimes, you might even be using a prompt you found online without fully adapting it to your specific task. A quick reread helps catch these issues and makes sure they are not the reason for the poor response from the LLM.

In our example, I think we can agree my prompt looks pretty solid 😅, so there’s nothing to fix here on that front.

The Prompt-Fixing Prompt

We’re going to use an LLM to help fix our prompt. It sounds simple, maybe you’ve even already thought about it. But if you jump off and just paste your problematic prompt into ChatGPT with the question, “What’s wrong with my prompt?”, you’re likely to end up with the same issue we’re trying to avoid: using a sub-optimal prompt to solve the problem. So let’s do better!

You are a prompt-debugging assistant. Your goal is to make sure that the 
input I provide leads to the expected output. 
Please help me identify and fix issues with my prompt.

Step-by-Step Debugging Process:
1. Interpretation
   - Analyze the current prompt and summarize what the model likely understands from the given instructions.
   
2. Diagnosis 
   - Identify any ambiguity, confusing wording, or structural issues in the prompt that might prevent the model from producing the desired output.
   
3. Suggested Fixes
   - Provide at least three concrete improvements to the prompt, ranging from minor tweaks to substantial revisions, that could make the instructions clearer.

4. Test Cases
   - Based on the revised prompt, suggest at least three sample inputs along with their expected outputs. This will allow me to test the effectiveness of the updated prompt.

5. Validation
   - If possible, describe how the revised prompt addresses specific issues from the initial version and ensures the expected result.

Current Prompt:
[Insert current prompt here]

Expected Output:
[Insert expected output here]

Now you can copy and paste!

But before that let me show you how this solves the not-so-obvious issue in our example and hopefully yours too.

The first thing the prompt does is try to understand how an AI model interprets your problematic prompt. If you think about it, this is exactly how we, as humans, will approach the problem — we reread the prompt to see what we might be missing. The prompt then pushes the LLM to identify any issues, such as ambiguity or unclear wording and offers suggestions with examples to improve it.

Finally, it provides Test Cases and Expected Results to guide further validation. This last step was a personal twist on the debugging prompt.

Ready to see the results?

Feel free to try the full prompt in any LLM you have access to and see the complete results for yourself.

But to explain briefly, the response from GPT-4o made me realize something important. My problematic prompt was asking the model to match the input value of “1” to a corresponding option from this list ‘1 — Autism’, ‘2 — Physical disability’, ‘3 — Registered blind or visual impairments uncorrected by glasses’, ‘4 — Deaf/sign language user’.

However, it wasn’t clear whether “1” should be directly mapped to “1 — Autism” or if there was some conditional logic in place to handle invalid inputs.

As a matter of fact, the way my prompt was worded suggested that “1” was an invalid input, even though “1 — Autism” was part of the list. This contradiction was confusing the model. Hence, I was getting an unexpected response. I was blown away because there was no way I was catching that myself.

Amplify the Prompt-Fixer

I know that felt like all of it — and technically it is — but I like to go the extra mile, and apparently, so do you. So let’s make it worth your while.

The Prompt-Fixing prompt as-is, is already a powerful tool in your arsenal. But what if we combined it with OpenAI’s new o1 models? Yes, I’m talking about the models known for their “critical thinking” capabilities.

Although my original problem didn’t require this, I decided to experiment with the prompt using the o1-preview model. The results were even more concise, precise, and actionable. If you get the chance, I highly recommend giving it a try.

And now, you know what I know!

But remember, when working with LLMs, always keep verifying and iterating. I like to call it the “LLM Debugging Loop.”

I’d like to keep an open conversation so try the Prompt Debugger out and leave your experiences and or challenges in the comments.

Thanks for reading and don’t be shy to drop me a word via LinkedIn or Twitter !