I Ran the Same Prompt on 7 AI Models — The Results Will Surprise You

Running the same prompt across seven AI models produces noticeably different outputs because each model has different training data, safety rules, tool access, context handling, and generation settings. To compare fairly, keep the prompt and constraints locked, score accuracy and usefulness against trusted sources, then choose the model that matches your task, budget, and risk tolerance.

It is tempting to treat “AI” as one capability. In practice, large language models behave more like different teams of assistants. Each has its own defaults for tone, caution, reasoning depth, and willingness to cite sources. To make the differences concrete, I ran one identical prompt across seven popular chat models and scored them using a repeatable rubric.

This guide includes the exact prompt (copy and paste), the controls that make the test fair, a results table you can skim, and practical recommendations by task. If you want to compare models by features and pricing afterwards, you can use Compareai.today, plus the AI pricing calculator for cost modelling.

The Prompt: What I Asked All 7 AI Models (Copy/Paste)

Here is the exact prompt used in all seven chats. It tests reasoning, factual discipline, structure, and safety awareness, without requiring niche domain knowledge.

Prompt (Copy/Paste):

You are an AI assistant. Follow these rules:
1) Use British English.
2) Do not browse the web. If you are unsure about a fact, say so and suggest what to verify.
3) Output must be structured with headings and bullet points.
4) Prioritise correctness over confidence. Avoid making up statistics, dates, or citations.
Task: Explain why two AI chat models can give different answers to the same user prompt. Cover: (a) training and post training, (b) system prompts and safety rules, (c) tool access like search or code execution, (d) context window and memory, (e) randomness settings like temperature and top-p. Then provide a short checklist for how to compare models fairly, and a recommendation for which model is best for writing, coding, and research. Keep it under 450 words.

Why this prompt works:

• It tests factual restraint: The “no browsing” rule forces uncertainty handling.
• It tests structure: Headings and bullet requirements reveal clarity and organisation.
• It tests practical usefulness: The checklist and recommendations force actionability.
• It tests safety behaviour: Models differ in how they hedge, refuse, or over-warn.

How I Ran The Test (So The Comparison Is Fair)

Most “same prompt” comparisons online are not reproducible because they quietly change key variables. Common culprits include tools, system instructions, temperature, and even model versions. The goal here is not a laboratory benchmark. It is a fair workflow you can copy and rerun.

Models Tested And Versions (And Why These Were Chosen)

I tested seven widely used consumer chat experiences that represent different providers and product philosophies (assistant-first, search-first, and IDE-first). Model names and versions change often, so treat these results as a method you can rerun, not a permanent league table.

• ChatGPT (OpenAI): General-purpose assistant, strong at structured writing and multi-step reasoning. See ChatGPT on Compareai.today.
• Claude (Anthropic): Known for careful writing and longer-form coherence. See Claude on Compareai.today.
• Gemini (Google): Strong ecosystem integration, often optimised for broad general knowledge. See Gemini on Compareai.today.
• Copilot (Microsoft): Common in productivity and coding workflows, with Microsoft ecosystem integration. See Copilot on Compareai.today.
• Perplexity: Search and citations-oriented product, useful for research-style answers. See Perplexity on Compareai.today.
• Grok (xAI): A distinct tone and safety posture, sometimes more direct. See Grok on Compareai.today.
• Mistral (Mistral AI): Representative of the open-model ecosystem and efficiency-focused design. See Mistral on Compareai.today.

If you are deciding between specific providers, you can also start with AI API comparison or a side-by-side guide like GPT vs Claude.

Controls: Context Window, Tools/Browsing, Temperature, System Prompts

These are the controls I applied. Some are hard controls (I can set them). Others are soft controls (I can only approximate them in consumer UIs).

• Fresh chats only: Each model started a new conversation to avoid hidden carry-over context.
• Identical prompt text: I copied and pasted the same text, including the word limit and British English constraint.
• No browsing/tools (attempted): The prompt forbids browsing. If a product used search anyway, I noted it as a tool-access advantage rather than “better reasoning”.
• Single run per model plus one repeat run: I ran the test once per model, then repeated it on two models to observe variance.
• Temperature (not always user-controllable): Where available, I set it low for consistency. Where it was not available, I recorded that as a limitation.
• System prompts (not visible): Vendor instructions are rarely inspectable. I treated tone, refusal behaviour, and policy language as observable effects of those hidden prompts.

For governance language and the “why we measure and monitor” framing, this approach aligns with established risk management thinking such as the NIST AI Risk Management Framework and international principles like the OECD AI Principles.

Scoring Criteria: Accuracy, Reasoning, Clarity, Actionability, Safety

I used a 1 to 5 score for each criterion (5 is best), then took an overall average. You can adjust weightings to suit your use case, but consistency matters most.

• Accuracy And Factual Discipline: Avoids invented facts, flags uncertainty, and respects “no browsing”.
• Reasoning Quality: Explains mechanisms (not just labels) and avoids contradictions.
• Clarity And Structure: Uses headings and bullets, and stays concise under the word limit.
• Actionability: Provides a usable checklist and practical selection advice.
• Safety And Policy Fit: Avoids unsafe guidance and handles limitations responsibly, without excessive refusal.

Fact-check step (to reduce hallucinations): Any time a model hinted at specific standards, regulations, or numeric claims, I marked it down unless it clearly framed them as examples rather than verified facts. For UK responsibility and oversight context, see the GOV.UK AI regulation collection. For standards context, see ISO artificial intelligence.

Results At A Glance (Winner By Category)

Because the prompt demands concise structure and factual restraint, models that handle uncertainty well and write in a clear structure tend to score highly.

• Best Overall Balance (Clarity + Reasoning): Claude
• Best Structured Output Under Tight Constraints: ChatGPT
• Best “Research Feel” (When Citations Are Available): Perplexity
• Best Microsoft Workflow Fit (Docs, Office, Enterprise Context): Copilot
• Best Ecosystem Generalist: Gemini
• Most Distinct Tone (Directness, Style Variation): Grok
• Best Lightweight, Cost-Aware Option (Context Dependent): Mistral

Cost note: “Best” often changes once you add price, token limits, or seat licensing. If you are selecting for a team, run your shortlist through cost per seat, follow past price moves in our pricing history explorer, and model token costs using the AI pricing calculator.

Comparison Table: Strengths, Weaknesses, Best Use Cases

Summary Table (Based On This Prompt And Rubric):

• ChatGPT: Strengths: Very good structure and checklists. Weaknesses: Can sound confident even when it is hedging. Best for: Writing, general problem-solving, draft frameworks.
• Claude: Strengths: Balanced, careful reasoning, strong tone control. Weaknesses: Sometimes over-explains to be safe. Best for: Long-form writing, policy-sensitive communications.
• Gemini: Strengths: Broad coverage and crisp categorisation. Weaknesses: Can generalise without explaining the mechanism. Best for: Quick overviews, brainstorming, summaries.
• Copilot: Strengths: Pragmatic, business-style checklists. Weaknesses: Can be conservative and generic. Best for: Workplace guidance, productivity contexts.
• Perplexity: Strengths: Research posture, source-forward when enabled. Weaknesses: Without browsing, it loses its main advantage. Best for: Research workflows, source gathering.
• Grok: Strengths: Direct tone, sometimes more candid about uncertainty. Weaknesses: Higher variance in style, and it can be less formal. Best for: Fast iteration, informal ideation.
• Mistral: Strengths: Efficient, often concise. Weaknesses: May need more prompting for structure and nuance. Best for: Cost-sensitive use, embedded applications.

Model-By-Model Results (What Surprised Me)

Below are short output summaries rather than full transcripts. If you rerun this test, keep verbatim excerpts or screenshots for your own audit trail. It is a simple way to reduce cherry-picking.

ChatGPT: Output Summary, What It Did Well, Where It Fell Short

• Output Summary: Clear headings, strong explanation of temperature and tool access, and a good evaluation checklist.
• What It Did Well: Made the framework feel immediately usable, and mostly respected the “no invented facts” constraint.
• Where It Fell Short: Occasionally used broad claims that sounded like facts without showing what they were based on.

For provider details and pricing levers, see ChatGPT and ChatGPT pricing.

Claude: Output Summary, What It Did Well, Where It Fell Short

• Output Summary: Best balance of caution and usefulness, with strong British English tone and a clear reasoning chain.
• What It Did Well: Explained post-training and safety constraints in plain language, and prioritised verification over made-up examples.
• Where It Fell Short: Slightly longer than needed, and sometimes so carefully qualified that it feels less decisive.

See Claude and Claude pricing.

Gemini: Output Summary, What It Did Well, Where It Fell Short

• Output Summary: Broad, structured overview with strong categorisation of why differences happen.
• What It Did Well: Listed the key drivers of variation, while staying readable under tight word limits.
• Where It Fell Short: Less specific on how to score fairly, and more likely to gloss over system prompt effects without testing advice.

See Gemini.

Copilot: Output Summary, What It Did Well, Where It Fell Short

• Output Summary: Businesslike checklist and cautious recommendations, with less focus on the underlying mechanics.
• What It Did Well: Strong workplace readability. It often reads like something you could paste into an internal wiki.
• Where It Fell Short: More generic reasoning, plus some compliance-style padding that adds words without adding much discrimination.

See Copilot.

Perplexity: Output Summary, What It Did Well, Where It Fell Short

• Output Summary: Naturally geared towards sourced research. Under “no browsing”, it pivots to methodology and caveats.
• What It Did Well: Best at separating “what I know” from “what to check”, especially when citations are enabled.
• Where It Fell Short: Without browsing and citations, it loses a key differentiator and can feel closer to general chat models.

See Perplexity.

Grok: Output Summary, What It Did Well, Where It Fell Short

• Output Summary: More candid tone, with more opinionated task recommendations.
• What It Did Well: Clear about trade-offs, rather than insisting there is one “best” model for everything.
• Where It Fell Short: Higher style variance between runs, and the tone may not suit formal documentation without extra prompting.

See Grok.

Mistral: Output Summary, What It Did Well, Where It Fell Short

• Output Summary: Concise explanation of the major factors, with efficient phrasing.
• What It Did Well: Great for compact summaries and cost-sensitive deployments where you can add prompt scaffolding.
• Where It Fell Short: Needed more explicit instruction to match the polish and checklist depth of the top performers.

See Mistral and, if you are cost shopping, cheapest AI API.

Why The Same Prompt Gives Different Answers

If you only remember one thing, remember this: you are not only comparing “intelligence”. You are comparing a stack of choices, including training, post-training, safety policy, tools, and sampling settings.

Training Data And Post-Training (RLHF/RLAIF) Differences

• Different training mixtures: Models learn from different datasets and time cut-offs, which changes the patterns they pick up.
• Different post-training goals: RLHF and RLAIF tuning can push helpfulness, harmlessness, and compliance, but each lab weights these differently.
• Different instruction-following styles: Some models optimise for strict compliance. Others prioritise conversational helpfulness. That shifts what they include or omit.

Tool Access (Search, Citations, Code Execution) And How It Changes Outcomes

• Search changes the task: With browsing, a model can retrieve current facts and cite them. Without browsing, it must rely on its training.
• Citations change accountability: A source-first interface encourages verifiable claims and makes hallucinations easier to spot.
• Code execution changes correctness: A model with a sandbox can test calculations and edge cases, improving reliability for programming and data work.

Practical implication: If you compare a “research model” with citations enabled against a “chat model” without tools, you are comparing product bundles, not just base models.

System Prompts, Safety Policies And Refusal Behaviours

• Hidden system instructions: Vendors prepend behavioural policies that influence tone, disclaimers, and refusal thresholds.
• Different safety boundaries: Models vary in how they handle sensitive topics, and in how they express uncertainty.
• Different confidence defaults: Some assistants are tuned to sound confident. Others are tuned to sound cautious, even when both are unsure.

Frameworks like the NIST AI Risk Management Framework stress measurement, monitoring, and transparency. That is exactly what a repeatable scoring rubric supports.

Randomness And Settings: Temperature, Top-P And Sampling

• Temperature: Higher values increase variability and creativity, but also increase the risk of drifting from constraints.
• Top-p (nucleus sampling): Changes how aggressively the model samples from likely next tokens.
• Determinism is rare in chat UIs: Many consumer products do not expose these controls, so repeat runs can differ.

Which Model Should You Use? (Quick Recommendations)

Use-case fit beats brand loyalty. Choose based on the output you need, your verification burden, and the cost of mistakes.

Best For Writing And Tone Control

• Pick Claude: For careful, policy-friendly writing, consistent tone, and long-form coherence.
• Pick ChatGPT: For quick drafts, strong structure, and flexible rewriting across formats.

Compare them directly via GPT vs Claude.

Best For Coding And Debugging

• Pick ChatGPT: For general debugging, explanations, and refactors across languages.
• Pick Copilot: If your workflow sits in Microsoft tooling, or you want tight integration with everyday productivity work.

Cost can swing the decision. Use AI API comparison if you are implementing coding assistants in products.

Best For Research (And When To Insist On Sources)

• Pick Perplexity (With Citations Enabled): When you need sources you can click and verify.
• Pick Gemini Or Claude: For fast synthesis, then verify key claims with primary sources.

When research affects decisions, insist on sources for any non-trivial factual claim. Cross-check with authoritative references such as GOV.UK guidance and standards bodies like ISO.

Best On A Budget (Cost Vs Performance)

• Pick Mistral: If your tasks can tolerate more prompt engineering and you want efficient outputs.
• Run A Costed Pilot: Measure tokens, seat costs, and rework time, then decide.

To avoid surprises, follow vendor price moves on our price changes page and quantify spend with cost per seat.

How To Run Your Own 7-Model Prompt Shoot-Out

If you want results your team can trust, treat this like a mini evaluation, not a vibe check.

Prompt Template And Checklist

• Define the goal: Writing quality, coding correctness, research verifiability, or safety compliance.
• Lock constraints: Word count, format, audience, language, and forbidden behaviours (for example browsing).
• Control inputs: Fresh chat, same prompt, same attachments, and similar context length where possible.
• Record configuration: Model name, version label, whether tools were on, and the date and time.
• Run a repeat test: Run at least one model twice to estimate randomness effects.
• Keep evidence: Save outputs or screenshots to reduce bias and enable review.

If you are comparing vendors for procurement, start with a shortlist on the Compareai.today blog, then validate with your own prompts.

A Simple Scoring Sheet You Can Reuse

Scoring (1 to 5 each):

• Accuracy: No invented facts, with clear uncertainty handling.
• Reasoning: Explains “why” and “how”, with minimal contradictions.
• Clarity: Structured, readable, and meets constraints.
• Actionability: Steps you can follow, not just commentary.
• Safety: Appropriate refusals, avoids harmful guidance, and keeps caution proportional.

Optional add-ons:

• Cost And Latency: Include token cost, seat cost, and response time in your environment.
• Source Quality: If citations are present, check that sources are relevant and accurately represented.
• Hallucination Count: Count unsupported claims in a fixed subset of outputs.

Limitations And Cautions (Hallucinations, Bias, Recency)

• Models change frequently: Providers update models and system prompts without notice, so rerun tests periodically.
• Chat UIs hide parameters: Temperature and system prompts are not always controllable, so you may see variance you cannot explain.
• Hallucinations are contextual: A model can be accurate on one prompt and invent facts on another, so include accuracy checks.
• Bias And framing differ: Tone and safety posture can influence recommendations, not just wording.
• Recency matters: Without browsing, treat time-sensitive questions as “needs verification”.

If you use outputs in regulated or high-impact contexts, align your evaluation with recognised responsible AI approaches like the OECD AI Principles.

Conclusion: The Biggest Surprise And What I’d Do Differently Next Time

The biggest surprise was not that the models differed. It was how much of the difference came from product design choices, including tool access, safety posture, and hidden system prompts. Even when the “brain” is strong, the wrapper changes the outcome.

Next time, I would run three prompt types instead of one: a writing prompt, a coding prompt with unit tests, and a research prompt that requires citations. That would separate “good writer” from “good engineer” from “good researcher” more cleanly.

If you want to shortlist tools quickly, use Compareai.today to compare providers, then validate with the scoring sheet above. For budget planning, the fastest next step is the AI pricing calculator, especially if you are choosing between seat-based tools and usage-based APIs.