What Deepify works with
The models your customers consult sit on a synthesis of the open market — products, prices, reviews, comparisons, complaints — and they hand it out, live and web-grounded, to anyone who asks. That is a body of market knowledge, and it is directly interrogable. Deepify's raw material is what comes back when you interrogate it properly: dozens of structured exchanges per run, across the assistants your customers actually use, from angles a customer would never think to try.
Two things about that material govern everything downstream. The prompts real people type are private to the providers — nobody outside them has that data — so rather than pretend otherwise, Deepify publishes the question set it uses. And the exchanges are real: every answer worked from came from a live assistant answering a live question, never from a model asked to imagine a customer.
So the two halves of the frame are treated differently, and the difference is the whole discipline:
- The assistants are a census. There is a small, finite set your customers actually use, and we query all of the ones that can be reached. There is no sampling error in choosing who to ask.
- The questions are a sample. For a projection, derived per run from the move itself and shown in the briefing. For an audit, drawn from a published, versioned frame listed in Part II. Either way the question set is stated — not a guess presented as data.
Every score carries an interval because the question space is sampled. The interval is the honest price of that, and it is why the number can be trusted.
Projection — Liquid GTM
Your move hasn't launched, so there is nothing to measure and no honest way to score it. A projection is a different kind of claim and it is held to a different standard: every one carries what it rests on, what would prove it wrong, and a ceiling it cannot exceed.
What a projection is built from
Before anything is projected, the move is taken apart. Every component is classified by how new it is — already exists, an upgrade to something you offer, new to the market, or new to the world — because that decides what evidence is even available for it. A rate change can be checked against the market. A category that doesn't exist yet cannot, and has to be reasoned about from things that do.
The engine then gathers three kinds of material, concurrently:
| Material | How it's gathered |
|---|---|
| Demand signal | The assistants are asked, in consumer language, about the demand each component depends on — does anyone want this, what would they pay, what would make them switch. |
| Competitor set | Established by live web search, with sources kept. Never assumed, never from memory. |
| Comparables | Deepify Ultra builds a web-verified dossier per novel component — who has done the nearest thing, what they charged, how it landed. |
A run reports how much of each it actually gathered. That count is not decoration: it is the input to the ceiling below, so the engine's own confidence is a function of how much material it managed to find.
Confidence is capped by evidence, in code
Confidence is not the model's opinion of itself. It is a ceiling computed from the evidence the run actually gathered, applied after the report is written, and it can only ever lower a number:
| Evidence gathered | Confidence ceiling |
|---|---|
| No verified source | 35 |
| Fewer than 3 sources | 70 |
| No competitor established | 30 |
| Fewer than 3 competitors | 65 |
For a projection about something that does not exist yet, the ceiling comes from evidence density — each live demand signal counts once, each web-verified comparable counts double:
none → 45 · under 4 → 65 · under 8 → 80 · otherwise → 95
Nothing reaches certainty, because nothing about an unlaunched product is certain.
Every claim carries its basis
Each claim is labelled measured, comparable, or projected — and that label is assigned in code, from the evidence actually collected, never by the model writing the report. A claim of "measured" on a run where nothing was measured is downgraded automatically. Any figure in the prose that cannot be traced back to a gathered source is flagged in the report itself.
A measured-sounding claim is settled by the code, from the raw signals. It is never left to the model that writes the summary.
Every projection carries a falsifier
Alongside its basis, each projected segment response states the specific thing that would prove it wrong — the fee that collapses the trust story, the match that has to feel meaningful rather than token.
This is the discipline the whole part rests on. A projection that cannot be wrong is not a projection, it is an opinion; and an opinion with a confidence score on it is worse than one without, because the number lends it a borrowed authority it did not earn. A falsifier is what makes a forward-looking claim checkable against reality once reality arrives — and it is what turns a briefing into something you can hold us to.
Measurement — the Audit
Once the product is live, what the layer says about it stops being a projection and becomes a fact — one that can be checked against another fact. Everything in this part is measurement: a verdict on every answer, a score with its uncertainty, and a record anyone can check.
The sample
All three assistants are queried live with web search enabled, because that is how your customer meets them:
| Assistant | Model |
|---|---|
| ChatGPT | gpt-4o-search-preview |
| Claude | claude-sonnet-4-6 |
| Gemini | gemini-2.5-flash |
Each question is asked between one and ten times per assistant. Repetition is deliberate: an assistant that answers differently on repeat asks is telling you something a single query would hide, and that variance is measured rather than averaged away.
The prompt battery — prompt-battery-2026.06.1
Six intents, because a customer asking "what's the rate" and a customer asking "should I get this" are not the same test, and a product can pass one and fail the other:
- Rate & fees · Overview & catches · Is it worth it
- How to open / eligibility · Comparison · Recommendation
The battery is versioned, and the version travels with every result. A score is only ever compared to another score run on the same frame.
The score
Every answer receives exactly one verdict against the verified first-party truth:
| accurate | The figure matches the verified truth. |
| partial | Directionally right, incomplete or imprecise. |
| differs | A figure was given and it is not the true one. |
| no figure | The question was answered without ever stating the figure. |
Accuracy is the share of judged answers that were right. Where a run could not be measured, it is reported as not applicable — never as zero. A zero is a result; an unmeasured run is an absence. Collapsing the two is how a dashboard starts lying, so the engine refuses to do it in code.
The interval, and why it's a Wilson interval
Every accuracy score is reported with a 95% Wilson score interval and its sample size:
half = (z / (1 + z²/n)) · √( p(1−p)/n + z²/4n² )
Why not the textbook interval
The one most people reach for is Wald — p ± z√(p(1−p)/n) — and it fails in exactly the conditions an audit operates in.
- It's too narrow when n is small. It overstates how much you know from a handful of answers.
- It runs off the end of the scale. Near 0% or 100% it produces bounds below zero or above one hundred, which are not possible accuracies.
- It collapses at unanimity. When every answer agrees, Wald reports an interval of zero width — perfect certainty, from three responses. That single property disqualifies it.
Wilson stays inside 0–100, behaves at the extremes, and widens honestly when the sample is thin. The practical consequence is the point: a small sample produces a wide interval and we show it wide. Asking more questions is the only thing that narrows it — which is what the battery is for.
How we know the grader is right
The grader is a model, so its judgement is itself measured against human judgement on a fixed, hand-labelled set — judge-gold-2026.06.1 — run through the exact production grading prompt.
Why κ, and not just the percentage
Raw agreement flatters any grader facing a lopsided set. Mark everything "accurate" on a set that happens to be mostly accurate and you score well while knowing nothing at all.
Cohen's κ subtracts the agreement you would get by chance. It asks how much better than luck the grader is, given how the labels are distributed:
Above 0.8 is generally read as near-perfect agreement. At 0.917, the agreement is real rather than arithmetic luck — and the grader returns the same verdict every time it sees the same case, so the number is stable as well as accurate.
The gold set is versioned. Every case and every result is published in the engine's model card, and the calibration is re-run on any change to the grading prompt or the grading model.
Measured visibility, not share of voice
The battery names your product, so it measures accuracy. Getting found in the first place is a different question — an assistant asked about you will talk about you — so the engine measures it separately, and measures it directly.
It derives the unbranded question a customer actually asks — "what's the best premium travel card in Canada? I put about $50k a year on my card" — sends it to the same assistants, and measures from their answers:
- How often you are named at all
- How often you are the first recommendation
- Your average position, where the answer ranks them
This is the number that decides whether a customer ever reaches you. Being mentioned often on questions that already name you is a different measurement entirely — and only one of the two is a customer.
The record
Every run emits a reproducibility record: the engine version, every model and its version, the questions verbatim, the temperature and run count, the benchmark with its sources and confidence, the full set of answers, the verdict tally, the scoring formula, and the interval method.
That record is fingerprinted with SHA-256, and the fingerprint is written to our ledger at the moment the run executes — before anyone has seen the result. The ordering is the point: a record sealed before the outcome is known cannot be quietly improved afterwards.
Anyone holding the briefing can recompute the fingerprint from the attached record and recompute the SHA-256 and match it against the record. No account, no login. The check confirms the record exists and when it ran; it never returns the contents of anyone's briefing.
Drift
Each run is compared against its predecessors, and a change is only reported as a change when the two confidence intervals do not overlap. Movement inside the margin is noise, and noise is not news.
Start with the shape of it
The nine steps behind every briefing, in plain English — from your question to a sealed, independently checkable record.
How does Deepify work? →