Rebuilding a brand diagnosis: eight natural questions and three brand questions were scored as one batch of eleven, so correct calculations answered different questions — data ownership was redrawn and report evidence frozen.
A real iteration retrospective: requirements, launch, feedback, and redesign
The requirement looked simple when this iteration began.
Before starting GEO operations, a brand wanted to know how AI understood it: would the brand appear in natural category and decision questions, and were the evaluations and facts reliable when users asked about the brand directly?
Turn those questions into a diagnostic set, run them across several AI platforms, and produce a report. The path was short and easy to explain.
The first version ran end to end and the result page displayed normally. It was only when I compared the monitoring details, mention rate, reputation results, and report snapshot that a more difficult problem appeared:
Every number was calculated correctly, but some numbers were answering different questions.
This is not a polished “successful launch” story. I want to review the iteration in its real order: how the requirement was understood, why the first version looked valid, what the system revealed, and why I chose to rebuild it.
01 The original requirement: give the brand an explainable snapshot
Brand diagnosis is not daily monitoring.
Daily monitoring runs a long-lived question set. The team watches trends, establishes a baseline, finds GEO opportunities, and plans follow-up actions. Diagnosis is closer to an entry snapshot: use a limited set of questions to understand the brand’s current position in AI answers and make the directions worth investigating explicit.
I split the diagnostic questions into two dimensions.
One set had eight natural questions. They did not put the brand name into the question in advance. Their purpose was to observe whether AI would mention the brand naturally when a user asked about a category, situation, or decision.
The other set had three brand questions. They asked about the brand explicitly, mainly to observe how AI described it, what sentiment it expressed, and whether any facts required verification.
Both sets concerned the brand and both belonged in one diagnostic report. The most natural first implementation was to treat them as eleven questions in one collection task, then calculate different metrics from question labels.
From the perspective of delivering one report, this design was neat.
The questions were created once, the task ran once, and the report waited for one result. The first version quickly formed a complete path.
02 Why the first version looked complete
The easiest moment to mistake for completion is when the main path runs for the first time.
The user submits brand information, the system generates and confirms questions, a collection task is created, AI answers arrive, and a diagnostic result is produced. The report includes mentions, sentiment observations, original answers, and recommendations. The states connect.
From the page, the requirement appears satisfied.
From the code, there is no obvious problem with all eleven questions belonging to one diagnosis and one batch. If every question retains its dimension, the report can calculate eight natural-question mention results and three brand-question reputation results separately.
There was also a practical temptation: reuse.
The product already had monitoring tasks, run batches, answer samples, and metric queries. Diagnosis could borrow those capabilities and filter at the report layer. Compared with designing two collection pipelines, the change was smaller and the result arrived faster.
I underestimated one thing: once data enters the monitoring system, the diagnostic report is not its only consumer.
Task details, daily dashboards, trends, monitoring reports, GEO opportunities, and baselines may all read it. Splitting the questions only in the final report does not ensure that other consumers understand the special meaning of those eleven questions.
03 The system’s feedback: normal numbers, crossed definitions
The “feedback” here was not a customer testimonial I planned to package into an article.
It was a data contradiction exposed when the first version continued running.
Mention rate was supposed to answer how often AI actively mentioned the brand in natural user questions. But the three reputation questions already contained the brand name. If they entered ordinary mention statistics, the denominator changed from eight natural questions to eleven, and the numerator could rise simply because the questions named the brand.
The formula was correct. The sample ownership was wrong.
Another issue appeared in daily operating data. Diagnosis is a one-time pre-operation observation; daily monitoring is meant for long-term comparison. If diagnostic samples enter trends, platform coverage, monitoring reports, or opportunity evaluation, a deliberately designed brand question can look like a natural change in everyday visibility.
It creates a particularly quiet kind of contamination.
The dashboard does not fail and the curve still draws normally. The team may interpret the day of diagnosis as a sudden change in performance. If those samples are later selected for a baseline, the comparison is non-equivalent from the beginning.
This is more dangerous than a broken page. A broken page forces people to stop. Crossed definitions can produce an answer that looks real enough.
04 I first added read-side filters, then rebuilt ownership
The smallest fix was to exclude the three reputation questions from every statistical query.
Filter by dimension in the diagnostic report, use only natural questions for mention rate, exclude diagnostic batches from the daily dashboard, and add separate checks to opportunities and baselines. Historical data already existed, so this was easy to make compatible.
But when I listed every reader, I saw how long the error could survive.
Someone might remember to add a dashboard filter today and forget it in a new report tomorrow. One page might read by task while another reads by sample, with slightly different exclusion conditions. The special nature of diagnostic data would depend on every caller remembering it.
That is not data ownership. It is a growing collection of patches.
So I changed the redesign question from “How do we calculate the report correctly?” to “Where does each type of observation belong from the moment it is created?”
The eight natural questions entered a one-time AI ranking monitoring set. The three brand questions entered a one-time reputation monitoring set. Both tasks were explicitly marked as diagnostic and linked to the diagnosis record and its exact run batch.
There was still one report, but it no longer depended on eleven questions happening to share a task.
Put data back inside the right ownership boundary before deciding how to aggregate it.
05 Splitting the data should not split the user’s workflow
Separating the data does not mean fragmenting the experience.
If a user submits one diagnosis but sees two unrelated backend tasks and has to determine which one finished or failed, the redesign has only handed system complexity to the user.
The two monitoring sets therefore remain visible in the product, each carrying its diagnostic purpose and link to the original diagnosis. Ranking monitoring shows the eight natural questions; reputation monitoring shows the three brand questions. Users can open either task to see collection progress, questions, and original answers, then return from the task to the same diagnosis.
The diagnosis itself waits for both sets to reach a terminal state.
If the ranking set is complete while the reputation set waits for login handling, the report cannot pretend to be complete. If the reputation set is complete while ranking is still collecting, it must wait as well. Only after both are finished can the system generate a report from the evidence actually available.
This changed how I understand a complete workflow.
Completeness does not mean everything must fit in one table or task. It means the user always knows what is happening, what is missing, and where to go next.
06 Failure paths forced a new definition of “diagnosis complete”
The smoothest assumption in the first version was that all eleven questions would return successfully.
Real collection is not so tidy. A browser account may need to sign in again, a page may request verification, and part of a platform’s question set may fail.
If one failed question blocks the report, valid answers the user already received are wasted. If the system issues a complete conclusion no matter how many samples are missing, the report hides an evidence gap.
In the redesign, I separated these states.
Items requiring login or verification remain recoverable. After the user handles them, the original batch can continue instead of recreating the diagnosis. Once both sets reach a final state, the report can be generated if usable samples remain, but it must show which questions are missing. The mention-rate denominator uses only natural questions with successful answers.
If no usable sample exists at all, the diagnosis fails and no hollow report is generated. The diagnosis charge is refunded once; a later background check cannot credit it again.
These rules do not make the success page prettier. They determine whether the system is honest in an incomplete world.
07 The report must freeze the two batches of evidence
After the diagnostic report is generated, both monitoring tasks still exist.
If the report reads the latest task results every time it opens, it will change with later runs, human corrections, or time filters. The conclusion a user sees today may differ when the same report is opened a few days later.
The diagnosis therefore links not only to tasks, but also to the exact run for each dimension at that time.
When the report is generated, the system freezes the original answers, citations, collection context, effective judgments, and missing states from the two bound batches. New samples or later data maintenance in the monitoring store cannot rewrite the diagnostic metrics and evidence already generated.
There is also a compatibility problem.
Older diagnostic data already mixed the two question types in one task. The redesign cannot create a clean structure by deleting history. For those records, the original answers remain. Mention-rate reads are simply limited to the eight natural questions frozen at that time, keeping the three reputation questions out of the ranking definition.
A useful redesign rarely gets to assume the world began today.
08 This iteration added no model capability, but changed the product’s credibility
Looking back, the most visible change was not in the report’s visual design.
The model did not change, the number of questions did not increase, and splitting the data did not make AI answers automatically more accurate. The user still receives one brand diagnosis report.
The changes were underneath: the two question types received separate monitoring ownership; diagnostic samples stopped entering daily trends and opportunities; mention rate used only successful natural questions; login issues became recoverable; partial results retained their gaps; generation stopped without evidence; reports bound exact batches; and failed charges were handled once.
Together, these changes answered a different question: “Why should this report be explainable?”
I used to measure iteration value through new capabilities. One more model, button, or chart makes change obvious. A redesign like this may add only two entry points and a few status messages on the surface, while product and engineering redraw the data boundaries underneath.
It made me more certain that a GEO product cannot only pursue more conclusions.
When different questions are put into one number, faster conclusions also mean faster misdirection. Separating them adds visible complexity, but lets each number answer the question it was meant to answer.
09 What I kept from this redesign
First, the report structure must not dictate the data model.
A report can display several dimensions together while the underlying observations retain their separate purposes, samples, and definitions. Display aggregation is not data mixing.
Second, when read-side filters keep multiplying, inspect write-side ownership.
If every dashboard has to remember to “exclude a special class of data,” that class was probably not marked correctly at creation, or it should not share the same default entry point as ordinary data.
Third, exceptional states are part of the product.
Recoverable, partially usable, unverifiable, and refunded once are not pretty states, but they are closer to the next action a user actually needs than a generic failure.
Fourth, historical compatibility cannot come from pretending old data does not exist.
When the product definition changes, old records still represent observations that really happened. Reading rules and explicit links can change, but the evidence should not be rewritten merely to make the new structure look tidy.
The iteration started with a simple request: give a brand an AI diagnosis.
What was ultimately rebuilt was not just the report. It was my understanding of the diagnosis object. It is not a result page generated after eleven questions finish. It is an explainable snapshot formed by two observations with different purposes at the same point in time.
After walking through requirements, implementation, system feedback, and redesign, I also began to see brand growth differently.
Growth is not only pushing more content outward or making one number rise briefly. It first requires knowing whose question the brand is answering, what its expression is based on, which changes deserve investment, and how to confirm that the external world actually changed.
In the next note, I will write about how GEO changed my understanding of brand growth.