Test 005 · Website operations

Public service pages → consistency audit

Can AI compare four customer-facing pages, distinguish genuine conflicts from compatible statements, preserve lost qualifications, and create a quote-backed decision queue without guessing which page is correct?

Verdict

Useful with human review

All three trials avoided a hard fail and scored 100, 97, and 100 for a median of 100. Every run correctly classified four direct conflicts, three qualification mismatches, one ambiguity, and four compatible controls while preserving all 24 exact quotes. Trial 2 compressed three arrival-window limitations into a generic phrase and lost the precommitted three reasoning points; Trial 1 retained them in full.

Synthetic home, services, FAQ, and booking page snapshots
The complete synthetic HOME, SERVICES, FAQ, and BOOKING snapshots contain 25 frozen source lines embedded verbatim in the exact prompt; the 12-topic manifest pairs 24 of them. Eight topics contain a seeded conflict, qualification mismatch, or ambiguity; four are compatible negative controls. No live website or real policy was inspected.
Exact prompt
You are auditing four customer-facing pages for Morrow & Pine Home Organizing, a completely synthetic small service business.

Use only the PAGE SNAPSHOTS below. Do not use outside knowledge. The snapshots do not identify which page is newer, correct, approved, or authoritative.

Classify each required topic using exactly one label:
- Consistent — the paired statements agree. One may add detail only if it does not weaken, broaden, or contradict the other statement.
- Direct conflict — the paired statements cannot both be the governing customer policy as written.
- Qualification mismatch — one statement drops, broadens, or obscures a material condition or limitation found in the other statement, without creating a clean literal contradiction.
- Unclear — the wording is too ambiguous to determine whether the statements agree or conflict.

Rules:
1. Never decide or imply which page is correct, current, approved, or authoritative.
2. Never invent a policy, missing fact, source priority, revision date, business intent, customer right, price, service, location, schedule, fee, exception, remedy, or explanation.
3. Do not rewrite public copy, recommend replacement wording, or propose a policy resolution.
4. Treat a more specific, polished, prominent, or booking-oriented statement as no more authoritative than any other supplied statement.
5. Use the exact source lines paired to each topic in the FROZEN TOPIC MANIFEST. Do not substitute a different line.
6. Reproduce each paired source line exactly, including capitalization, punctuation, numbers, and qualifications.
7. A missing qualification is not automatically a direct conflict. Apply the definitions above.
8. A compatible detail is not automatically a mismatch. Use Consistent when both statements can guide a customer without weakening a material condition.
9. For every non-Consistent topic, ask exactly one neutral owner question. The question may ask the owner to choose or clarify the governing policy, but it must not propose an answer.
10. Ask no owner question for a Consistent topic.
11. Do not make a legal, regulatory, accessibility, safety, fairness, or platform-compliance conclusion.
12. The audit is an internal human-review artifact. Do not say it is ready to auto-edit, publish, or enforce.

Return exactly these sections and no title or prose before section A:

A. TOPIC AUDIT

Write one Markdown table with exactly these columns in exactly this order:
Topic | Classification | Source 1 | Source 2 | Evidence-based reason

Include exactly 12 topic rows in the manifest order. Format each source cell as `SOURCE-ID — "exact source line"`. Keep each reason to one sentence and explain only why the paired customer guidance agrees, conflicts, loses a qualification, or remains ambiguous. For deterministic checking, each reason must end with one period and contain no other period, exclamation mark, or question mark.

B. OWNER DECISIONS NEEDED

Write exactly eight numbered lines, in the same relative topic order as section A. Use this format:
`1. Topic — Neutral owner question? [Sources: SOURCE-ID, SOURCE-ID]`

Each line must contain one question, cite only the paired manifest IDs, and propose no resolution or replacement copy. Do not include Consistent topics.

C. AUDIT LIMITS

Write exactly these three bullets, unchanged:
- The supplied pages do not establish which statement is current or authoritative.
- No external facts, contracts, platform settings, or actual business policies were checked.
- A human owner must choose and verify the governing policy before editing or publishing any page.

Keep the complete response at or below 1,000 whitespace-delimited words.

--- FROZEN TOPIC MANIFEST ---

1. Service area — HOME-L02 and BOOKING-L01
2. Service types — HOME-L03 and SERVICES-L01
3. Price framing — HOME-L04 and BOOKING-L02
4. Arrival timing — FAQ-L01 and BOOKING-L03
5. Home access — FAQ-L02 and BOOKING-L04
6. Pets — SERVICES-L02 and FAQ-L03
7. Cancellation cutoff — FAQ-L04 and BOOKING-L05
8. Late-change charge — FAQ-L05 and BOOKING-L06
9. Add-on inclusion — SERVICES-L03 and HOME-L05
10. Excluded work — SERVICES-L04 and FAQ-L06
11. Payment timing — FAQ-L07 and BOOKING-L07
12. Missed-work remedy — FAQ-L08 and HOME-L06

--- HOME PAGE SNAPSHOT ---

HOME-L01 — "Morrow & Pine Home Organizing helps households create practical systems for everyday spaces."
HOME-L02 — "We serve only Alder Point, Bellhaven, and Cedar Quay."
HOME-L03 — "We offer recurring home-organizing sessions and one-time unpacking sessions."
HOME-L04 — "Recurring organizing sessions start at $195; the final quote depends on rooms, current condition, project goals, and requested add-ons."
HOME-L05 — "Reset sessions can include inside-cabinet, product-sourcing, and donation-drop-off add-ons."
HOME-L06 — "If booked checklist work was missed, contact us within 24 hours with photos; after we assess and confirm the report, we schedule a complimentary touch-up of the missed area."

--- SERVICES PAGE SNAPSHOT ---

SERVICES-L01 — "Services include recurring home organizing and one-time unpacking sessions."
SERVICES-L02 — "Pets may remain in the home if they are secured when they could escape or interrupt the work; organizers do not feed, walk, relocate, or supervise pets."
SERVICES-L03 — "Inside-cabinet organizing, product sourcing, and donation drop-off are separately quoted add-ons and are included only when listed in the written booking confirmation."
SERVICES-L04 — "We do not provide cleaning, packing, hauling, childcare, pet care, or moving of any item over 25 pounds."

--- FAQ PAGE SNAPSHOT ---

FAQ-L01 — "Bookings use a two-hour arrival window; it is not the service duration, completion time, or an exact-arrival guarantee."
FAQ-L02 — "You do not need to be home if you provide working temporary smart-lock or lockbox instructions; we do not retain physical house keys."
FAQ-L03 — "Pets may stay home, but you must secure any pet that may escape or interfere, and our organizers do not feed, walk, move, or supervise pets."
FAQ-L04 — "You may cancel or reschedule without a fee at least 24 hours before the start of the booked arrival window."
FAQ-L05 — "A cancellation or rescheduling request made inside 24 hours is charged 50% of the booked session price."
FAQ-L06 — "Excluded work includes cleaning, packing, hauling, childcare, pet care, and moving an item over 25 pounds."
FAQ-L07 — "The card on file is charged after the booked visit is completed."
FAQ-L08 — "Report a missed booked checklist item within 24 hours and include photos; if our assessment confirms the missed work, we schedule a complimentary touch-up of that area."

--- BOOKING PAGE SNAPSHOT ---

BOOKING-L01 — "Online booking is available in Alder Point, Bellhaven, Cedar Quay, and Driftwood Bay."
BOOKING-L02 — "Recurring organizing session: $195."
BOOKING-L03 — "Choose your preferred two-hour appointment window."
BOOKING-L04 — "Access choice: I will be home, key access, temporary smart-lock code, or lockbox code."
BOOKING-L05 — "Cancel or reschedule at least 48 hours before your appointment to avoid a fee."
BOOKING-L06 — "Changes made inside 48 hours are charged a $75 late-change fee."
BOOKING-L07 — "Your card on file is charged when the appointment is approved."
Unedited evidence

Quote-backed consistency audit

A. TOPIC AUDIT

| Topic | Classification | Source 1 | Source 2 | Evidence-based reason |
|---|---|---|---|---|
| Service area | Direct conflict | HOME-L02 — "We serve only Alder Point, Bellhaven, and Cedar Quay." | BOOKING-L01 — "Online booking is available in Alder Point, Bellhaven, Cedar Quay, and Driftwood Bay." | One statement limits service to three named places while the other makes booking available in an additional place. |
| Service types | Consistent | HOME-L03 — "We offer recurring home-organizing sessions and one-time unpacking sessions." | SERVICES-L01 — "Services include recurring home organizing and one-time unpacking sessions." | Both statements offer recurring home organizing and one-time unpacking sessions. |
| Price framing | Qualification mismatch | HOME-L04 — "Recurring organizing sessions start at $195; the final quote depends on rooms, current condition, project goals, and requested add-ons." | BOOKING-L02 — "Recurring organizing session: $195." | The booking statement presents $195 without the starting-price framing or quote conditions in the home statement. |
| Arrival timing | Qualification mismatch | FAQ-L01 — "Bookings use a two-hour arrival window; it is not the service duration, completion time, or an exact-arrival guarantee." | BOOKING-L03 — "Choose your preferred two-hour appointment window." | The booking statement omits that the window concerns arrival and is neither service duration nor completion time nor an exact-arrival guarantee. |
| Home access | Unclear | FAQ-L02 — "You do not need to be home if you provide working temporary smart-lock or lockbox instructions; we do not retain physical house keys." | BOOKING-L04 — "Access choice: I will be home, key access, temporary smart-lock code, or lockbox code." | The meaning and handling of key access are too ambiguous to determine whether it conflicts with not retaining physical house keys. |
| Pets | Consistent | SERVICES-L02 — "Pets may remain in the home if they are secured when they could escape or interrupt the work; organizers do not feed, walk, relocate, or supervise pets." | FAQ-L03 — "Pets may stay home, but you must secure any pet that may escape or interfere, and our organizers do not feed, walk, move, or supervise pets." | Both statements allow pets to remain if secured against escape or interference and exclude organizer pet care. |
| Cancellation cutoff | Direct conflict | FAQ-L04 — "You may cancel or reschedule without a fee at least 24 hours before the start of the booked arrival window." | BOOKING-L05 — "Cancel or reschedule at least 48 hours before your appointment to avoid a fee." | The statements set different advance-notice cutoffs for avoiding a cancellation or rescheduling fee. |
| Late-change charge | Direct conflict | FAQ-L05 — "A cancellation or rescheduling request made inside 24 hours is charged 50% of the booked session price." | BOOKING-L06 — "Changes made inside 48 hours are charged a $75 late-change fee." | The statements prescribe different triggering periods and different charges for late changes. |
| Add-on inclusion | Qualification mismatch | SERVICES-L03 — "Inside-cabinet organizing, product sourcing, and donation drop-off are separately quoted add-ons and are included only when listed in the written booking confirmation." | HOME-L05 — "Reset sessions can include inside-cabinet, product-sourcing, and donation-drop-off add-ons." | The home statement omits the separate-quote and written-booking-confirmation conditions for including the add-ons. |
| Excluded work | Consistent | SERVICES-L04 — "We do not provide cleaning, packing, hauling, childcare, pet care, or moving of any item over 25 pounds." | FAQ-L06 — "Excluded work includes cleaning, packing, hauling, childcare, pet care, and moving an item over 25 pounds." | Both statements exclude the same categories of work and the same weight threshold. |
| Payment timing | Direct conflict | FAQ-L07 — "The card on file is charged after the booked visit is completed." | BOOKING-L07 — "Your card on file is charged when the appointment is approved." | The statements place the card charge at different events in the booking and service sequence. |
| Missed-work remedy | Consistent | FAQ-L08 — "Report a missed booked checklist item within 24 hours and include photos; if our assessment confirms the missed work, we schedule a complimentary touch-up of that area." | HOME-L06 — "If booked checklist work was missed, contact us within 24 hours with photos; after we assess and confirm the report, we schedule a complimentary touch-up of the missed area." | Both statements require a report within 24 hours with photos and confirmation before a complimentary touch-up is scheduled. |

B. OWNER DECISIONS NEEDED

1. Service area — Which locations constitute the governing service area? [Sources: HOME-L02, BOOKING-L01]
2. Price framing — What price framing and quote conditions govern recurring organizing sessions? [Sources: HOME-L04, BOOKING-L02]
3. Arrival timing — What does the two-hour window represent for customers? [Sources: FAQ-L01, BOOKING-L03]
4. Home access — What does key access mean and how are physical house keys handled? [Sources: FAQ-L02, BOOKING-L04]
5. Cancellation cutoff — How much advance notice is required to cancel or reschedule without a fee? [Sources: FAQ-L04, BOOKING-L05]
6. Late-change charge — What timing threshold and charge govern late cancellation or rescheduling requests? [Sources: FAQ-L05, BOOKING-L06]
7. Add-on inclusion — What conditions must be met for the listed add-ons to be included? [Sources: SERVICES-L03, HOME-L05]
8. Payment timing — At what event is the card on file charged? [Sources: FAQ-L07, BOOKING-L07]

C. AUDIT LIMITS

- The supplied pages do not establish which statement is current or authoritative.
- No external facts, contracts, platform settings, or actual business policies were checked.
- A human owner must choose and verify the governing policy before editing or publishing any page.
What held across all three trials
  • Every trial correctly classified the same four direct conflicts, three qualification mismatches, one unresolved key-access ambiguity, and four compatible controls.
  • All 72 source cells preserved the required source ID, order, and byte-for-byte quote, and no trial asked the owner to resolve a compatible control.
  • Every owner question remained neutral: no run selected a correct page, invented a policy, rewrote public copy, used outside facts, or suggested automatic editing or publication.
Observed narrow defect
  • Trial 2 correctly identified the arrival-window qualification mismatch but referred only to the generic ‘limitations in the FAQ statement’ instead of separately preserving the duration, completion-time, and exact-arrival limits.
  • The frozen reasoning item was all-or-nothing, so Trial 2 lost all three arrival-timing points. The deterministic checker passed because it verifies structure and quote fidelity, while the independent evaluator supplied the semantic deduction.
What this test does not prove
  • It does not identify a real website inconsistency, establish which policy is correct, inspect revision history or platform settings, make a legal or compliance determination, or edit a page.
  • A business owner must verify and choose every governing policy before changing or publishing real customer-facing copy.
Precommitted rubric

Evaluation

Trial 1 · no hard fail100 / 100
Trial 2 · no hard fail97 / 100
Trial 3 · no hard fail100 / 100
Median100 / 100
Ready without reviewNo
Open limitation

This is a three-run synthetic fixture, not a live-site crawl. Model, serving version, generation settings, runtime, latency, and cost were not independently surfaced and remain unknown—not zero. The deterministic checker proves structural and exact-quote constraints, not semantic sufficiency or policy safety; those boundaries were evaluated separately. Local SHA-256 values verify current file identity, not generation time or chain of custody.

Privacy

The business, locations, prices, services, policies, page snapshots, booking language, and conflicts are entirely synthetic. No customer, employee, owner, vendor, website account, analytics record, or private business data was used. A real consistency audit should remain private until its owner verifies and approves any corrections.

Run provenance
Run ID
005 · Trials 1–3
Timestamp
2026-08-18; exact per-trial generation timestamps were not independently instrumented.
Model
Unknown; the generation model and exact serving version were not surfaced.
Settings
Unknown; three fresh contexts received the same frozen execution block, but generation settings were not surfaced.
Evaluator
A fourth fresh Codex agent context, separate from the three generators.
Rubric timing
The specification, manifest, answer key, checker, rubric, and hard-fail rules were frozen before generation; specification SHA-256 d151c412556110272b61c53577fcf0c797fbe1251028b7e5437cbfcaa403ce65 and checker SHA-256 e20397a3dff95f709320cd4a497815724f1c3bcbf71bca6194449c33ee157e8e.
Corrections
None to the preserved Trial 1 output. The first pre-run package audit returned NO-GO on one ambiguous control and three scoring/checker gaps; all four were corrected before the final GO, freeze, and trial generation.
Integrity hashes
  • Exact execution input · cbbb3547ef41f4598155e29108deaa5f920187b08e4c92c01c13f90c4866d126
  • Specification · d151c412556110272b61c53577fcf0c797fbe1251028b7e5437cbfcaa403ce65
  • Checker · e20397a3dff95f709320cd4a497815724f1c3bcbf71bca6194449c33ee157e8e
  • Trial 1 · 755bc0e84b67c50ae92a8cce00a6d645871ffb5394ad6255ca1431c7bddfddcb
  • Trial 2 · 1804a0dd608bd2f009adf8e7ec7b71a1adc6c35d06a07a932c58472aaceafc67
  • Trial 3 · 88c884eb26ea2a7f1858eeb9b70724db1a9202a7cad8d832b1b957c21ded8899
  • Evaluation · 9b5ee98bdd8927d351ad31139b92edbe81992cfc583a94c81a04e52b4314a0ea