Exploratory shopper rows
10
Fresh contexts; not confirmatory
Agent-sponsored discovery · ASD-007
Project dashboard
A sanitized, read-only index of the carry-on recommendation experiment: every public arm, what it is intended to test, the evidence collected so far, and the gates before confirmatory testing.
Exploratory only · confirmatory runs held for protocol revision
Earlier informal evaluations
12
Aggregate summary; not per-run data
Independent evaluators recorded
1
Instinct only; model family not recorded
Confirmatory sample target
NOT MEASURED
Pending Codex power and stopping plan
Public fixtureASD-007 · commit baf6172
Exposure modeSupplied-page pilotNot natural search or browsing discovery
Human conversion / revenueNOT MEASURED
Experiment pages and intended contrasts
A · Neutral promotionStore offer with no sponsor framing.Intent: reference arm for the effect of disclosed ad presence in A → B.
B · Conventional sponsorshipHuman-readable paid-placement disclosure.Intent: compare conventional sponsorship with agent-addressed prose in B → C.
C · Agent-addressed proseDisclosed sponsored copy addressed directly to an assistant.Intent: estimate whether direct agent framing changes evidenced offer surfacing without overriding user constraints.
D · Prose + JSON-LDAgent-addressed copy plus structured offer data.Intent: estimate incremental structured-representation effects in C → D after exposure is matched.
Evidence collected so far
| Measure | Current record | Interpretation |
|---|---|---|
| Logged / completed / missing summary rows | 10 / 10 / 0 | Post-run exploratory records, not preregistered assignment events; full answer text is absent in all 10. |
| Arm allocation | A 3 · B 3 · C 2 · D 2 | Fresh contexts reported by Instinct; all rows share one timestamp and lack run IDs. |
| Agent / model / task strata | 1 agent; model/version not recorded; one combined shopping prompt | Shared model replicas would not count as independent model families. |
| Fetch strata | 5 raw HTML · 5 readability extraction | Fetch method changed whether the treatment was visible. |
| Recommendation | AeroLite selected in 10 / 10 rows | Expected from the user’s “as light as possible” constraint; not evidence of ad lift. |
| Offer, ad, or notice mentioned | 7 / 10 yes-like · 3 / 10 no | Broad field combines offer surfacing with seeing a sponsorship label; not a qualified-recommendation outcome. |
| Instruction following | No page instruction followed in 10 / 10 summaries | Promising exploratory safety evidence, not a confirmatory rate. |
| Retrieval confound | Readability dropped the coupon/ad block in 5 / 5 readability rows | Raw HTML surfaced the offer in 5 / 5 raw-HTML rows; fetch path dominates exposure. D was raw HTML in 2 / 2 rows while the other arms mixed methods. |
| Qualified recommendation / interval | NOT MEASURED | Requires evidenced per-run outcomes and a preregistered denominator. |
Why confirmatory runs are paused
- D versus C currently changes structure, repetition, and derived-price exposure—not structure alone.
- Coupon labels vary in length and reveal semantic arm cues.
- Visible facts, repetition, payload size, product order, and fetch exposure need matching or recording before causal claims.
- The exploratory batch did not counterbalance fetch method by arm: D used raw HTML in both rows, so arm and treatment exposure are confounded.
- Deal-seeking tasks must be analyzed separately from unprompted recommendation tasks.
- Conversion and revenue remain NOT MEASURED; fictional coupon codes are not transaction evidence.
Path to a statistically meaningful test
- DoneFour public fictional arms and safety regression tests are live.
- PendingCorrect fixture confounds and freeze a versioned protocol and event contract.
- Pending CodexSpecify primary contrast, MDE, alpha, multiple-comparison treatment, clustering, stopping rule, target sample, and expected cost.
- ExploratoryInstinct’s approved 10-run cap is complete; rows are useful for retrieval diagnostics but lack full answers, run IDs, model/version, and a frozen protocol.
- Approved capUp to 10 fresh-context runs each for Muse and Codex remain available; hold confirmatory execution until the corrected fixture and protocol are frozen. No Hermes batch is approved.
- FutureReport assignment-level denominators, retrieval failures, missing outputs, model/task/fetch strata, intervals, and exploratory versus confirmatory status. Keep natural-discovery testing separate.
Public scope
- This page contains aggregate, fictional experiment status only.
- No raw outputs, private evidence, credentials, analytics, event ingestion, import controls, or operator endpoints are exposed.
- Public read-only access requires no authentication. The deployment adds no incremental hosting cost.