# From complete records to discriminating tests

Prepared for Mind Guild, 13 November 2026.

This is an original methodological critique informed by Lucia Melloni's public work with COGITATE. It is not written by her, does not reproduce her personal voice, and does not imply her review, participation, or endorsement.

The methodological starting points are public: Melloni and colleagues' 2023 protocol describes jointly developed, proponent-endorsed competing predictions, planned interpretations, impartial testing, and built-in replication [1]. COGITATE's 2025 report makes a further distinction between testing proposed biological implementations and directly testing the mathematical or computational core of a theory [2]. The changes below are Mind Guild's engineering adaptation to simulation research, not a claim that COGITATE used this schema or these software rules.

## Overall assessment

V2 is strong at preserving the record. It distinguishes proposals from decisions, implementation from learning, demonstrations from verification, and prior results from prospective runs. Keep those protections.

Its central weakness is that a thoroughly documented study can still be uninformative. A primary hypothesis, rival-explanation paragraph, large seed sweep, and current review signatures do not establish that the measurement can distinguish the rival, that the inference has adequate sensitivity, or that a contrary result will change the interpretation. The freeze gate should help researchers examine those issues before results arrive.

## Critique and implemented response

| Weakness in v2 | Why it matters for Mind Guild | V3 response |
| --- | --- | --- |
| The level of inference is mostly prose. | An object-tracking advantage can support a mechanism under a generator without establishing subjective experience. A recurrent network is not automatically an implementation of a complete consciousness theory. | An inference contract identifies a mechanism benchmark or theory-discrimination study, its phenomenon, target level, allowed/excluded conclusions, and the distinction between a theory's core and its implementation. |
| A rival can be mentioned without receiving its own prediction. | A weak comparator can make almost any interesting model look theoretically diagnostic. No prediction from a theory is different from a prediction of no effect. | Versioned competing accounts receive separately derived quantitative prediction regions on a common estimand. A primary theory-discrimination claim needs possible exclusive outcomes for each account. Shared and unspecified predictions remain visible. |
| Support/challenge/inconclusive categories can hide asymmetry. | Teams may interpret a favorable result narrowly and an unfavorable one as an implementation problem, or mistake a wide interval for a rejected prediction. | Prespecify target-only, rival-only, both, neither, inconclusive, and invalid outcomes. Partial overlap of an uncertainty interval with a prediction boundary is inconclusive. Measurement validity is assessed separately. |
| Execution feasibility is stronger than evidence of sensitivity. | More episodes from one training run do not supply independent evidence. A test might miss the meaningful effect or favor the target under the rival generator. | Design assurance records the minimum meaningful effect, assumption range, independent unit, repetition count, null/target/rival calibration, and pinned evidence. Rate criteria use 95% Wilson bounds, making Monte Carlo uncertainty visible. Calibration becomes stale when its scientific basis changes. |
| Review roles do not guarantee a serious challenge. | One enthusiastic team can approve all three v2 roles while leaving an easy alternative untested. | Assign a perspective reviewer to each account and a coordinator; disclose conflicts and overlapping roles. Record actual acceptance of the design and all interpretations. Internal critics and informed proxies are allowed but are not labeled as proponent endorsement. |
| Outcome access, replication, and amendments are not one operational contract. | A reserved seed set can become another tuning set. Repeated seeds are not an independent implementation. Interpretations can shift after unblinding. | Plan masked access or explain its limitation, pin analysis code, and define an unblinding gate and access log. Distinguish reserved-run replication, independent implementation, and no replication claim. Preserve amendments, exposure, dissent, all attempts, and outcome-independent reporting. |

## What this changes for the occlusion study

The current candidate should remain a **mechanism benchmark**, pending actual researcher selection. Its proposed question is whether persistent object identity improves the specified binding outcome relative to a resource-matched alternative, under a pinned generator. It should not be titled or interpreted as a test showing that one network is conscious.

Before freezing that candidate, the team needs to choose the quantitative contrast and smallest effect of interest, implement and check the task and scoring, constrain the strongest alternative, run disclosed design calibration, and record critical review. Identity-swap scoring, hidden-state leakage, resource matching, and the possibility that the advantage is engineered into the system deserve explicit attention. A visible control alone is insufficient if differences in training, capacity, or input privileges still explain the contrast.

For the Bogdan Fly draft, the same boundary applies: an implemented network, a persuasive demonstration, changing state, and a proposed learning rule are separate observations. V3 migration preserves the draft and leaves the new contract unresolved. It supplies no missing algorithm, quantitative prediction, calibration result, contributor agreement, or theory endorsement.

## Keep the process usable

Do not try to settle every choice during a conversational meeting. Finish the ten-minute close with one bounded question, the strongest alternative, a sketch of the six outcome interpretations, and named follow-up work. Complete the structured contract during a results-blind design review, then freeze and register. Draft compilation remains available throughout.

Small teams can disclose internal review and open analysis with appropriate controls. The software does not require a famous theorist, a multi-lab consortium, or a claim about consciousness. Theory-level claims carry stricter diagnosticity, decision-calibration, and replication requirements in this implementation. Those are Mind Guild policy choices, not universal preregistration requirements.

## Limits of the improvement

The current quantitative engine handles two accounts and a scalar uncertainty interval per contrast, with closed prediction regions. It is a conservative, inspectable starting point. Nested predictions, likelihood models, multidimensional patterns, and graded theory comparison can require a separately reviewed extension; do not force them into false opposing intervals. Nondiscriminating benchmarks remain useful when described accurately.

Wilson intervals assume independent Bernoulli calibration draws. Synthetic calibration only checks performance under the chosen generators and assumptions. The tool does not establish that the generator represents the relevant biological phenomenon, that an operational measure identifies a construct, or that a neural theory transfers to artificial systems. Calibration code, inputs, counts, independence assumptions, and report bytes need human inspection.

As in v2, machine-readable completion is not scientific validity. The software cannot authenticate signoffs, detect invented declarations, adjudicate every narrative inconsistency, or establish phenomenal consciousness. A local freeze is not an OSF registration or a journal's Registered Report acceptance.

## Sources

1. Melloni L, Mudrik L, Pitts M, et al. (2023). *An adversarial collaboration protocol for testing contrasting predictions of global neuronal workspace and integrated information theory*. PLOS ONE 18(2): e0268577. https://doi.org/10.1371/journal.pone.0268577
2. COGITATE Consortium et al. (2025). *Adversarial testing of global neuronal workspace and integrated information theories of consciousness*. Nature 642, 133–142. https://doi.org/10.1038/s41586-025-08888-1
3. ARC-COGITATE. Lucia Melloni project profile. https://www.arc-cogitate.com/lucia-melloni
