Skip to content

MCP research workflow experiments, 2026-09-27

Decision: determine which bounded social-science and public-transparency questions the fork MCP can support, then repair reproduced server defects without overstating source coverage.

Hypotheses: server-native access restrictions can refuse reads outside the selected public datasets while keeping legitimate local and HTTPS Parquet queries usable; correct error signaling and unique result-column requirements prevent silent client mistakes; a dated Federal Register key prevents cross-date matches. Research hypotheses and their population/causal limits are preregistered separately by the medium-effort research agents before new observations.

Arms: the deployed endpoint observed in the retained blind test is the historical baseline. Compare a locally tested candidate and then the same failing requests and positive controls against its deployed version. Freeze synthetic query inputs; record live publication pins when real data changes. Security configuration alternatives are compared separately before choosing an implementation.

Cases: the actual /etc/os-release read, duplicate aliases and duplicate labels from a join, unknown tables, invalid row limits, refused writes to a nonexistent table, the C2-15835 cross-date join, and ordinary bounded source queries. Positive controls include configured local Parquet, HTTPS Parquet, scalar SQL, CTEs, and legitimate composite-key joins. External file tests use harmless OS metadata or constructed fixtures only.

Held constant: the supplied fork endpoint, advertised tool workflow, bounded result limits, and stored query arguments. Each agent owns a separate evidence directory; no implementation edits overlap. The research agents stop at their preregistered case and request limits. No load or denial-of-service tests, sensitive file reads, unrequested data regeneration, or inferred causal conclusions.

Decision rule: adopt a fix only after the original failing case and its positive controls pass, the relevant repository checks pass, and the deployed endpoint reproduces the intended behavior. A dataset's missing coverage remains a documented research limit unless a source-faithful bounded repair can be validated. Repeated exploration stops when it resolves the declared decision or reaches its bound. Report local, deployed, and source-qualified evidence separately.

Evidence roots (local, outside Git):

  • ~/.codex/artifacts/spicy-regs-mcp-blind-20260927T175947Z/ — immutable historical HTTP observations and report.
  • ~/.codex/artifacts/spicy-regs-social-chaos-20260927/ — preregistered social-data workflows.
  • ~/.codex/artifacts/spicy-regs-reference-chaos-20260927/ — source vocabularies and identity comparisons.
  • ~/.codex/artifacts/spicy-regs-duckdb-boundary-20260927/ — isolated native access-control comparison.
  • ~/.codex/artifacts/spicy-regs-mcp-repair-20260927/ — candidate, deployment, and live regression receipts.

The user explicitly authorized fixes and deployment. Deployment preserves the selected fork account, Worker, bucket, and existing published data.

Findings and repairs

The blind baseline and follow-up workflows reproduced defects in the server's query interface. The candidate repairs these without regenerating published data:

Failure or usability gap Repair and decisive check
A public SELECT read /etc/os-release from the server filesystem. Permit only the exact selected Parquet paths/URLs, disable other external reads and persistent secrets, and lock settings after trusted view construction. Local, nested, traversal, symlink, glob, and remote controls are retained.
EXPLAIN ANALYZE DELETE ... passed the outer EXPLAIN classification and executed its inner statement. Accept only DuckDB's SELECT statement type. A constructed local table lost its row before the fix and remains unchanged after it. All EXPLAIN forms are refused; supported read shorthands still work. No deployed table was modified in this experiment.
Duplicate result labels silently overwrote earlier values in JSON rows. Refuse duplicate labels with explicit AS-alias guidance. The real FCC join paired with an aliased control demonstrates both distinct status values.
Unknown tables, invalid row caps and refused writes reported MCP success. Raise tool errors and declare row-cap bounds in the input schema. HTTP checks require isError: true.
Result caps silently omitted rows. Fetch at most one extra row and report truncated only when a row was omitted. Equal-to-cap and over-cap cases are paired.
The FR docket-link declaration joined only on document number, despite reused numbers on different dates. Declare both document number and publication date. The live composite-key baseline measured 603,935 distinct keys with zero missing parents. C2-15835 supplies the retained cross-date counterexample.
Empty machine-readable join arrays implied less connectivity than the column descriptions provide. Explain that the measured join declarations are not an exhaustive relationship inventory, including JSON-array joins. No unvalidated links were added.

The independent review caught the EXPLAIN issue after the initial candidate; its findings and closure checks remain under ~/.codex/artifacts/spicy-regs-mcp-review-20260927/. Two implementation failures also remain in the evidence: spill configuration must precede external-access restriction, and persistent-secret configuration must precede httpfs view binding. The built Linux container exposed the second ordering issue. Its repaired setup was tested against real HTTPS Parquet, not just mocks.

Research coverage and interpretation

The complete table guide maps every advertised table to a use and a limitation. The frozen inventory measured 91 readable tables: 85 populated and six empty, with matching declared schemas and unchanged observed publication pins throughout. Raw schema/count responses and the full research matrix are under ~/.codex/artifacts/spicy-regs-full-data-inventory-20260927/. Counts establish availability at that time, not completeness or current source qualification.

The social-data experiment completed five bounded workflows: FCC participation, comment opportunities, lobbying disclosure, vote-to-term attribution and FEC collection transparency. They produced useful descriptive results, while exposing limits that queries must retain. Filings are not people, disclosed organizations are not resolved identities, source mixtures affect comment-window comparisons, and absent collections do not prove empty source populations.

The cross-product comparisons used frozen RefSpec and SpicyDocs inputs alongside live SpicyRegs results. Source-scoped agency IDs resolved more mentions than exact raw-name equality in the selected diagnostic. RefSpec's co-occurrence rankings could map a bureau to its parent and must not be treated as entity identity. Unique vocabulary variants did not improve the fixed test queries; unemployment remained ambiguous. No heuristic was promoted into an authoritative crosswalk.

Iceberg decision

Keep Iceberg as the comments ingestion/update store and published Parquet as the public MCP read surface. The deployed fork had no active catalog configuration. A read-only comparison found catalog table UUID 01a0d3fc-be18-70b0-abeb-e0f6e58d6cff, snapshot 3869492088423596413 and schema 0 identical to the comments mirror's source receipt. Both public file ETags and sizes matched that receipt, whose comments population is 26,314,331. This is snapshot/receipt/object agreement, not an independent full-content equivalence scan. The catalog comparison is retained under ~/.codex/artifacts/spicy-regs-catalog-decision-20260927/.

Direct catalog access showed no freshness benefit in this measurement. Its dynamic manifests, object paths and credentials have not been qualified against the public server's exact-file restrictions. The MCP therefore explicitly refuses complete direct-catalog configuration before network setup; its old optional attach branch was removed. ETL catalog ownership is unchanged.

Revisit direct reads if measured mirror lag, time-travel needs or query cost justify them, and after a restricted, explicitly snapshot-pinned implementation passes the same controls. A nearer provenance improvement is to expose the comments publication receipt and publish immutable comments generations: the current comments URLs remain mutable and non-atomic, unlike managed family pins.

Local validation

The final candidate passed the repository's normal test selection: 3,150 passed, two skipped and four integration cases deselected. Ruff, ty, dictionary checks, and generated-table-page comparison passed. Cloudflare's type check and deployment dry run passed; the final Linux image additionally passed actual HTTP tool calls against the public data and a forced spill with external access disabled.

The Linux HTTP harness initially used the wrong field names for the returned join metadata. Its server response was correct; the corrected assertion was applied to the retained response without repeating requests. That harness error and the two earlier setup errors remain visible in the repair evidence.

Deployed result

The authorized deployment targets https://spicy-regs-mcp.mdeeb.workers.dev/mcp, using the existing fork bucket and Cloudflare account. Worker version b085691b-f5df-44e4-8cb9-05b60410d714 deployed container image sha256:82ea85c51b9decad1ae682f3f4da6e4dbe6f600c98f4187c0c49bd0921566629. The application advanced to version 9, cleared its active rollout, and reported no container health errors. A request made during replacement still reached the old schema; that observation is retained separately. As Cloudflare's rollout documentation explains, deploy success starts replacement rather than proving completion.

After replacement, all 18 runtime assertions passed against the public endpoint: row-cap schema, discovery of every advertised view, the original OS-file read, nested file read, unselected HTTPS read, duplicate labels, unknown table, three invalid row caps, direct and EXPLAIN-wrapped write refusals, paired result caps, aliased values, real agency and comments reads, and the composite FR join metadata. Raw requests/responses are under the repair directory's deployed-after-rollout/; verification.json records the result. The harness counts related assertions together, not one assertion per sentence above.

The source/build SHA-256 manifest and tracked patch were captured before publication and checked unchanged after deployment. The prior Worker version bb1fb318-292d-403f-9370-f4a599c94182 and prior image digest are retained in worker-before.json and containers-before.json for operational recovery. This delivery changes the MCP service; it does not certify or republish source datasets, publish documentation, or establish full sandbox security beyond the retained bounded cases.

The independent social-data agent then used its four reserved production calls. All passed: the original FCC duplicate-label query is refused with alias guidance, the aliased query preserves both statuses and exactly matches its three baseline rows, and truncation is true only when the third row is omitted. The FCC proceedings publication pin changed between the original experiment and this retest; the FCC filings pin and selected control rows did not. Both pins are retained in spicy-regs-social-chaos-20260927/retest-assessment.json. The agent consumed its preregistered 35 data/description calls and stopped.