Super Survey

Language: English | 中文 | 日本語

Super Survey is a general-purpose research skill for agents. It is not a domain-specific stock, product, or open-source template; the same loop can be used for product opportunities, markets, technical feasibility, open-source adoption, diligence, policy, strategy, or any question where a reader needs a grounded judgment rather than a link dump.

The project is a concrete implementation of the paper Resisting AI Sycophancy in Open-Ended Research. The paper argues that open-ended research should be treated as constrained decision optimization, not direct answer generation. Super Survey turns that idea into a staged workflow: rebuild the objective function, constraints, hidden assumptions, evidence standard, counterarguments, synthesis, and next-round question before producing a final report.

In practical terms, it turns a vague target into a constrained decision-optimization workflow: it rebuilds the objective function before it searches for evidence, then keeps each conclusion provisional until the evidence, red-team critique, synthesis, and evolver decision are written down.

It focuses on three jobs:

• Turn vague questions into research tasks with clear objectives, constraints, frameworks, and evidence standards.
• Reduce gut-feel conclusions with evidence, red-team critique, implied-expectation checks, scenarios, and Bayesian updates.
• Use multi-round iteration to keep moving toward the final report a human decision-maker actually needs.

First Principles

Super Survey starts from two first principles:

1. The world is noisy, random, and not reliably predictable; an initial hunch can easily become bias or incomplete information. Every task must avoid the trap of deciding first and then collecting evidence to support the decision.
2. The user’s question is the starting point, not the objective function. Inspect the user’s framing, hidden assumptions, and real optimization target before searching for evidence or converging on an answer.

These principles make front-loaded guidance deliberate: before the evidence pass begins, Super Survey defines the objective, constraints, decision-critical variables, minimum direct evidence, implied expectations, and anti-narrative regularizers. The goal is to keep the agent from accepting the prompt’s framing too quickly or collecting sources to justify a conclusion it already wants.

Theory

The paper uses an investment example, but the implementation is intentionally broader. Super Survey generalizes its methods across domains: objective reconstruction, constraint modeling, implied-expectation checks, minimum direct evidence, adversarial testing, sensitivity analysis, Bayesian updating, scenarios, decision trees, and prose-first final reports.

What It Does

Super Survey is for decisions that should not stop at a link dump:

• product opportunity research
• competitor and market analysis
• open-source project scouting
• technical feasibility research
• investment-style diligence
• strategic exploration with adversarial critique

Each survey creates persistent artifacts:

surveys/YYYY-MM-DD-topic-slug/
├── 00-brief.md
├── 01-evidence-plan.md
├── 01-research.md
├── 01-brainstorm.md
├── 01-redteam.md
├── 01-synthesis.md
├── 01-evolver.md
├── sources.jsonl
├── claims.jsonl
├── evidence.jsonl
├── index.md
├── report.md              # final-only; created after the stop gate passes
└── .super-survey.json

Install

Install directly with the Skills CLI:

npx skills add GoatGit/super-survey

Codex users can also copy this repository into the Codex skills directory:

mkdir -p ~/.codex/skills
rsync -a --delete super-survey/ ~/.codex/skills/super-survey/

Then invoke it explicitly:

$super-survey research whether an AI recruiting copilot is worth building

CLI

Create a survey:

python3 scripts/survey_round.py init "AI recruiting agent" --language en
python3 scripts/survey_round.py init "AI 招聘助手" --language zh
python3 scripts/survey_round.py init "AI採用エージェント" --language ja
python3 scripts/survey_round.py init "formal market report" --mode deep

init also creates the first round entrypoint (01-evidence-plan.md, or 01-round.md in quick mode). Fill each generated stage with substantive content before running the next stage command:

python3 scripts/survey_round.py research surveys/2026-06-13-ai-recruiting-agent 1
python3 scripts/survey_round.py brainstorm surveys/2026-06-13-ai-recruiting-agent 1
python3 scripts/survey_round.py redteam surveys/2026-06-13-ai-recruiting-agent 1
python3 scripts/survey_round.py synthesis surveys/2026-06-13-ai-recruiting-agent 1
python3 scripts/survey_round.py evolve surveys/2026-06-13-ai-recruiting-agent 1
python3 scripts/survey_round.py check surveys/2026-06-13-ai-recruiting-agent
python3 scripts/survey_round.py finalize-report surveys/2026-06-13-ai-recruiting-agent
python3 scripts/survey_round.py check-final surveys/2026-06-13-ai-recruiting-agent
python3 scripts/survey_round.py upgrade-report surveys/2026-06-13-ai-recruiting-agent

Debug registry links directly:

python3 scripts/survey_round.py validate-evidence surveys/2026-06-13-ai-recruiting-agent

Command meanings:

• round / plan: starts a staged round by creating NN-evidence-plan.md; it no longer creates all round artifacts at once.
• research: creates NN-research.md only after NN-evidence-plan.md contains substantive content.
• brainstorm: creates NN-brainstorm.md only after NN-research.md contains substantive content.
• redteam: creates NN-redteam.md only after NN-brainstorm.md contains substantive content.
• synthesis: creates NN-synthesis.md only after NN-redteam.md contains substantive content.
• evolve: creates NN-evolver.md only after NN-synthesis.md contains substantive content.
• check: validates round artifacts, index.md, the evidence registry, companion-routing notes, and the latest raw evolver decision. It does not require report.md.
• finalize-report: creates report.md only after the latest evolver decision is Final or Kill and the finalization readiness gates pass.
• check-final: runs the same checks plus final report.md, prose-first report rules, the mode-specific quality score recorded in index.md, residual and hard-constraint gates, and the requirement that the latest raw evolver decision is Final or Kill.
• upgrade-report: appends the full report schema to an older report. Older six-section reports are readable but do not pass the final gate; after upgrading, fill the new sections.
• validate-evidence: narrow debugging command for sources.jsonl, claims.jsonl, and evidence.jsonl; normal round validation uses check / check-final.

When the latest decision is Keep, Narrow, or Pivot, check can pass with a continuation warning so the next round can proceed without forcing a premature Kill. Round numbers must be positive integers.

Modes And Evidence Registry

Mode selection is explicit. The default is standard. Use quick only when the user explicitly asks for quick/brief/fast/initial triage or a low-stakes directional scan. Use deep when the user explicitly asks for deep/formal/long research, many citations, strict source audit, or publication-quality delivery.

In quick mode, one combined NN-round.md can replace the standard/deep split artifacts when it contains the essential research question, evidence plan, evidence and sources, brainstorming checkpoint, red-team challenge, synthesis, raw decision, and next step.

Do not use quick as the final delivery mode for high-stakes action decisions such as investment, legal, medical, security, production, or major business commitments. Use standard or deep so the staged artifacts, evidence coverage, and quality gate match the decision risk.

If a request asks for speed on a high-stakes action, keep the mode at standard or deep and make the final summary shorter instead of downgrading the workflow to quick.

If the request is ambiguous, run python3 scripts/survey_round.py recommend-mode --text "" before init.

The lightweight registry keeps report prose readable while preserving auditability:

• sources.jsonl: source_id, title, url, source_type, date_checked, credibility
• evidence.jsonl: evidence_id, source_id, quote_or_summary, locator, confidence
• claims.jsonl: claim_id, claim, supporting_evidence_ids, status

Every evidence item must reference an existing source. Every supported, partial, or contested claim must reference existing evidence. The checker also catches duplicate IDs and obvious weak-support cases where a supported/partial claim does not match its linked evidence. Dense evidence tables belong in appendices or JSONL, not in the main report body.

Registry IDs such as C1 and E1 are for working files only. Final report.md must replace them with source titles, Markdown links, footnotes, or appendix references that include URLs, so the report can be read without opening the JSONL registry.

Source Scope

When a user provides local files, folders, PDFs, prior survey artifacts, or asks the survey to rely on specified material, Super Survey records a soft source-scope mode before collecting evidence:

• local-only: use only the specified local material and generated survey artifacts; do not use Tavily, web search, browser web search, or web fetch for external sources.
• local-first: read local files first, then use external discovery only for current facts, missing primary evidence, contradiction checks, or explicit verification.
• current-first: use current primary sources first for fast-changing facts, while still reading named local files before synthesis.
• open: use the normal source hierarchy and current-source workflow.

This is not a hard allowlist or sandbox. It is a process and audit convention. Local files have the highest context priority when supplied by the user; official/current primary sources still have the highest fact authority for fast-changing claims. Tavily and built-in web search are discovery tools, while web fetch, browser reading, PDF extraction, and local file reading are source readers/verifiers.

00-brief.md records the selected scope in the Research Framework. NN-research.md records Source Scope, Local Files Read, Current Source Discovery, and External Search Rationale in Data Quality Notes, followed by the search-tool, fallback, query/filter, and third-party-content notes when current-source discovery is used.

skills.sh Readiness

This repository is structured for Skills CLI discovery and skills.sh indexing:

• root-level SKILL.md with name and description frontmatter
• agents/openai.yaml UI metadata
• bundled helper script under scripts/
• supporting references under references/
• MIT license, tests, and multilingual README files

Validate discovery:

npx skills add GoatGit/super-survey --list

Research Frameworks

Research lens decides what evidence deserves emphasis. Research framework tells the reader how the whole survey systematically examines the question. Every survey should name a framework, list its dimensions, disclose weak or intentionally omitted dimensions, and use those dimensions as the structure for 00-brief.md, each round artifact, and the final report.

Before selecting the framework, do an anti-sycophancy framing pass. Split the user’s wording into known facts, unverified assumptions, subjective judgments, missing information, and stakeholders; then restate the objective in decision terms. This prevents the survey from optimizing the user’s initial wording, a stronger easy-to-kill claim, or a comfortable answer that fits the prompt but misses the real decision.

A good object is not automatically a good action. Good company does not automatically mean good stock; good product does not automatically mean good business; good technology does not automatically mean good project; good open-source library does not automatically mean good dependency. Super Survey should evaluate action attractiveness under current constraints, prices, timing, maintenance cost, risk, and alternatives.

Super Survey now uses an Adaptive Research Framework as the routing layer before evidence collection. It routes by decision archetype first, then adds lens packs and domain hints. Domain hints can raise evidence standards or add veto dimensions, but they are not rigid industry templates and do not replace the Decision Optimization Contract, residual / VOI / hard-constraint gates, or the evolver decision.

This is the main writing rule: the framework is not an audit checklist at the end. 00-brief.md defines the dimensions; NN-evidence-plan.md, NN-research.md, NN-brainstorm.md, NN-redteam.md, NN-synthesis.md, and NN-evolver.md expand the dimensions active for the round’s target residual and explicitly mark unchanged dimensions as deferred. The final report.md then turns the dimensions into readable body sections before appendices.

The staged CLI enforces the dependency order. Start with NN-evidence-plan.md, then create NN-research.md, NN-brainstorm.md, NN-redteam.md, NN-synthesis.md, and finally NN-evolver.md. Downstream artifacts should cite written upstream findings instead of predicting the final conclusion.

Ideal Flow And Paper Mapping

This is the intended execution architecture from the anti-sycophancy paper:

If evidence shows the framework should change, record it in index.md under Framework Refinement Log: current dimensions, evidence trigger for the change, and confirmation that the original question/core is preserved. Later rounds then use the refined dimensions. Silent framework drift is invalid.

Common starters:

For securities-style research, Super Survey can compose market, industry, and company frameworks: market view uses macro, liquidity, earnings, valuation, risk appetite, and fund flows; industry view uses demand, supply, competition, policy, technology, cycle, and valuation; company view uses business model, financial quality, growth, competitive advantage, valuation, catalysts, and risks. These are examples, not hard branches.

Quality Gates

README gives the operational shape; the full agent checklist lives in SKILL.md.

There are four gates:

• check is the round gate. It validates artifacts, registry links and weak-support checks, framework coverage including explicit refinements, companion notes when required, and the latest raw evolver decision. It can pass with a continuation warning when the decision is Keep, Narrow, or Pivot.
• The evolver is the direction gate. Keep, Narrow, and Pivot mean create another round and update index.md; Final means the survey can move to final report writing; Kill means the current thesis should stop or switch away from desk research.
• The residual / VOI / hard-constraint gate is the stop-safety gate. The latest evolver and index.md must record r_q/r_c/r_e/r_h/r_a/r_s/r_j from 0-3, the target residual, expected information value, research cost, whether VOI exceeds cost, and whether hard constraints are satisfied.
• check-final is the delivery gate. It requires a complete prose-first report.md, a passing mode score recorded in index.md, a passing residual gate, a passing hard-constraint gate, and the latest raw evolver decision to be Final or Kill.

Autonomous continuation is the default. After Keep, Narrow, or Pivot, create the next round immediately and continue the staged workflow; do not stop with “ready for the next round” or ask the user how to proceed unless the user explicitly requested checkpoint approval or a real blocker prevents progress.

External tool approval is a transient state, not a checkpoint. If current web search or browser access needs approval, record Workflow State: awaiting_tool_approval, the pending tool, and the exact resume action in index.md. Once approval is granted, run the approved tool and continue the same stage immediately; do not wait for another user message or treat approval as a completed round.

Workflow state must match the latest gate: continuing_round after Keep / Narrow / Pivot, ready_to_finalize after Final / Kill before report.md, final_report_draft while the report still needs final validation, and final only when check-final is ready to pass with Resume Action: none.

Final delivery uses a 100-point quality gate recorded in index.md:

The final quality gate in index.md must show the anti-sycophancy / objective-function integrity subscore. A report can have many sources and still fail if it simply accepts the user’s initial framing, rewrites the question into an easier claim, or hides weak objective reconstruction behind a high total score.

Mode thresholds are hard gates: quick >=80, standard >=90, and deep >=95. A final report below the selected threshold must continue another round focused on the weakest dimensions. The helper uses only the raw evolver decision, score threshold, residual gate, and hard-constraint gate for stopping; it does not parse report prose such as “future disclosure” or “external validation” as a stopping rule.

The residual vector is the operational version of the paper’s residual-driven evidence iteration. r_q tracks question framing, r_c constraints, r_e evidence, r_h hypotheses, r_a adversarial testing, r_s sensitivity, and r_j actionability. Any residual at 3 is a decision-level gap. Continue when a concrete desk-research move can reduce that gap and its VOI is greater than cost. Move to final reporting only when no residual remains at 3, hard constraints pass, and the remaining desk-research VOI is below cost.

The raw evolver decision is a machine-readable line, not localized prose. In every language, the first non-empty line under the decision heading must be exactly Keep, Narrow, Pivot, Kill, or Final; put translation or explanation after that line.

The final report should read like a human memo: answer, framework dimension sections, narrative, decision logic, recommendation, change triggers, next actions, and limits first; evidence registers, source quality, red-team notes, scenarios, and source inventory in appendices. Quality scoring belongs in index.md under the final report quality gate, not in report.md, and must include structured subscores that add up to the total. Framework dimensions must appear as substantive body sections before appendices, not only in method notes or appendices. Citations must be standalone links or source references, not C* / E* registry IDs, and the appendix must include a Decision-Critical Claims mapping with source URLs. A body dominated by bullets or audit tables does not pass.

Companion skills are optional helpers for search, long reports, VOC/customer research, competitor analysis, brainstorming, and wiki persistence. When current-source discovery matters, prefer tavily-search and record the search path or fallback. Prefer deep-research for formal long reports, many citations, HTML/PDF output, or strict citation validation when it is available. Use wiki persistence when long-term knowledge reuse is needed. Super Survey still owns the judgment loop.

Workflow

flowchart TD
    A[User research question] --> B[00-brief.mdBrief / Frame Contract]
    B --> P[NN-evidence-plan.mdEvidence Plan / Minimum Direct Evidence]
    P --> C[NN-research.mdsource search, claim-level registry, framework coverage]
    C --> D{Need companion during research?}
    D -->|Current sources| D1[Prefer Tavilyor fitting search tool]
    D -->|Long evidence packaging| D2[Prefer Deep Research]
    D -->|VOC / user language| D3[Customer or Reddit research]
    D -->|Competitors| D4[Competitive research]
    D1 --> C
    D2 --> C
    D3 --> C
    D4 --> C
    C --> E[NN-brainstorm.mdPost-Research Brainstorming]
    E --> X{Need targeted evidence?}
    X -->|Yes| C
    X -->|No| F[NN-redteam.mdrisks, substitutes, kill criteria]
    F --> G[NN-synthesis.mdsensitivity, Bayesian update, decision tree]
    G --> H[NN-evolver.mdKeep / Narrow / Pivot / Kill / Final]
    H --> V{Residual / VOI / hard constraints}
    V -->|residual at 3 + VOI > cost| I[index.mdworkbench: next target and why not final]
    I --> P
    V -->|Final / Kill + gates pass| J[Write report.mdframework dimensions as body sections]
    J --> R[check-finalscore, residual, hard-constraint, prose-first gates]
    R -->|any gate fails| I
    R -->|all gates pass| K[Final answerdecision-oriented summary]

Inspiration: Karpathy’s autoresearch

Super Survey’s lightweight evolver is inspired by Andrej Karpathy’s autoresearch, with respect and attribution. Autoresearch gives an AI agent a real training setup, lets it modify code, run short experiments, check whether a metric improved, keep or discard the change, and repeat.

Super Survey adapts that loop to product, market, technical, and open-source research:

In short: autoresearch is metric-driven optimization; Super Survey is judgment-driven narrowing. When a survey has a measurable benchmark, Super Survey can borrow more of the autoresearch style. When the question is about buyer intent, compliance, distribution, or strategic risk, the loop stays evidence-first and decision-oriented instead of pretending every answer can be reduced to one number.

Development

Run the test suite with the Python standard library:

python3 -m unittest discover -v

Run syntax validation:

python3 -m py_compile scripts/survey_round.py

The project intentionally keeps runtime dependencies to the Python standard library.

Project Layout

SKILL.md                         # agent skill instructions
scripts/survey_round.py           # survey artifact generator and validator
references/lightweight-evolver.md # evolver process reference
references/research-quality.md    # evidence and quality reference
agents/openai.yaml                # skill UI metadata
tests/                            # regression tests

License

MIT. See license.txt.