FirstResearch: Auditable Question Formation for LLM Scientific Discovery Agents
Yufeng Wang
Why It Matters
What makes this one worth your time
This work addresses the critical issue of transparency in LLM-generated research questions, which is essential for ensuring scientific rigor and accountability in AI-assisted research.
FirstResearch enhances the auditability of LLM-generated scientific questions.
Summary
The paper introduces FirstResearch, a framework for forming auditable research questions for LLMs in scientific discovery, featuring a structured Research Question Certificate that enhances the inspectability of proposed questions.
Key contributions
- Introduction of the FirstResearch framework for structured research question formation.
- Development of the Research Question Certificate that records essential components for auditability.
- Empirical evaluation showing performance improvements over baseline LLM agents.
Notable insights
- The structured Research Question Certificate includes various components that enhance the auditability of LLM-generated questions.
- The framework demonstrates significant performance improvements over existing baselines, indicating the effectiveness of explicit derivation constraints.
Possible limitations
- Preliminary results rely on LLM judges rather than human domain experts, which may affect the validity of the findings.
- The abstract does not address potential scalability issues or the applicability of the framework across diverse scientific domains.
Abstract
arXiv:2607.05682v1 Announce Type: new Abstract: LLM systems for scientific discovery increasingly assist with ideation, literature synthesis, experiment planning, and report generation, but the first research question they propose can remain difficult to audit: it may sound plausible without exposing the mechanism, falsifier, or assumption that a scientist should inspect. We introduce FirstResearch, a first-principles research-question formation framework for scientific LLM agents whose core artifact is a structured Research Question Certificate. The certificate records primitive definitions, assumptions, a mechanism model, a tension or contradiction, a falsifiable hypothesis, a minimal decisive test, and a failure update rule, making the proposed question inspectable before downstream execution. On ten LLM-agent research topics, FirstResearch outperforms controlled prompt-level baselines inspired by AI co-scientist, Agent Laboratory, and AI Scientist-v2 under a primary DeepSeek-blind-judge protocol. A Gemini-2.5-Flash independent-judge rescore of the same 40 baseline packages preserves the system-level ranking, with FirstResearch scoring 4.86/5 versus 4.38/5 for the strongest baseline and Pearson agreement of 0.865 on average score. A one-repeat ablation checkpoint further suggests that the certificate-centered core is the strongest component: certificate-only scoring reaches 4.90/5 under DeepSeek and 4.88/5 under Gemini, while removing certificates drops below 1/5 under both judges. These results are preliminary and use LLM judges rather than human domain experts, but they support a narrow scientific-discovery claim: explicit derivation constraints are a promising mechanism for making LLM-generated scientific questions more auditable. Code, prompts, saved outputs, and reproduction scripts are available at https://github.com/louiswang524/FirstResearch.