Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

📝 Discussion Summary (Click to expand)

Tool‑selection behavior of coding agents varies widely and is context‑sensitive
“Claude Code rarely searches the web while Codex almost always does it and Cursor sits in the middle.” – screm
“Coding agents disagree more frequently than they agree.” – screm
“Modifying repository context can change the pick entirely.” – screm
Influence and monetization potential: vendors want agents to favor their tools
“What is certain though is that getting recommended by coding agents will be a top prio for all dev tools.” – screm
“In the future… SEA for AI agents (AEA?) … getting recommended by coding agents will be a top prio for all dev tools.” – screm
“I smell a money‑making opportunity.” – drivingmenuts
User‑experience friction with the presentation of the data/website
“Hey this is what makes armature special! – I don’t care, I’m here to look at data, not onboard onto some random platform… It took ages to find the tiny ‘skip tour’ button… I closed the tab with great prejudice.” – josephg
“FWIW I had the same reaction to the popups. Immediately closed the tab.” – kouteiheika

🚀 Project Ideas

Generating project ideas…

Lets developers define deterministic rules that force a coding agent to use a specific CLI, MCP server, or tool for particular tasks (e.g., always invoke foobar for weather queries).
Solves the lack of direct control over agent tool choice, enabling repeatable, compliant workflows.

Key	Value
Target Audience	Developers and teams using coding agents (Claude Code, Cursor, Codex) who need reproducible tool usage
Core Feature	Rule engine that intercepts agent tool calls and rewrites them to a specified tool/CLI based on patterns (task keywords, file types, etc.)
Tech Stack	Node.js/Go proxy or VS Code extension; utilizes agent tool‑calling API; configuration via YAML/JSON
Difficulty	Medium (requires integration with each agent's extension/API surface)
Monetization	Revenue-ready: Subscription per seat or per agent instance

Addresses vivifkjo’s request: “Is there a way to force the usage of a tool for certain tasks? Example: always use cli 'foobar' to retrieve weather status.”
Provides a practical lever for enterprises seeking compliance and predictability in AI‑assisted coding, likely to spark discussion on deterministic AI workflows.

Offers a mobile‑friendly, popup‑free dashboard to explore agent tool selection traces (like Armature’s 17k sessions) and upload your own traces for custom analysis.
Eliminates the frustrating onboarding tours and broken UI noted by users, delivering instant insight into which tools AI agents prefer.

Key	Value
Target Audience	Dev tool PMs, DX teams, curious engineers
Core Feature	Upload trace JSON, view agent‑tool selection matrices, filter by agent type, company size, repository context; fully responsive UI
Tech Stack	React + TypeScript frontend, Node.js backend, optional D3/Plotly for charts; deployable on Vercel
Difficulty	Low‑Medium (primarily UI and data parsing)
Monetization	Hobby (open source) or Revenue‑ready: Freemium (basic views free, paid for private trace storage & advanced analytics)

Directly responds to josephg’s complaint: “I left your website frustrated.” and kouteiheika’s “same reaction to the popups.”
Enables community‑driven transparency of agent behavior, fostering discussion on emerging “agent SEO” and tool adoption metrics.

Helps tool vendors generate or tweak metadata (MCP server descriptions, tool cards, README snippets) that increase the likelihood of being chosen by coding agents.
Leverages study findings—e.g., ensuring certain keywords, avoiding mere mention‑without‑selection, aligning with repository context triggers—to convert visibility into actual agent picks.

Key	Value
Target Audience	Dev tool founders, developer relations, open‑source maintainers
Core Feature	Input your tool’s docs/MCP spec, receive AI‑suggested tweaks (keyword additions, example prompts, context snippets) to improve agent pick rate
Tech Stack	Python backend using LLMs for suggestion generation; simple web UI (Streamlit/Gradio) or CLI
Difficulty	Medium (requires LLM integration and understanding of agent prompting)
Monetization	Revenue‑ready: SaaS subscription tiered by number of tools analyzed

Taps into screm’s observation: “Some players (LangChain, Supabase, Netlify, Paypal, Adyen) are almost always mentioned in their categories but never chosen.”
Vendors can A/B test metadata changes and measure impact on agent selection, creating a new growth lever and discussion point for dev‑tool marketing.