Introducing Company Agent Bench
Joshua Qin and Phil Chen · 12 November 2026
Read the originalConcept, a research lab, tested how well four AI agent products understand a company from its chat history, and the best answered 75.9% of 205 questions correctly.
Joshua Qin and Phil Chen of Concept, an applied research lab, built a test from one month of the lab's own messages in Slack, the workplace chat product. The test asks an AI agent who wrote a message, who was tagged in it, and who wrote each reply. The authors argue that an agent can complete tasks well and still not know its colleagues' roles, priorities and working relationships.
- The 205 questions cover about 10 teammates. Each agent searched a copy of the Slack workspace that held only the messages sent before the answer appeared.
- Four AI agent products were tested at their default settings. ChatGPT Dots scored 75.9%, Grok Bot 72.7%, Claude Tag 67.8% and Devin 66.1%.
- The products recognised who owns a responsibility more often than what a person cares about. On questions about who wrote a message, they scored 73% to 91% on the first kind and 42% to 54% on the second.
- Several AI models given only tools to search and read Slack outscored the products. The best, Claude Fable 5.2, scored 82.2%.
A team that adds an AI agent to its chat tool can expect the agent to know who owns what better than what each person cares about. That advice is an inference and is not the authors' claim. The test covers one small company for one month, and Concept is building its own technology in this field.
17 November 2026: as AI agents and staff outside engineering produce the stories, specifications and tools that others build on, a team must name who checks them and who looks after them.