How Anthropic Enables Self-Service Analytics

"As many data science and data engineering teams can attest, enabling self-service business analytics has traditionally been a slog."

"很多数据科学和数据工程团队都会认同:让业务方能自助做分析,历来是一场苦战。"

"Making the data model more accessible to less technical coworkers via wide and denormalized tables often leads to overlapping views with inconsistent definitions as the business scales (and does little to bridge the gap for employees with little desire to learn SQL)."

"为了让技术不那么强的同事更容易上手,而用宽表、反范式(denormalized)的表来暴露数据模型,随着业务扩张,往往会催生出一堆口径不一致、彼此重叠的视图(而且对那些压根不想学 SQL 的员工,几乎起不到弥合作用)。"

"Alternatively, creating more ringfenced environments for users often misses the long tail of business questions and leads to metric and dashboard bloat as teams silo their work."

"另一条路——为用户搭建更"围栏化"(ringfenced)的环境——又常常覆盖不到长尾的业务问题,并且随着各团队各自为政,导致指标和看板(dashboard)泛滥膨胀。"

"The rise of LLMs provides an additional path for self-service analytics that avoids those challenges. However, pointing Claude at a warehouse and letting the agents execute can create a false sense of precision."

"LLM 的兴起,为自助式分析提供了一条能绕开这些难题的新路径。然而,把 Claude 指向一个数据仓库、让 agent 直接执行,会制造出一种"精确"的假象。"

"The initial elation of liberation from ad-hoc requests turns into dread with the realization that this setup separates stakeholders from the underlying infrastructure, documentation, and expertise that previously steered them toward carefully curated datasets."

"一开始,从应付临时取数请求中解放出来的喜悦,会变成恐惧——当你意识到:这种做法把业务方(stakeholder)和底层的基础设施、文档与专业知识隔开了,而正是这些东西,过去把他们引向那些精心整理过的数据集。"

"At Anthropic, 95% of business analytics queries are automated via Claude, with ~95% accuracy in aggregate. By giving this often rote, repetitive work to Claude, our data science team can focus on more strategic work like causal modeling, forecasting, and machine learning."

"在 Anthropic,95% 的业务分析查询通过 Claude 实现了自动化,整体准确率约 95%。把这些往往枯燥、重复的活儿交给 Claude 后,我们的数据科学团队就能聚焦于更具战略性的工作,比如因果建模(causal modeling)、预测和机器学习。"

"After meeting with dozens of Anthropic's top Claude Code users and having seen myriad design patterns for analytics agents, we've cultivated some best practices for other data teams working with LLMs."

"在与几十位 Anthropic 顶尖 Claude Code 用户交流、并见识了形形色色的分析 agent 设计模式之后,我们为其他用 LLM 的数据团队总结出了一些最佳实践。"

"In this post, we'll share these tips and approaches to maximizing Claude's ability to drive self-serve business insights, including:"

"在这篇文章里,我们会分享这些把 Claude 自助洞察能力发挥到极致的技巧和方法,包括:"

"Why analytics accuracy is a context and verification problem, not a code generation issue; the three failure modes that cause most errors; the agentic analytics stack we built to address these errors; how we measure effectiveness; and a basic template for how we create the majority of our skills (see the appendix)."

"为什么分析的准确率是个上下文与验证问题、而非代码生成问题;造成大部分错误的三种失败模式;我们为应对这些错误而搭建的 agentic 分析栈;我们如何衡量效果;以及我们创建大多数 skill 所用的一个基础模板(见附录)。"

"LLMs' generative abilities are a double-edged sword: the mechanisms that enable creative solutions to complex problems can also hallucinate erroneous output. To fully understand the challenges with analytics agents, it's useful to compare them to coding agents."

"LLM 的生成能力是一把双刃剑:那些让它能为复杂问题想出创造性方案的机制,同样会幻觉(hallucinate)出错误的输出。要彻底理解分析 agent 的挑战,把它们和编码 agent 对比一下会很有帮助。"

"Coding is an open-ended solution space that rewards the models' creativity, while documentation and tests provide natural guardrails against hallucination."

"编码是一个开放式的解空间,奖励模型的创造力,同时文档和测试为对抗幻觉提供了天然的护栏(guardrail)。"

"In contrast, for analytics use cases, there's often only a single correct answer using a single correct source in which there's no deterministic way of proving the correctness."

"相比之下,在分析场景里,往往只有唯一一个正确答案、对应唯一一个正确来源,而且没有任何确定性的办法去证明它的正确性。"

"For self-service agentic business analytics, the complexity mainly lies in the ambiguity of the data. The central problem comes down to our ability to map a user's question to specific and up-to-date entities in our data model and know the correct way of working with them."

"对于自助式的 agentic 业务分析,复杂性主要在于数据的歧义。核心问题归结为一点:我们能不能把用户的问题映射到数据模型中具体且最新的实体(entity)上,并知道处理它们的正确方式。"

"If we can do that, then the resulting execution and SQL becomes trivial."

"只要能做到这一点,随之而来的执行和 SQL 编写就变得微不足道了。"

"We've identified three attributes of this problem that account for an overwhelming majority of inaccurate responses:"

"我们识别出了这个问题的三个特征,它们解释了绝大多数不准确的回答:"

"With hundreds of viable options in a data model (out of potentially millions of fields), the agent is unable to choose the correct fields that best answer a user's question. For example, in measuring the number of active users: what actions constitute being 'active'? Do you include fraudulent users? What lookback window do you use?"

"在一个数据模型里(可能多达数百万个字段),有数百个看似都说得通的选项,agent 没法挑出最能回答用户问题的那些正确字段。举个例子,统计活跃用户数时:哪些行为才算"活跃"?要不要把欺诈用户算进去?用多长的回溯窗口(lookback window)?"

"Data sources, business definitions, and schemas change constantly; assets and agent knowledge go stale and start returning subtly wrong answers."

"数据源、业务定义和 schema 一直在变;数据资产和 agent 的知识会过时,开始返回那种细微出错的答案。"

"The right information may actually be in the data model and properly annotated, but given the vastness of the search space, the agent simply doesn't find it."

"正确的信息其实可能就在数据模型里、而且标注得当,但鉴于搜索空间太过庞大,agent 干脆就没找到它。"

"At Anthropic, the main way we minimize these three errors is via our agentic data stack. Each layer exists primarily to attack one or more of these problems:"

"在 Anthropic,我们把这三类错误降到最低的主要办法,就是我们的 agentic 数据栈。每一层存在的首要目的,都是为了攻克其中一个或多个问题:"

"Entity ambiguity: data foundations and sources of truth shrink the space of plausible entities until there's a single governed answer. Staleness: maintenance and validation processes keep everything from rotting as the business changes. Retrieval failure: skills make sure the agent reliably finds and correctly uses that answer."

"实体歧义:数据地基(data foundations)和真相来源(sources of truth)把"看似合理的实体"的空间不断收窄,直到只剩一个受治理的标准答案。陈旧:维护和验证流程让一切在业务变化时不至于腐坏。检索失败:skills 确保 agent 能可靠地找到、并正确使用那个答案。"

"In this section, we'll discuss how we built each layer."

"在这一节里,我们会讲讲每一层是怎么搭起来的。"

"The most important aspect of ensuring analytics agents are accurate is via strong data foundations, which include the data models, transforms, tests, and tables in a data warehouse, along with the metadata describing them."

"确保分析 agent 准确,最重要的一环是打牢数据地基——它包括数据仓库里的数据模型、转换(transform)、测试和表,以及描述它们的元数据(metadata)。"

"Standard data engineering and data quality practices such as dimensional modeling, shift-left testing, freshness and completeness checks on critical pipelines all still apply (and we won't relitigate these)."

"标准的数据工程和数据质量实践——比如维度建模(dimensional modeling)、测试左移(shift-left testing)、对关键管道做新鲜度与完整性检查——依然全都适用(这些我们就不再赘述了)。"

"What does change is that the end user of your data model is no longer a data expert (e.g. data scientist), but rather agents acting on behalf of users with varying degrees of data expertise or understanding of the underlying infrastructure."

"真正变了的是:你数据模型的终端用户不再是数据专家(比如数据科学家),而是代表用户行事的 agent,而这些用户对数据的专业程度、对底层基础设施的理解,参差不齐。"

"This shift presents a challenge in that the results can't require the user to validate the underlying correctness simply because the end user doesn't know."

"这一转变带来一个挑战:结果不能要求用户去校验底层的正确性——原因很简单,终端用户根本不懂。"

"The data foundations layer is aimed primarily at ambiguity: if revenue, for example, resolves to one governed dataset instead of forty plausible candidates, the problem largely disappears before the agent ever has to search."

"数据地基这一层主要瞄准的是歧义:比如,如果"收入"能解析到唯一一个受治理的数据集、而不是四十个看似都说得通的候选项,那在 agent 还没开始搜索之前,这个问题就基本消失了。"

"It's also where the first staleness defense lives, since the same repo that defines the canonical models is the natural place to enforce that they stay current. We've seen a few practices work especially well:"

"它也是抵御陈旧的第一道防线所在,因为定义规范模型(canonical model)的那个仓库,正是强制让它们保持最新的天然位置。我们发现有几种做法效果特别好:"

"Create canonical datasets: By far the most common failure is that the agent can't map a concept ('revenue for product X') to the single correct table, column, and metric definition, usually because there are multiple plausible candidates with subtly different implementations."

"创建规范数据集:目前为止最常见的失败,是 agent 没法把一个概念("产品 X 的收入")映射到唯一正确的那张表、那一列、那个指标定义上,通常是因为存在多个看似合理、实现却有细微差别的候选项。"

"The fix is fewer, more heavily governed logical models: curate a small set of canonical, single source-of-truth datasets that are clearly owned, consumption-ready, and discoverable, then aggressively deprecate the near-duplicates."

"解法是:更少、治理更严的逻辑模型——精选出一小批规范的、单一真相来源(single source-of-truth)的数据集,它们归属清晰、开箱即用、易于发现,然后积极地把那些近似重复的版本废弃掉。"

"Physical rollups and caches still matter for cost and performance, but they should derive mechanically from the canonical models rather than living alongside them as alternatives. The goal is that when an agent searches for a concept, it finds a single governed answer."

"物理层的汇总表(rollup)和缓存对成本和性能仍然重要,但它们应当机械地从规范模型派生出来,而不是作为"备选项"和规范模型并列存在。目标是:当 agent 搜索一个概念时,它找到的是唯一一个受治理的答案。"

"Enforce your standards: We've found the foundations only hold if the canonical models and metric definitions are enforced by tooling (the agent is structurally routed to them first), by CI (changes that bypass them fail review), and by mandate (downstream teams build on the governed layer or explain why not)."

"强制执行你的标准:我们发现,只有当规范模型和指标定义被三件事强制约束时,地基才立得住——被工具约束(agent 在结构上被优先路由到它们)、被 CI 约束(绕开它们的改动通不过评审)、被制度约束(下游团队要么基于受治理的那一层来搭建,要么说明为什么不这么做)。"

"Governance without enforcement otherwise quickly decays back to the multiple candidates problem."

"否则,没有强制执行的治理,会很快退化回"多个候选项"那个老问题。"

"Colocate artifacts: Our main defense against constantly changing data models and business logic is colocation. Nearly all data code (i.e., modeling, semantic layer, reference docs, canonical dashboard definitions) lives in a single repo, with CI checks that protect cross-layer integrity."

"把产物放在一起(colocate):我们对抗"数据模型和业务逻辑不断变化"的主要手段,就是同地化(colocation)。几乎所有数据代码(即建模、语义层、参考文档、规范看板定义)都放在同一个仓库里,配上保护跨层完整性的 CI 检查。"

"If a modeling change would break a downstream dashboard or invalidate a documented metric, CI flags it and the fix ships in the same PR."

"如果一处建模改动会弄坏下游某个看板、或让某个已记录的指标失效,CI 会把它标出来,而修复就在同一个 PR 里一起发出去。"

"Treat metadata as a first-class product: Coding agents perform well partly because codebases are legible: READMEs, type signatures, docstrings, etc. Your warehouse can be just as legible, but only if column and table descriptions, canonical metric definitions, grain documentation, valid value ranges, lineage, ownership, and model tiering are maintained with the same rigor as the transformations themselves."

"把元数据当作一等产品:编码 agent 表现好,部分原因是代码库是"可读的"——README、类型签名、docstring 等等。你的数仓也可以同样可读,但前提是:列和表的描述、规范指标定义、粒度(grain)文档、有效取值范围、血缘(lineage)、归属、模型分级——这些要和转换逻辑本身一样严谨地维护。"

"While not a new insight, good governance provides critical context that helps the agent choose the right dataset."

"这虽算不上什么新洞见,但良好的治理提供了关键的上下文,帮助 agent 选对数据集。"

"If data foundations are the data warehouse itself, sources of truth are the reference surfaces the agent consults to navigate it."

"如果说数据地基就是数据仓库本身,那么"真相来源"就是 agent 用来在仓库里导航时所参考的那些"参照面"。"

"This layer reduces concept <> entity ambiguity and turns 'weekly active users' in a stakeholder's question into a specific, governed entity in your data model. Roughly in descending order of trust:"

"这一层减少的是概念↔实体歧义,把业务方问题里的"周活跃用户"变成你数据模型里一个具体、受治理的实体。大致按信任度从高到低排列:"

"The compiled metric and dimension definitions. If a question maps cleanly to a defined metric, the agent calls a function and gets one number, the same number every other surface in the company produces."

"也就是编译好的指标(metric)和维度(dimension)定义。如果一个问题能干净地映射到某个已定义的指标,agent 就调用一个函数、拿到一个数字——这个数字和公司里其他任何渠道算出来的都一样。"

"Our agents are structurally required (by skill instruction) to leverage the semantic layer first (see the appendix)."

"我们的 agent 在结构上被(通过 skill 指令)要求优先使用语义层(见附录)。"

"One idea we tried that didn't work: bootstrapping the semantic layer by having an LLM auto-generate metric definitions from raw tables and query logs."

"有一个我们试过但行不通的想法:让 LLM 从原始表和查询日志中自动生成指标定义,以此来给语义层做冷启动(bootstrap)。"

"It produced plausible-looking definitions that encoded the very ambiguities we were trying to eliminate, and was net-negative on our evals versus a smaller, human-curated layer. Therefore we recommend generating the documentation with Claude, but having a human own the definition."

"它生成了一批看似合理的定义,却恰恰把我们想消除的那些歧义编码了进去,在我们的评测里,它相比一个更小、由人工精选的语义层是净负面的。因此我们的建议是:用 Claude 来生成文档,但定义本身要由人来负责。"

"When the semantic layer doesn't cover a question, lineage and table ranking (based on number of references) let the agent reason about which upstream models feed a concept, which are deprecated, and which share grain."

"当语义层覆盖不到某个问题时,血缘(lineage)和表的排名(基于被引用次数)让 agent 能推断出:哪些上游模型在喂养某个概念、哪些已被废弃、哪些共享同一粒度。"

"This transforms 'I don't know the metric' into 'I know which governed model to aggregate from.' It's also the backbone of the freshness and provenance signals we surface in online validation below."

"这把"我不知道这个指标"变成了"我知道该从哪个受治理的模型去聚合"。它也是我们在下文"在线验证"里所呈现的新鲜度和溯源(provenance)信号的骨架。"

"Historical SQL from dashboards, notebooks, and prior analyses. Intuitively, this should be high-value: it's a record of every question already answered correctly."

"来自看板、notebook 和过往分析的历史 SQL。直觉上,这应该很有价值:它记录了每一个曾被正确回答过的问题。"

"In practice, we found that giving the agent raw retrieval access to thousands of prior queries moved accuracy by less than a point. Unstructured retrieval couldn't map a new question to the right precedent."

"但实践中我们发现,给 agent 提供对成千上万条历史查询的原始检索权限,准确率的变化还不到一个百分点。非结构化的检索没法把一个新问题映射到正确的先例上。"

"What does work is distilling that corpus into structured per-domain reference docs and reusable analysis patterns described in skills. Treat the query history as raw material for curation, not as a source of truth the agent reads directly."

"真正有效的,是把这些语料提炼成结构化的、按域(domain)组织的参考文档,以及写在 skill 里的可复用分析模式。把查询历史当作用来加工的原材料,而不是让 agent 直接去读的真相来源。"

"The layer most teams skip, and the one we underrated the longest. An agent that doesn't understand your business will answer what the user asked, but not what they meant."

"这是大多数团队会跳过的一层,也是我们低估得最久的一层。一个不懂你业务的 agent,回答的是用户"问了什么",而不是用户"想问什么"。"

"It won't know that 'the Q2 launch' refers to a specific product, that two teams define the same term differently, or that a question is being asked because a board meeting is on Thursday."

"它不会知道"Q2 那次发布"指的是某个具体产品,不会知道两个团队对同一个术语定义不同,也不会知道某个问题之所以被问,是因为周四有个董事会。"

"We pipe in a company knowledge graph consisting of indexed docs, roadmaps, decision logs, and our organizational structure so the agent can resolve ambient references and ask better clarifying questions."

"我们灌入一张公司知识图谱,由索引过的文档、路线图、决策日志和我们的组织架构组成,这样 agent 就能解析那些"环境性"的指代,并提出更好的澄清式问题。"

"The common failure pattern across all four is the same one from the data foundations layer: poor or stale documentation."

"这四者共同的失败模式,和数据地基那一层是同一个:文档质量差或过时。"

"Claude is exceptionally useful for closing the gap (drafting column descriptions, proposing metric docs from query patterns, flagging undocumented models in CI), but the curation and ownership are managed by humans."

"Claude 在弥合这道缺口上格外有用(起草列描述、从查询模式中提出指标文档、在 CI 里标出无文档的模型),但精选和归属是由人来管理的。"

"In the next two sections, we discuss how to make that ownership cheap enough that it actually happens."

"在接下来的两节里,我们会讨论:怎么把这种"归属"做得足够便宜,让它真的能发生。"

"If the sources of truth are the agent's declarative knowledge (i.e., what a metric means) then a skill is its procedural knowledge: which sources to consult in what order, how to navigate ambiguous data, and what a finished analysis looks like."

"如果说真相来源是 agent 的陈述性知识(declarative knowledge,即一个指标"是什么意思"),那么 skill 就是它的程序性知识(procedural knowledge):按什么顺序查哪些来源、怎么在含糊的数据里导航、一份做完的分析长什么样。"

"In Claude Code, a skill is a folder of markdown the agent reads on demand. At Anthropic, the skills we developed are hugely value additive."

"在 Claude Code 里,一个 skill 就是一个装着 markdown 的文件夹,agent 按需读取。在 Anthropic,我们开发的这些 skill 带来了巨大的增量价值。"

"Without skills, Claude's ability to answer analytics questions accurately didn't exceed 21% on our evals. Adding skills gets these numbers consistently above 95% in aggregate and regularly around 99% in certain domains. See the appendix for a skeleton we use to create a majority of our skills."

"没有 skill 时,Claude 在我们评测里准确回答分析问题的能力不超过 21%。加上 skill 后,这个数字整体稳定在 95% 以上,在某些域里还经常达到 99% 左右。我们创建大多数 skill 所用的骨架,见附录。"

"Create pairwise skills: a knowledge skill acts as a thin top-level router that allows additional domain details to load on demand. It says 'try the semantic layer first, but if there's no coverage, here are ~30 reference files for this domain describing the relevant tables, columns, joins and gotchas.'"

"创建成对的 skill:一个 knowledge skill 充当薄薄的顶层路由器,让额外的域细节按需加载。它说的是:"先试语义层,但如果没覆盖到,这里有约 30 个本域的参考文件,描述相关的表、列、join 和坑(gotcha)。""

"This router is, in effect, our answer to retrieval failure: rather than letting the agent search a million-field warehouse, it narrows the space to a few dozen curated files before a query is ever written."

"这个路由器,实际上就是我们对"检索失败"的答案:与其让 agent 去搜索一个百万字段的仓库,不如在写出任何查询之前,先把范围收窄到几十个精选文件。"

"The unbook skill encodes the process a senior analyst would follow: clarify the question, find sources (via the knowledge skill), run the query, and then loop the result through adversarial review sub-agents."

"另一个(配对的)unbook skill 则把一位资深分析师会遵循的流程编码进去:澄清问题、找来源(通过 knowledge skill)、跑查询,然后把结果再丢给对抗式审查(adversarial review)子 agent 过一遍。"

"It also bundles a dozen reusable analysis patterns (retention curves, rate decomposition, funnel analysis) so that common requests don't get reinvented each time."

"它还打包了十来个可复用的分析模式(留存曲线、比率分解、漏斗分析),这样常见请求就不必每次重新发明一遍。"

"Create proper reference docs: written for retrieval by an LLM. Our reference docs describe tables (grain, scope, and exclusions), the mechanics of gotchas (e.g., 'exclude known free-email domains, but keep custom ones like anthropic.com'), and explicit routing triggers (e.g., 'IF the question is about experiment lift… DO NOT use for raw event counts') without prescriptive recipes that go stale."

"创建像样的参考文档:专为 LLM 检索而写。我们的参考文档描述表(粒度、范围和排除项)、坑的机理(例如"排除已知的免费邮箱域名,但保留 anthropic.com 这样的自定义域名"),以及显式的路由触发条件(例如"如果问题是关于实验提升(experiment lift)……请勿用于原始事件计数"),同时不写那种会过时的死板配方。"

"Treat skill maintenance as a first class citizen: Skill docs describe a data model that changes daily, so without active maintenance they're wrong within weeks. We watched our offline accuracy drift from ~95% at launch to ~65% over a month before we treated this as an engineering problem."

"把 skill 维护当作一等公民:skill 文档描述的是一个每天都在变的数据模型,所以没有主动维护,它们几周内就会出错。在我们把这当成一个工程问题来对待之前,我们眼睁睁看着离线准确率在一个月里从上线时的约 95% 漂到了约 65%。"

"That meant colocating skill markdown files in the same repo as our transformation models, so the PR that changes a model is the same PR that updates the doc describing it. A code-review hook flags any reporting-model change that doesn't touch a skill file."

"这意味着把 skill 的 markdown 文件和我们的转换模型放进同一个仓库,这样改模型的那个 PR,就是更新描述它的文档的那个 PR。一个 code-review hook 会把任何"改了报表模型却没动 skill 文件"的改动标出来。"

"Roughly 90% of our data-model PRs now include a skill change in the same diff. We also regularly prune skill scaffolding as models improve and previous failure modes no longer apply."

"如今我们大约 90% 的数据模型 PR,都在同一个 diff 里包含了 skill 的改动。随着模型变强、过去的失败模式不再适用,我们也会定期修剪 skill 的脚手架。"

"Create a consistent and seamless experience across all surfaces: the same skill must provide the same answer to questions in Slack, in the IDE, in a dashboard tool, and in standalone agent sessions."

"在所有渠道上创造一致、无缝的体验:同一个 skill,对 Slack 里、IDE 里、看板工具里、以及独立 agent 会话里的同一个问题,必须给出同一个答案。"

"We did this by ensuring one canonical source (the data repo) and that skill changes are synced automatically. On merge, the skill syncs to a plugin marketplace (for IDE users), to cloud-storage blobs (for hosted apps that read a single file), and is served directly as resources over MCP."

"我们的做法是:确保只有一个规范来源(数据仓库),并让 skill 的改动自动同步。一合并,skill 就会同步到一个插件市场(给 IDE 用户)、同步到云存储的 blob(给那些读取单个文件的托管应用),并通过 MCP 直接作为资源提供。"

"We also designed for portability from the start by avoiding hardcoded repo paths and surface-specific namespaces."

"我们从一开始就为可移植性做了设计——避免写死仓库路径,也避免使用某个渠道专属的命名空间。"

"Finally, validation is how you find out which of the three failure modes is still leaking through."

"最后,验证(validation)是你用来搞清楚:三种失败模式里,还有哪一种在漏过去。"

"A common pattern we see is that data teams will set up elaborate analytic environments without having any process to understand the accuracy of their analytics agents."

"我们常看到一种模式:数据团队搭起了精巧的分析环境,却没有任何流程去搞清楚自己分析 agent 的准确率。"

"One way of addressing this gap is via offline evals, which are simple question / answer pairs. You can think of offline evals similar to offline testing for an ML model in that they don't tell you the performance of your online agents, but they do give you a good sense of whether you'll have any critical gaps."

"弥补这个缺口的一个办法是离线评测(offline eval),也就是简单的问答对。你可以把离线评测类比为 ML 模型的离线测试:它们不会告诉你线上 agent 的真实表现,但确实能让你对"是否存在关键缺口"有个不错的判断。"

"We deploy two kinds of offline evals at Anthropic. Dashboard-based evals are auto-generated by Claude (then human validated), covering the most common stakeholder questions. Long tail evals are where we feed Claude business context (roadmaps, table docs) and have it generate plausible questions across the rest of the domain."

"在 Anthropic 我们部署了两类离线评测。基于看板的评测由 Claude 自动生成(再经人工校验),覆盖最常见的业务方问题。长尾评测则是:我们把业务上下文(路线图、表文档)喂给 Claude,让它在域的其余部分生成看似合理的问题。"

"We also continuously harvest every time a stakeholder corrects the agent in a thread as that correction is a candidate eval. Other best practices include:"

"我们还持续地收割:每当业务方在某个会话串里纠正 agent,那次纠正就成了一个候选评测。其他最佳实践包括:"

"Anchor ground truth so it can't drift: An eval written against live data goes stale the moment the underlying number moves. Pin every eval to a snapshot date, write it against a stable fact table, or have the grader judge the agent's query rather than its number. Wire the suite into CI so a PR touching a dependency re-runs the affected evals."

"锚定标准答案,让它无法漂移:针对实时数据写的评测,在底层数字一变动的瞬间就过时了。把每个评测钉在一个快照日期上、针对一张稳定的事实表(fact table)来写,或者让评分器去判断 agent 的查询、而不是它给出的数字。把整套评测接进 CI,这样一个动了某依赖项的 PR 会重跑受影响的评测。"

"Store results like telemetry, not like test logs: Every run lands in a warehouse table with the skill version, git SHA, model ID, per-assertion pass/fail, token count, and wall-clock. 'Did that change help?' becomes a query, and you get the time-series to catch slow regressions that a single CI run won't."

"把结果当遥测(telemetry)存,而不是当测试日志存:每次运行都落进一张数仓表,带上 skill 版本、git SHA、模型 ID、每条断言的通过/失败、token 数和墙钟时间(wall-clock)。"那次改动有没有帮助?"就变成了一句查询,而且你能拿到时间序列,捕捉那种单次 CI 跑发现不了的缓慢回退。"

"Gate launches per domain: A domain owner can't announce the agent to their stakeholders until their slice of the eval set clears some threshold (we initially used ~90%). It forces reference-doc fixes before users see the failures."

"按域设置上线门槛:一个域的负责人,在自己那部分评测集越过某个阈值之前(我们最初用约 90%),不能向自己的业务方宣布这个 agent 可用。这逼着大家在用户看到失败之前,先把参考文档修好。"

"Create the appropriate number of evals: The number of evals you should have depends on the complexity of the business area and the complexity of the underlying data model. Calibrate by tracking how well offline accuracy predicts online accuracy: we've found there are diminishing returns past a few dozen per topic (e.g., 'growth'), and that ceiling drops with each new model generation."

"创建数量合适的评测:你该有多少评测,取决于业务领域的复杂度和底层数据模型的复杂度。校准的办法是追踪离线准确率对在线准确率的预测能力有多好:我们发现,每个主题(比如"增长")超过几十个评测后就会边际递减,而且这个上限随着每一代新模型的到来还在下降。"

"Offline eval accuracy should be ~100%; every correct answer should also be hitting your semantic layer (if you have one). Again, this level of accuracy doesn't tell you your system isn't going to produce a wrong answer, just that there are no obvious gaps, assuming you have proper eval coverage."

"离线评测准确率应当接近 100%;每个正确答案也都应该命中你的语义层(如果你有的话)。再强调一次,这种程度的准确率并不能告诉你系统不会产出错误答案,只能说明没有明显的缺口——前提是你的评测覆盖到位。"

"Every structural decision about the skill (e.g., which sources to expose, whether a sub-agent earns its latency, whether to merge two skills into one) is made by holding our offline eval set fixed. We vary exactly one component and compare pass rates."

"关于 skill 的每一个结构性决策(例如暴露哪些来源、某个子 agent 值不值它带来的延迟、要不要把两个 skill 合成一个),都是在固定离线评测集的前提下做出的。我们只改动一个组件,然后比较通过率。"

"Each run only takes an hour and replaces a lot of arguments. The methodology matters more than any single result:"

"每次跑只要一小时,却能省掉一大堆争论。方法论本身,比任何单一结果都更重要:"

"Design for null results. Our most useful ablation was a negative one. We gave the agent direct grep access to our entire dashboard, transformation, and analyst-notebook SQL (thousands of files). We then verified in transcripts that it actually read them before every answer. Accuracy moved by less than a point in either direction."

"为"零结果"做设计。我们最有用的一次消融实验,是一个否定的结果。我们给了 agent 对我们全部看板、转换、分析师 notebook 的 SQL(成千上万个文件)的直接 grep 权限。我们还在记录(transcript)里核实了:它在每次回答前确实读了这些文件。结果准确率在任一方向上的变化都不到一个百分点。"

"We then checked the obvious confounds: was the answer actually in the corpus for the questions it got wrong? About 80% of the time, yes. Did 'answer present' predict 'now gets it right'? No, the flip rate was flat. The information was there, the agent saw it, and it still didn't use it."

"接着我们检查了那些显而易见的混杂因素:对那些它答错的问题,答案其实在语料里吗?大约 80% 的情况下,在。"答案存在"能预测"现在答对了"吗?不能,翻转率是平的。信息就在那儿,agent 也看到了,它却依然没用。"

"That single experiment told us our bottleneck wasn't access to prior work, it was structure (i.e., mapping a question to the right entity). That insight redirected months of roadmap."

"这一个实验就告诉我们:瓶颈不在于"能不能拿到前人的成果",而在于结构(即把一个问题映射到正确的实体)。这个洞见改写了好几个月的路线图。"

"Ablate at PR granularity. Every meaningful skill edit gets a before / after run on the relevant eval slice, with the delta in the PR description. It keeps 'I improved the docs' honest and catches the surprisingly common case where a well-intentioned addition makes things worse."

"在 PR 粒度上做消融。每一次有意义的 skill 改动,都在相关的评测切片上跑一遍前/后对比,并把差值写进 PR 描述里。这让"我改进了文档"这句话保持诚实,也能抓住那个出乎意料地常见的情况:一处好心的添加,反而把事情弄糟了。"

"Keep a short list of what didn't work. Two of ours: stacking additional rounds of doc refinement past a certain point (we hit three consecutive net-negative iterations: the docs were getting longer, not better), and swapping the adversarial reviewer to a cheaper model to cut latency (it lost most of the accuracy wins, for no real speedup)."

"留一份"什么没用"的简短清单。我们的两条:在某个点之后继续叠加更多轮的文档精修(我们连续撞上三次净负面的迭代:文档变长了,而不是变好了);以及把对抗式审查者换成更便宜的模型来削减延迟(它丢掉了大部分准确率收益,却没换来真正的提速)。"

"Negative results are cheap to record and they prevent the next person from re-running the same experiment."

"负面结果记录起来很便宜,而且能让下一个人不必重跑同一个实验。"

"The final step is ensuring the actual online system performance is as accurate as possible. Some of the steps we take include:"

"最后一步,是确保实际的在线系统表现尽可能准确。我们采取的一些手段包括:"

"Adversarial review: we've found that employing a Claude skill to aggressively challenge all underlying assumptions on a potential final answer increased accuracy by 6% within our eval set, but at the cost of 32% more tokens and 72% higher latency."

"对抗式审查:我们发现,用一个 Claude skill 去激进地挑战某个潜在最终答案背后的所有假设,在我们的评测集里把准确率提高了 6%,但代价是多花 32% 的 token、延迟高出 72%。"

"Provenance footer: every response carries a footer that contains which source tier it came from (semantic layer › curated reference › raw table), how fresh the underlying data is, and who owns the model."

"溯源页脚(provenance footer):每条回答都带一个页脚,标明它来自哪个来源层级(语义层 › 精选参考 › 原始表)、底层数据有多新、以及这个模型归谁负责。"

"It doesn't make the answer more correct, but it does help the consumer judge how much they can trust the response. A 'raw table, freshness unknown' footer is a signal to verify before forwarding upstream, and it's one of the few mitigations we have for silent failures."

"它不会让答案更正确,但确实能帮使用者判断该有多信任这条回答。一个"原始表、新鲜度未知"的页脚,就是个信号:在往上转发之前先核实——而这是我们应对无声失败为数不多的缓解手段之一。"

"Data quality checks: it's possible that your agent is using the right field in the appropriate way, but the data itself is incorrect. Adding basic data quality checks to ensure the referenced field is up-to-date, complete, and has no anomalies is generally good hygiene."

"数据质量检查:有可能你的 agent 用对了字段、用法也得当,但数据本身就是错的。加上基本的数据质量检查,确保被引用的字段是最新的、完整的、没有异常,通常都是个好习惯。"

"Passive monitoring: two production signals we track continuously are the share of agent queries that resolve through the semantic layer, and the share of responses that use correction language ('that's the wrong table,' 'you're missing the fraud filter'). Both feed a dashboard reviewed weekly alongside the offline pass rate."

"被动监控:我们持续追踪两个生产信号——通过语义层解析的 agent 查询占比,以及回答里出现纠正性措辞("用错表了"、"你漏了欺诈过滤")的占比。两者都汇入一个看板,每周和离线通过率一起复盘。"

"Active correction harvesting: the part that closes the loop. A scheduled agent scans stakeholder channels every few hours for similar correction language, drafts a one-line fix to the relevant reference doc, and opens a PR tagged to the domain owner."

"主动收割纠正:闭环的那一环。一个定时 agent 每隔几小时扫描业务方的频道,寻找类似的纠正性措辞,为相关参考文档起草一行修复,并开一个 @ 给域负责人的 PR。"

"The fix path is deliberately boring — edit a markdown file, merge, auto-sync everywhere — so a domain owner doesn't spend too much time on the task. The same corrections feed back into the offline eval set."

"这条修复路径被刻意设计得很无聊——改一个 markdown 文件、合并、自动同步到各处——这样域负责人就不必在这件事上花太多时间。这些纠正同样会反哺到离线评测集里。"

"The failure mode none of this fully catches is the silent one. The answer is wrong, but looks plausible and is used without objection."

"以上这些都无法完全抓住的那个失败模式,是无声的那个:答案是错的,但看起来合情合理,被毫无异议地用掉了。"

"Our mitigations are the provenance footer, explicit human sign-off on anything leadership-bound, and a standing eval for each domain's top KPIs that sanity-checks against the blessed dashboard daily, though we don't have a robust solution yet."

"我们的缓解手段是:溯源页脚、任何要呈给管理层的东西都需明确的人工签字,以及为每个域的核心 KPI 设一个常驻评测、每天对标官方认可的看板做合理性检查——尽管我们还没有一个稳健的解法。"

"If you're starting from zero, a handful of canonical datasets, a few dozen offline evals, and a thin knowledge skill will capture most of the upside; everything else in this post is what we added once those were built."

"如果你从零开始,那么几个规范数据集、几十个离线评测、外加一个薄薄的 knowledge skill,就能拿到大部分收益;这篇文章里的其他一切,都是我们在这些东西搭好之后才加上去的。"

"We also shared many best practices, and not all of them will be appropriate for every data team. Align with your organization on a few principles that will affect your approach by asking:"

"我们也分享了许多最佳实践,但并非每一条都适合每个数据团队。通过下面这几个问题,和你的组织在几条会影响你方法的原则上对齐:"

"How important is a correct answer today vs. in the future? AI models are progressing at a rapid pace. We often see companies building a significant amount of infrastructure to account for current model shortfalls that become moot once those models improve."

"一个正确答案,今天有多重要、将来又有多重要?AI 模型正在飞速进步。我们经常看到公司为弥补当前模型的不足而搭建大量基础设施,可一旦模型变强,这些设施就变得多余了。"

"Knowing where models fall short, and waiting for model improvements to fill the gap has significantly less overhead, but may not fit your company's risk tolerance."

"清楚模型在哪儿不足、然后等模型进步来填补缺口,开销要小得多,但可能不符合你公司的风险承受度。"

"How do you anticipate the complexity of your business to change over time? Some of the processes we discussed may be overkill if, for example, you don't produce much data, you only have a few consumers of the output, or your data model is likely to remain simple."

"你预计自己业务的复杂度会怎样随时间变化?我们讨论的一些流程可能是杀鸡用牛刀——比如你产出的数据不多、输出的消费者只有寥寥几个,或者你的数据模型大概率会一直保持简单。"

"How technical is the intended audience of the output? Phrased differently, if you're building this analytics system for data scientists who can recognize when an answer is incorrect, you may be more tolerant of errors compared to a situation in which the audience has no familiarity with the underlying data model."

"输出的目标受众有多懂技术?换个说法:如果你这套分析系统是给能识别"答案何时不对"的数据科学家用的,那相比受众对底层数据模型一无所知的情形,你可以对错误更宽容一些。"

"How much are you willing to spend for improved accuracy? We've found certain processes like adversarial validation can significantly improve accuracy, but often at a higher cost and latency."

"为了更高的准确率,你愿意花多少?我们发现,对抗式验证(adversarial validation)之类的某些流程能显著提升准确率,但往往伴随更高的成本和延迟。"

"What is your comfort around access controls and internal data privacy? Agents are often significantly more performant the more context they have; however, broad data access cuts against most companies' governance posture. This determines whether you're building one agent or many scoped ones."

"你对访问控制和内部数据隐私的接受底线在哪儿?agent 往往是上下文越多、表现越好;然而,宽泛的数据访问权限,和大多数公司的治理姿态是冲突的。这决定了你是要做一个 agent,还是做许多个权限受限(scoped)的 agent。"

"Whatever your route, our greatest gains have come from addressing each of the three failure modes: collapsing ambiguity into a single governed answer, making the answer easily discoverable, and flagging when either has gone stale."

"无论你走哪条路,我们最大的收益都来自解决那三种失败模式:把歧义坍缩成唯一一个受治理的答案、让这个答案易于被发现,以及在两者中任何一个变陈旧时把它标出来。"

"This article was written by Chen Chang, Clement Peng, Justin Leder, Johanne Jiao, and Josh Cherry, members of the Data Science and Data Engineering team. The authors would like to thank Michael Segner for his contributions."

"本文由数据科学与数据工程团队的成员 Chen Chang、Clement Peng、Justin Leder、Johanne Jiao 和 Josh Cherry 撰写。作者们感谢 Michael Segner 的贡献。"

"What follows is the skeleton of our main warehouse skill: the real file's structure, with internal specifics replaced by [bracketed placeholders]. It isn't meant to be copied verbatim; it's meant to show the kinds of sections we found worth writing down."

"下面是我们主仓库 skill 的骨架:真实文件的结构,内部细节用 [方括号占位符] 替换。它不是用来逐字照抄的,而是用来展示——我们发现哪几类小节值得专门写下来。"