Keep Runnable evals for component behavior and dataset wiring, but do not treat a green chain score as evidence that an agent completed the user’s task. LangChain’s undated, vendor-maintained documentation says Runnables such as chat models, retrievers and chains can be passed directly into evaluate() or aevaluate(). That establishes a component evaluation path, not a whole-agent outcome guarantee.
The practical choice is therefore not Runnable or agent. Decide which claim each check is allowed to support, then budget separately for an end-to-end environment and failure review when the claim is task completion. No supplied study compares failure detection or costs across these levels, so this is a placement recommendation, not a measured ranking.
LangChain’s documented path tests an object under a contract
The direct Runnable path is useful because it gives teams a defined way to evaluate components without constructing a separate agent harness. It also imposes a precise dataset requirement: the chain’s input variables must match the keys of the example inputs. LangChain’s example expects inputs in the form {"text": "..."} when the chain uses text.
Treat that contract as part of the test setup, not as clerical detail. If the dataset and component disagree about their interface, fix that before interpreting the result as model or chain behavior. A passing run can support a narrow statement: this object produced acceptable results for these examples under this evaluator. It does not establish that a larger agent reached the required final state after all of its decisions, tool calls and retries.
The important distinction is not merely the Python class being evaluated. A chain may contain several steps. The question is whether the check evaluates a component’s response to supplied examples, or lets an agent attempt an entire task in an environment and then assesses the outcome against the user’s requirements.
LangSmith adds a separate axis: offline and online evaluation, distinguished by when and where they run. That distinction still does not settle the level under test. Moving an evaluation online does not turn a component check into evidence of whole-task completion, and an offline end-to-end check still needs an evaluator that recognizes completion correctly.
A useful taxonomy, with vendor framing attached
Vercel’s agent-evaluation post groups checks by the LLM calls they typically require: deterministic tests use zero, one-turn evals use one, and end-to-end evals use two or more across a whole task. Vercel presents call count as a proxy for the effort to build the check, its running time and its cost.
That is a helpful planning vocabulary, but it is a vendor argument and a self-described proxy, not a measurement of a LangChain deployment. The more consequential distinction is what each level observes:
- A deterministic or component check can validate a specific interface or output property.
- A one-turn eval can examine a single decision, such as the agent’s first command.
- An end-to-end eval can run the complete task in an environment and compare the final result with the user’s requirements.
Vercel’s central argument is that individually reasonable steps can still produce the wrong final outcome. That explains why adding more rows to a passing chain dataset may improve component coverage without supplying any new evidence about task completion.
It does not follow that component evals are disposable, or that an end-to-end pass rate is automatically trustworthy.
Even an end-to-end score can misrecord achievement
The human-grounded audit of all 165 WebArena Lite tasks separates two capabilities that an automatic accept rate combines: whether the agent completed the task and whether the evaluator recognized that completion.
The distinction matters operationally. The paper identifies exact string matching, output formatting, partial state checks and differences in stopping behavior as factors that “can all change the recorded score without changing what the agent achieved.” A binary result also cannot distinguish several importantly different runs: an agent that never reached the relevant page, one that completed most requirements, one that reached a correct state and then kept acting, or one that returned a semantically correct answer in an incompatible format.
This is counter-evidence against solving component blindness by trusting the highest-level dashboard instead. An end-to-end environment supplies the opportunity to observe the task outcome; it does not guarantee that the scorer interpreted that outcome correctly.
For a release decision that depends on task completion, I would not use an unreviewed automatic pass rate as the sole gate. The audit offers a concrete review model: compare failed runs with the task instruction, environment state, screenshots, execution records and final response. The labor for that review belongs in the evaluation budget. It is not included automatically because a framework produced a score.
Trajectories expose failures that endpoints hide
Some checks belong between a single component and the final outcome. Their job is to establish whether the agent’s process remained coherent, whether a retry changed anything and whether intermediate states stayed usable.
In one study of repair agents working on real Python bugs, a plain retry edited no source file in 30.9% of failures: 56 cases, including 22 where the agent declared the task done without making a change. When the previous attempt’s transcript was supplied, the no-edit share fell to 13.8%, or 25 cases.
That is a single-system result, not a predicted improvement for LangChain agents. It does show why failure review needs trajectory evidence. A final score of zero does not distinguish “attempted the wrong repair” from “declared success without attempting anything,” and those failures call for different engineering work.
Intermediate integrity can matter independently of the final artifact as well. The commit-untangling study reports that 20.6%–37.4% of baseline coding-agent patch histories contained at least one version failing runnability checks, while its RucTangle histories passed those checks after every commit. Here, “runnable” means that the code remains runnable after each commit. It is unrelated to LangChain’s Runnable interface, despite the shared term.
The transferable lesson is narrower than the numbers: if intermediate states matter to the user or to subsequent work, inspecting only the final patch can miss a relevant failure.
Component checks still earn their place
The evidence against overreading component scores is not evidence against component instrumentation. A runtime failure-detection and repair study includes a check that validates tool results against the shapes their tools can return. In that system, it flagged 0 of 1825 healthy episodes while catching 46% of injected context corruption, with 215 of 218 detections occurring within one step of onset.
The same study closed detection into repair by rolling flagged runs back to their last fact-gathering step and running them again. It recovered 45% of failures, compared with a 16% resampling control, and lifted net task success from 52% to 73% for about one extra model call per run.
Those results come from another agent and environment. They do not predict detection rates, repair lift or runtime expense in LangChain. They do preserve an important place in the rubric for cheap, specific checks: a tool-shape violation can be diagnosed directly without waiting for an end-to-end evaluator to infer that something went wrong. Conversely, a clean shape check says nothing by itself about whether the overall task was completed.
Budget the environment and the review, not just dataset rows
The source set does not measure the fixture, infrastructure or review cost of evaluating a LangChain agent end to end. Those figures should appear in a project plan as assumptions to be tested, not as benefits or costs established by the literature.
Two reference points help explain why verified data deserves its own budget line:
- The Evaluation Context Protocol paper, citing Anthropic research secondhand, recommends starting with 20–50 explicit tasks drawn from real-world failures, bug trackers and support queues rather than a curated wishlist. Treat this as attributed starting guidance, not a required dataset size.
- A shopping-agent trajectory study measured roughly one verifier-accepted training row per thousand raw trajectories in a representative run. That was a training-data pipeline, not a LangChain eval pipeline. It should not be used to forecast eval-dataset yield, but it demonstrates how expensive verified trajectory data became in one system.
For planning, assume that task-level evaluation will require an environment in which the agent can attempt the task, a way to inspect the resulting state and time to review failed trajectories. Then measure those costs in your own system. Vercel’s call-count taxonomy can help organize the estimate, while the repair study’s extra-call figure belongs only to its reported setup.
Where each check belongs
This rubric places checks according to the claim they need to support. It does not rank their effectiveness, because no supplied evidence makes that comparison.
| Level | Use it to support | Do not use it alone to claim | Cost basis |
|---|---|---|---|
| Runnable or deterministic component check | Component behavior on selected examples, interface alignment and specific output properties such as tool-result shape | The agent completed a multi-step user task | LangChain documents the direct evaluation path; no comparative cost measurement is supplied |
| One-turn eval | The quality or correctness of a single decision | A sequence of reasonable decisions produced the required outcome | Vercel’s typical one-call classification is a vendor proxy |
| Trajectory or runtime check | Intermediate validity, corruption detection, no-edit retries and broken states within a run | The final user requirement was met, unless a separate outcome check establishes it | Repair and review figures are single-study results; local review labor remains unmeasured |
| End-to-end task eval | The final result of a whole task attempted in an environment | The score is necessarily correct, or component and trajectory failures are absent | Environment, fixtures and failure review require separate local estimates; two or more calls is Vercel’s proxy |
If the team cannot yet fund the end-to-end environment, the defensible response is to narrow the claim attached to the dashboard. Do not promote a chain eval into an agent eval merely because it is the check already available.
Portability is a separate design decision
The ECP paper also argues that evaluation frameworks remain tightly coupled to orchestrators, naming LangChain, LlamaIndex, CrewAI and AutoGen, as well as vendor-hosted observability platforms. It characterizes the result as vendor lock-in.
That is a claim about the ecosystem, not a measured switching cost for your implementation. If changing orchestrators is a plausible requirement, inventory which artifacts would remain usable outside the current stack: task definitions, input datasets, outcome requirements, evaluator logic and retained trajectory evidence. A portability concern may justify keeping some of those artifacts independent, but the supplied research does not establish that a particular protocol or framework will accomplish that for you.
Finally, LangChain’s cited API documentation is undated and unversioned. Confirm evaluate(), aevaluate() and the input-key behavior against the version you deploy before building around them.
Until evidence from your own system shows otherwise, the safe interpretation of a green LangChain chain eval is “this component passed these examples.” Whole-task completion requires task-level evidence, and even that evidence needs an evaluator and failure review capable of recognizing what the agent actually achieved.
Frequently Asked Questions
What is the specific dataset requirement for LangChain Runnable evals?
It also imposes a precise dataset requirement: the chain’s input variables must match the keys of the example inputs. LangChain’s example expects inputs in the form {"text": "..."} when the chain uses text.
How does Vercel categorize agent evaluation checks?
Vercel’s agent-evaluation post groups checks by the LLM calls they typically require: deterministic tests use zero, one-turn evals use one, and end-to-end evals use two or more across a whole task.
What does the human-grounded audit of WebArena Lite tasks separate?
The human-grounded audit of all 165 WebArena Lite tasks separates two capabilities that an automatic accept rate combines: whether the agent completed the task and whether the evaluator recognized that completion.
Join the discussion
Share a useful perspective or ask a question about this article.