Evaluate the narrowest LangChain runnable that owns the failure you are chasing, then make the dataset match that boundary. LangSmith’s runnable-evaluation documentation says chat models, retrievers, chains and other Runnable objects can be passed directly to evaluate() or aevaluate(). For chains, it also requires the chain’s input variables to match the dataset example’s input keys. Boundary selection and dataset schema are therefore parts of the experiment design, not reasons to build a custom harness.
The release consequence is less convenient: a passing component evaluation is not an end-to-end result. LangSmith documents per-output tracing, which supplies evidence for diagnosis, but connecting that evidence to whole-application outcomes remains your responsibility. Treat a component score as a release signal only after you have made that connection on your own stack.
The product behavior described below comes from LangChain’s vendor documentation, accessed on 20 November 2026, with no independent evaluation of LangSmith included. The checklist and release recommendations are editorial synthesis of that documented contract, not reported performance results.
Start with the failure, not the framework object
Direct Runnable support gives you several useful evaluation boundaries:
- A chat model when the question concerns the model’s response at that boundary.
- A retriever when retrieval is the behavior under examination.
- A chain or sub-chain when the relevant behavior emerges from a particular composition.
- The whole runnable when the question is whether the application completes its intended task.
The documentation establishes that these objects can be evaluated directly. It does not establish that scores at different boundaries are interchangeable. That distinction drives the recommendation.
Consider a hypothetical agent that returns a poor final answer. Evaluating its retriever can help isolate retrieval behavior, but a good retriever score cannot show that the final answer used the retrieved material correctly. Evaluating the whole runnable can show that the answer failed, without by itself assigning the failure to retrieval. These evaluations answer different questions, so neither universally replaces the other.
If you already know which component owns the failure, start narrowly. You will evaluate the behavior of interest without requiring the rest of the application to define the score. If you do not know where the failure lives, begin with the whole runnable and use its outputs and traces to choose a narrower follow-up.
This ordering is an editorial recommendation, not a measured LangSmith result.
The dataset is an interface contract
Changing the evaluation boundary changes the input contract. LangSmith’s chain example is explicit:
Note that the input variables of the chain must match the keys of the example inputs. In this case, the example inputs should have the form {“text”: ”…”}.
That example does not mean every Runnable uses a text key. It means the key set must be derived from the Runnable you chose. A dataset prepared for a whole application should not be assumed to fit a sub-chain, and a dataset prepared for one chain should not be assumed to fit another merely because their purposes look similar.
Before running an evaluation, make three checks:
- Write down the selected Runnable’s input variables.
- Compare them with every dataset example’s input keys.
- Recheck that mapping whenever you move the evaluation boundary.
The documentation does not specify exactly how every mismatch fails, so the safer plan is to perform this check yourself rather than rely on a particular error or coercion behavior. It also does not prove that key matching is the only schema requirement for every Runnable or API version.
What belongs in an example
LangSmith’s evaluation concepts divide an example into three parts:
- Inputs: a dictionary of input variables passed to the application.
- Reference outputs: an optional dictionary used only in evaluators.
- Metadata: optional additional information that can be used to create filtered views of a dataset.
Reference outputs deserve particular attention because their role is easy to blur. They do not get passed to the application. A reference answer can help an evaluator judge factual accuracy, but it is not an extra application input or an implicit prompt field. If your application genuinely needs a value at runtime, that value belongs in the application’s input contract, not in an evaluator-only reference field.
Metadata serves a different purpose again. The documented use is filtered dataset views. It can help organize evaluation analysis, but the documentation does not say that metadata automatically correlates separate component and end-to-end experiments.
Choose evaluator semantics before interpreting green
LangSmith lists five evaluation approaches: Human, Code, LLM-as-judge, Decision model, and Pairwise. The label attached to a score matters because these approaches do not all supply the same kind of evidence.
Code evaluators are deterministic, rule-based functions suited to checks such as:
- whether a chatbot response is non-empty;
- whether generated code compiles;
- whether a classification exactly matches the expected class.
These checks can be useful without capturing overall task success. A non-empty response can still be wrong, and compilable code can still fail its intended behavior. The evaluator defines what the green result means.
LLM-as-judge evaluators can be reference-free or reference-based. The documentation’s examples include offensive-content or criteria checks without a reference, and factual-accuracy assessment relative to a reference. In the reference-based case, the example’s reference output supplies material to the evaluator, not to the application.
Decision model evaluators use a decision model, such as Jev, as the judge: instead of a prompt and an output schema, you define typed questions, such as a probability that a statement is true, a choice from a fixed list, or a score along ordered levels, and each question’s answer becomes its own piece of feedback. Pairwise evaluators compare outputs from two application versions using heuristics, LLMs with pairwise prompts, or human reviewers, an approach LangSmith’s evaluation concepts recommends when directly scoring an output is difficult but comparing two outputs is straightforward.
What that page does not settle are judge versions, scoring thresholds and other evaluator parameters. Confirm those in the documentation for the version you intend to run before using their scores as gates.
A practical selection rule follows from the documented examples: use deterministic code when the failure can be expressed as an exact, repeatable check, and use a judge when the criterion requires judgment that the documented judge approach supports. Where neither approach adequately represents task completion, a component metric should remain diagnostic rather than become the release criterion.
Traces locate evidence; they do not join the argument for you
The runnable-evaluation page states:
The runnable is traced appropriately for each output.
That establishes trace scope at the output level. It gives you a trace associated with each evaluation output, rather than leaving you with only an aggregate score. This is what makes trace-based failure inspection possible at the chosen boundary.
It does not establish two stronger claims:
- that separate component and end-to-end runs are joined automatically;
- that a trace explains every failure without examining the relevant inputs, outputs and evaluator.
The joining step therefore belongs in your evaluation plan. Run the component boundary and the whole-runnable boundary on examples whose relationship you can identify, retain their per-output results and traces, and compare component behavior with the final outcome. The exact identifier, sampling and concurrency mechanics are not established by the documentation, so this article cannot prescribe a universal implementation.
The comparison should allow disagreement. A retriever evaluation might pass while the whole runnable fails, or a component might fail without changing the final outcome. Those disagreements are reasons to revise the diagnosis or evaluator, not inconveniences to average away. Task decomposition increases the number of component measurements available, but it also increases the burden of showing that those measurements explain end-to-end behavior.
A boundary-and-schema checklist
The following checklist is editorial synthesis. The middle column summarizes documented behavior; the final column states the additional decision required before treating a score as release evidence.
| Stage | What the documentation establishes | What your evaluation plan must decide |
|---|---|---|
| Boundary | Chat models, retrievers, chains and other Runnables can go directly into evaluate() or aevaluate() in the runnable guide. | Which boundary owns the diagnostic question, and which whole-runnable result is required for release? |
| Input schema | A chain’s input variables must match the example-input keys; the documented example uses {"text": "..."}. | Do the keys match at this exact boundary, and have you rechecked them after changing boundaries? |
| References | Reference outputs are optional and are used only in evaluators, according to the evaluation concepts. | Is each reference being used for judgment rather than mistaken for application input? |
| Metadata | Optional metadata can support filtered dataset views. | How will you organize and relate the runs needed for your analysis, without assuming automatic correlation? |
| Evaluator | LangSmith lists Human, Code, LLM-as-judge, Decision model and Pairwise approaches. | Does the evaluator measure the failure you care about, and are its semantics sufficiently verified for a gate? |
| Trace and outcome | The Runnable is traced for each output. | Have you compared component evidence with end-to-end outcomes, including cases where they disagree? |
If any of these questions remains unanswered, the score can still be useful. Its scope should simply remain narrower than the decision you are trying to make.
Self-hosted availability does not change the evidence standard
Evaluation is not documented as cloud-only. The self-hosted LangSmith page says:
The base installation includes observability, evaluation, prompt engineering, Insights, and Chat.
That supports using the same evaluation planning discipline in a self-hosted deployment. It does not establish that every installation or version exposes identical options, and the page does not cover sampling, concurrency or evaluator configuration there. Check those details against the version you operate rather than treating base-installation availability as proof of operational equivalence.
AndroidWorld’s 100% is a separate research question
The task-decomposition paper “Do Multi-Agents Dream of Electric Screens? Achieving 100% on AndroidWorld Through Task Decomposition” is relevant because it raises the same joining problem: when does decomposed work support a claim about complete tasks? It does not answer that question for your LangChain application.
Two different 100% figures sit near this benchmark, and they should not be conflated. The five leaderboard entries at 100% that the paper cites are self-reported single trials, per its Table 3 caveat:
Cited rows are self-reported and not independently verified; every one that states a trial count, including all five at 100%, is a single trial.
The paper’s own headline is a different measurement: on three pre-registered held-out seeds, with three runs per task, the final system reaches a pass@3 of 100.0% (95% CI 96.8–100.0) and a mean success of 98.0%. Neither the cited single-trial rows nor this multi-run figure is an expected outcome for another agent.
The protocol also limits casual comparisons, and it differs between the paper’s two systems. In the initial-system study, each task has a 30-minute wall-clock cap, and reaching that cap counts as failure. A task is limited to 400 graph steps, defined as LangGraph node steps rather than device actions (Appendix D). Each LLM call has a 180-second timeout, two retries on the same model and no model fallback. Infrastructure failures allow up to three attempts for a cell, while an agent failure is never re-run. The final system behind the pass@3 raised the limits to 1500 graph steps and a 3600-second wall-clock cap per task (Appendix G), a change that itself shows how protocol-dependent the headline percentage is.
The outcome verifier adds further protocol-specific meaning. It judges requirements using device evidence only, treats agent statements as claims, rejects at most three times, and uses its final answer as the reported answer. These are not incidental details that can be discarded while retaining the headline percentage. They define part of what the reported result measured.
None of this shows that task decomposition is ineffective. It shows that the paper’s result cannot establish improved end-to-end reliability for your stack. If decomposition is useful in your application, your own evaluation still has to connect component scores, whole-runnable outcomes and the relevant traces.
What remains to verify before implementation
The documentation establishes direct Runnable evaluation, chain key matching, example structure, the evaluator taxonomy, per-output tracing and self-hosted base-installation availability. It does not establish current evaluate() or aevaluate() parameter signatures, dataset-upload mechanics, sampling or concurrency behavior, evaluator thresholds, versioned judge behavior, or every possible schema constraint. It also supplies no performance measurements for your application.
Check those operational details against the API and deployment version you will use, and run a small preflight evaluation before committing a large dataset or release process.
My recommendation is therefore conditional: use the narrowest Runnable for diagnosis, enforce the dataset contract at that boundary, and choose an evaluator whose semantics match the failure. Require whole-runnable evidence for release.
Frequently Asked Questions
What must match between a chain and its dataset examples?
Note that the input variables of the chain must match the keys of the example inputs. In this case, the example inputs should have the form {“text”: ”…”}.
Are reference outputs passed to the application during evaluation?
Reference outputs deserve particular attention because their role is easy to blur. They do not get passed to the application. A reference answer can help an evaluator judge factual accuracy, but it is not an extra application input or an implicit prompt field.
Does the base self-hosted installation include evaluation?
The base installation includes observability, evaluation, prompt engineering, Insights, and Chat.
Join the discussion
Share a useful perspective or ask a question about this article.