How to Wire Agent Evals in LangSmith: What evaluate() Requires

LangChain’s official docs state two requirements for wiring a first eval harness in LangSmith, and only two: pass a LangChain runnable directly into evaluate() or aevaluate(), and make every example input’s keys exactly match the chain’s input variables. If you read a concept primer and stalled somewhere between “evals are good” and a running pipeline, that gap is smaller than it looks.

What evaluate() actually requires

The core contract comes from LangChain’s docs page on evaluating a runnable:

langchain Runnable objects (such as chat models, retrievers, chains, etc.) can be passed directly into evaluate() / aevaluate().

That sentence does real work. Your eval target does not need to be wrapped in an adapter, converted to a custom function, or rebuilt for the harness. A chat model, a retriever, a chain: any of these is a runnable, and any runnable goes straight into the call. The async variant, aevaluate(), takes the same shape. For a team with an existing LangChain agent, the harness’s entry point is the object you already have.

The second requirement is the one that bites in practice, and the docs state it just as directly:

Note that the input variables of the chain must match the keys of the example inputs. In this case, the example inputs should have the form {“text”: ”…”}.

Every example in your dataset is a dict. The keys of that dict must be exactly the input variable names your chain expects. If your chain takes a variable called text and your dataset rows use question or input, the harness cannot wire them together. This is not a validation warning the library will necessarily surface helpfully; it is a structural contract. Rename the chain’s variables or reshape the dataset, but one side has to move.

A practical checklist for a first harness reduces to this: pick the runnable you want to grade, build a small dataset whose example inputs use that runnable’s exact input variable names as keys, and call evaluate() (or aevaluate()) with both. Everything else, evaluators, experiment naming, feedback, is layered on top of a loop that already runs.

What to check when a run produces nothing

The input-schema rule is the first thing to check when a run produces nothing: a key mismatch means the example data and the chain never connect. Whether the mismatch surfaces as a clear error or a confusing one is an inference from the matching rule rather than a documented behavior, so treat it as the likeliest place to look, not a known bug.

If your eval target is a LangGraph graph rather than a chain, the same runnable-evaluation page has a section titled “How to evaluate a langgraph graph.” The input-key matching rule above is stated for chains; start from that section for graph wiring rather than assuming the chain instructions transfer unchanged.

Where your traces live

Before standardizing on LangSmith, the hosting decision deserves more attention than the API does, because LangChain’s own pages are in mild tension about it. The LangSmith platform page advertises “SaaS with data residency in the US, Europe, and APAC,” alongside bring-your-own-cloud and self-hosted deployment. The observability page is more specific about the default:

When using LangSmith hosted at smith.langchain.com, data is stored in GCP us-central-1. If you’re on the Enterprise plan, we can deliver LangSmith to run on your kubernetes cluster in AWS, GCP, or Azure so that data never leaves your environment.

Read together: regional residency exists as an option, but the default hosted service puts your traces, which contain your prompts, your retrieved context, and often your users’ inputs, in one Google Cloud region in the central United States. For a team under EU data-residency obligations, the default is not the right answer, and the existence of an EU option on a marketing page is not the same as knowing how to get an account provisioned there. Resolve this before your first production trace, not after.

The same page notes that LangSmith tracing is not LangChain-only: it covers applications built with the OpenAI SDK, Anthropic SDK, Vercel AI SDK, LlamaIndex, or custom implementations, with OpenTelemetry support for existing pipelines. That breadth cuts both ways, and it is worth being honest about which way it cuts for you.

Trace volume, retention, and the exit-cost math

LangChain states that LangSmith has a free tier for development and small-scale production, and that paid plans scale with trace volume. Trace volume is therefore the cost dial: every run you trace, including every eval experiment, counts.

The specific numbers below come from a third-party guide, not from LangChain itself, and may be out of date. LangChain’s pricing page, not aggregated guides, is the source for these figures; verify each one there before budgeting against them:

The exit-cost question deserves a direct answer. If you adopt LangSmith, your datasets, traces, and feedback live in a vendor platform. The same vendor pages note that LangChain will not train on customer data and that customers own all rights to their data, which addresses ownership on paper. But ownership is not portability. Whether you can get your traces out in bulk, on a schedule, rests on the export claim from that single unverified third-party source. Before standardizing, confirm the current export mechanics yourself; this is a bounded verification step, and it is cheap. Assuming export works because a guide said so is not the same thing.

The framework breadth raises a subtler exit cost. Because LangSmith traces OpenAI SDK, Anthropic SDK, and Vercel AI SDK applications too, it is easy for the platform to become the observability layer for your whole stack, not just the LangChain parts. That is convenient, and it also means switching later means re-instrumenting everything, not just swapping an eval harness.

LangChain also claims its SDK uses an async callback handler sending traces to a distributed collector, so application performance is never impacted and agents keep running if LangSmith has an incident. That is the vendor’s self-assessment. The architecture described is plausible and is the standard way to isolate tracing from the serving path, but “never impacts performance” is a marketing absolute, and you should load-test the claim rather than adopt it.

Verdict

Wire the loop the way the docs describe: pass your runnable directly into evaluate() or aevaluate(), and make every example input’s keys match the chain’s input variables exactly. Those two rules are well-supported, and for most teams they are the entire blocker between reading about evals and running one.

Standardizing is a separate decision from getting a first harness running. I would run the harness on the free tier against workloads you can afford to re-run, and treat two things as homework before LangSmith becomes load-bearing: confirm where your account’s data actually resides given the default us-central-1 storage, and verify current pricing, retention windows, and export behavior against LangChain’s own pricing page and docs rather than any aggregated guide. If your eval target is a LangGraph graph, the runnable-evaluation docs carry a section for that; read it rather than adapting the chain instructions by hand. If a lost trace history or a residency violation would be expensive for you, the hosting and export answers matter more than anything evaluate() does.

Frequently Asked Questions

What are the two requirements for wiring a first eval harness in LangSmith?

pass a LangChain runnable directly into evaluate() or aevaluate(), and make every example input’s keys exactly match the chain’s input variables.

Where is data stored by default when using hosted LangSmith?

When using LangSmith hosted at smith.langchain.com, data is stored in GCP us-central-1.

Does LangSmith support tracing for applications built with other SDKs?

The same page notes that LangSmith tracing is not LangChain-only: it covers applications built with the OpenAI SDK, Anthropic SDK, Vercel AI SDK, LlamaIndex, or custom implementations, with OpenTelemetry support for existing pipelines.