Exact match
Does the product meet the request: model, size, color, budget and delivery destination? Check the exact variant, not just a similar product.
Research · Coming soon
Agent Commerce Evals.
Finding a plausible product is easy. Finding the right variant, in stock, at the actual delivered price is the test.
We’re designing tests of how well agents find products across merchant platforms. Below is the proposed method. Results and agent comparisons are coming soon.
What we plan to measure
Price & delivered cost
Does the quoted price match the merchant, including shipping, tax and currency? Mark any total that cannot be verified as unknown.
Stock & freshness
Can that exact variant be bought when checked? Record the source and check time, then verify again at handoff.
Time & cost to answer
How long does a correct, verified answer take? Report response times, tool calls and token costs alongside accuracy.
Useful handoff
Can the agent take the right item to a supported purchase step? Report a successful handoff separately from a completed purchase.
Proposed method
Same task.
Same conditions.
Give each agent the same requests, delivery destination, currency and budget. Test across merchant platforms, including unclear requests and out-of-stock variants.
Compare browsing with catalog tools using the same model where possible. Record model versions, prompts, available tools and time limits. Repeat each task to distinguish consistent performance from a lucky answer.
Check answers against accessible merchant sources at the time of each run. Publish the evidence for each task, including failures and unknowns. Clearly label runs that use Index.
Tested agents, catalog coverage and a release date will be shared with the results. No scores or rankings have been published yet.
Building a commerce agent?
We’d like to hear which real shopping tasks it struggles with.