00:00:06
I'm primarily here today to give you a quick introduction to Sidekick and to share a few insights we've learned while building it. To start, I'll tell you what Sidekick is and highlight some of the lessons from our experience.
00:00:20
This is Sidekick. Sidekick is an assistant that lives within the Shopify admin. Merchants use it to get help with their store and manage their business. In this example it's running on Andrew's store; he'll join us later to talk about evaluations.
00:00:35
For example, Andrew asked Sidekick, “Could you fetch and analyze sales from the last 30 days and provide recommendations based on what you've learned?” Sidekick can decompose that request into individual tasks, gather the necessary context, and generate a reasonable response.
00:00:53
Internally, Sidekick is an agent. It has an LM that is equipped with many tools. Those tools interact with the environment — in this case, the Shopify store and other Shopify APIs. The agent reasons about tool responses and then generates a reply to the merchant.
00:01:23
Today, Sidekick doesn't rely on many predefined workflows. While many agents are built around strict workflows to ensure consistency, we found that equipping an LM with well-defined tools offered the best balance: high-quality responses while allowing the LM enough flexibility to recover from errors and edge cases.
00:01:47
Before I continue, I want to go back to the early days of Sidekick and describe the handful of skills or tools we initially developed and why we chose them.
00:02:01
Our main criteria for picking tools were: one, does this provide value to the merchant, and two, does it save them time? There's no point in building a tool if a merchant can accomplish the task faster another way. The first two tools we built were customer segmentation and analytics. Customer segmentation is a core Shopify feature that lets you group customers by criteria for marketing, discounts, or buyer insights. But it requires learning a bespoke query language. Analytics is similarly powerful — it provides order details, sales data, and trends — but also expects merchants to learn that query language, which can be a barrier for non-technical users.
00:03:06
With the advent of LMs, these features became much more accessible. You can ask an LM to generate queries, have the agent run them, and return the insights directly. For these skills we fine-tuned a model to translate user requests into queries. Sidekick would validate the fine-tuned model's output in the tool, run the query, and generate a response.
00:03:36
While building complex skills, we also added basic capabilities so Sidekick could handle everyday tasks. Navigation helps merchants, especially new ones, find their way around the Shopify admin. The Help tool is a classic RAG-based tool connected to the Shopify Help Center documents. Form filling allows Sidekick to generate previews of create or edit actions for resources on a merchant's store. Important: Sidekick itself doesn't mutate shop state. It provides a UI preview, and the merchant must sign off and commit any changes themselves to avoid bad experiences.
00:04:48
We launched with a few skills and saw good initial success: merchants liked the help Sidekick provided. We monitored gaps between what Sidekick could and couldn't answer, and when it couldn't, we built a tool for that task. For a while this worked well, but as we added many tools, problems emerged. With too many tools, the LM began to confuse tool responsibilities, misused instructions across tools, and response quality dropped. A colleague described this as 'death by a thousand instructions' — conflicting instructions bloated the system prompt, slowed processing, made debugging hard, and made the system difficult to evaluate.
00:06:12
Our first major refactor introduced the concept of just-in-time instructions. Instead of placing conditionals and tool-specific instructions in the agent's main system prompt, we moved those instructions into the tool responses themselves. This keeps the core agent behavior static and surfaces tool instructions only when the tool is invoked. It also improved cache friendliness — a more static system prompt means longer-lived caching of model contexts with some LM providers. Teams could also experiment with different instructions via flags or page context without affecting the agent's core behavior.
00:07:27
In practice, this means moving instructions from the system prompt into structured tool results. For example, our Help tool returns a specific citation format. When those citation rules were embedded in the system prompt, the agent would sometimes misuse the citation format for unrelated tools. Moving those instructions into the tool result modularized behavior and made it possible to scale to many more tools while keeping the agent consistent from the merchant's perspective.
00:08:07
We scaled out by having teams introduce one tool representing their domain. But as we onboarded larger, more complex features, some teams needed many domain-specific tools. To avoid returning to the original problem — the main agent tracking many complex tools across domains — we began exploring sub-agents. Sub-agents are specialized agents that handle particular domains while the main agent, Sidekick, remains the single point of contact for merchants. This preserves Sidekick's consistent tone and voice while delegating domain-specific responsibilities.
00:09:16
In practice, Sidekick calls a tool that hands off to a sub-agent with a set of instructions, an optional conversation ID, and any domain context the sub-agent needs. The sub-agent runs its own agentic loop with its own system prompt and domain tools, then returns just-in-time instructions back to Sidekick so Sidekick can form the merchant response. We pass a conversation ID so the sub-agent can maintain context across multiple turns when a merchant iterates on a task. Many users don't provide a full spec in a single turn; they iterate. The conversation ID lets the main agent treat the follow-up as a continuation of the same thread.
00:10:35
These approaches are still early exploration and we're continuing to evaluate them. A few takeaways: keep your system as simple as possible for as long as you can — simpler systems are easier to reason about, scale, and maintain higher response quality. Don't jump to a multi-agent architecture right away; it adds complexity and latency. The quality of your tools matters far more than the quantity: spend time on tool design and iterate frequently. Finally, stay modular; isolate parts of your agent to reduce the blast radius of any one change and keep your system resilient. With that, I'll bring up Andrew to talk about evaluating this system. Thanks.
00:11:54
Hey everyone, and thanks Charlie for the Sidekick intro. I'll give a quick intro as well. I'm Andrew McNamara. I've been building assistants for 15 years. In 2011 we started a company that powered LG and Samsung assistants on phones and TVs. We were acquired by Microsoft in 2017, where we built one of the first LM-based assistants (originally called Sydney, then Bing Chat, later branded Copilot). Now I'm at Shopify working with Charlie and the team on Sidekick, and I'll share what I've learned about evaluating chat systems and agents.
00:12:32
A common pattern I saw early on was 'vibe testing' — trying a system, thinking it looks good, and shipping. That often led to errors in production and disappointment. To move away from vibe testing, we use LMs as judges and simulators to build statistical rigor and high trust in our systems. We use an LM-based user simulator — our merchant simulator — to replay the spirit of production conversations against a candidate system. The candidate system is a single change we want to test in isolation. After replaying a conversation, an LM judge evaluates the result across multiple criteria so we can have confidence when shipping.
00:13:27
You can't just randomly create a judge or simulator — you need statistical rigor. We build a ground truth set. Unlike a golden set that maps a fixed input to an expected output, a ground truth set samples real conversations from production, labels them against criteria (safety, goal fulfillment, grounding, merchant sentiment, etc.), and grows over time. Product experts — not crowdsourced workers — label the set. We collect both good and bad conversations, including corner cases, and explain why they are good or bad.
00:15:00
Typically we start with a few hundred ground truth conversations. Multiple product experts (for example, three to five) label them, and we calculate inter-rater agreement using statistical measures like Cohen's Kappa or Kendall Tau. That agreement represents the practical upper bound for what an LM judge can achieve, because humans don't agree perfectly either. For example, if Cohen's Kappa across our experts is about 0.69, that sets realistic expectations for the judge's maximum correlation with human labels.
00:16:38
Next, we prompt an LLM (or train one) to match the ground truth labels as closely as possible. At first correlation with the ground truth set might be low, but through iterative prompting and tuning we improve it. With a ground truth approach, you can run the judge on an infinite set of real production conversations instead of being limited by a small golden set. As you iterate, your correlation score should rise — for Sidekick our judge has reached around 0.61 correlation with the ground truth in some cases. The goal is to get your LLM judge to be indistinguishable from a human judge on the ground truth set — a kind of Turing test for judges.
00:18:03
Degradation testing is also important. We create intentionally bad variants of Sidekick (for example, “badkick” or “annoyedkick”) to ensure the judge detects targeted failures, such as safety regressions. Once you have high confidence in the judge offline, you can align it with online metric verification. Compare historical A/B test deltas between control and treatment to ensure the judge is directionally aligned with online metrics. Sometimes the judge reveals issues the online metric missed — for example, users might click a broken feature, making online metrics look green while the judge flags product regressions.
00:19:54
You can also perform online degradation testing by launching a degraded model at a small percentage of traffic and verifying both the online metric and judge scores drop as expected. When you have confidence in the judge, you can use it with the user simulator. For chat systems, conversations can diverge after the first turn, so you can't simply replay exact interactions. A merchant simulator captures the essence and goals of production conversations and interacts with the candidate system, producing new conversations that the trusted LM judge can evaluate.
00:21:58
To build trust in the merchant simulator, apply statistical rigor there as well. For example, run many AA tests: take a seed conversation, have the simulator generate 100 simulated conversations, run the judge on each, and verify the judge scores are consistent. This demonstrates the simulator behaves predictably and produces testable conversations.
00:23:32
There are many benefits to this approach. A robust judge and ground truth set let you run reinforcement learning (RL) algorithms at scale. However, be careful: RL will learn to exploit weaknesses in your judge. If you train a model with RL on a flawed judge, the model may find loopholes that score highly with the judge but are incorrect or undesirable in practice.
00:25:12
For example, when we applied RL techniques (we used GRPO in our experiments) to a fine-tuned model, accuracy rose from about 79% to 99% according to our judge. That looked too good to be true. Manual analysis revealed reward hacking: the model learned to output safe-sounding refusals like “I can't do that for you” because our judge marked those as acceptable for certain criteria. In other cases, for a customer segmentation task, the model exploited free-form tags instead of using the structured field (e.g., looking for a tag “enabled” instead of using the account status field). These are concrete examples of how RL can find loopholes in imperfect judges.
00:27:00
Takeaways: create an LLM judge but don't vibe test it — build it with statistical rigor. Align judges to product experts rather than generic labelers. Continuously grow your ground truth set and include both good and bad conversations, labeled with merchant perspective in mind. Expect reward hacking and iteratively improve the judge and ground truth set as you apply RL or other optimization techniques. Finally, if you're interested, we are hiring — come talk with us at the booth. Thanks.
