My Eval's Playbook

I was one of the founding members building user-centric evaluation frameworks at Meta AI.

When I entered the GenAI space, most evaluation approaches were rooted in research benchmarks—valuable, but often disconnected from how users actually experience AI in the real

As an early member working on text response quality at Meta, I saw this gap clearly.

So I approached evals as a product problem, not just a measurement problem.

Evals are the control system for GenAI products. They connect model behavior to user value—shaping reliability, alignment, and overall experience.

A good eval doesn’t just measure quality; it drives it. It pushes models toward the right behaviors while leaving room for iteration and improvement.

There’s no single “right” eval.

Designing evals is an exercise in creative problem-solving. You build them based on:

• The behaviors you want to encourage

• The user signals that actually matter

Here’s an ops playbook to build great evals.

Stage	What	Estimated timeline	STO
Understand the Scope	You can’t solve everything in one eval—avoid trying to capture entire response quality at once. Clearly define the specific aspect of model behavior you’re trying to optimize.	<1 week	PM
Define Product Spec	Define what success looks like for your product and how the model is expected to behave. This is key to driving performance improvements and informing downstream decisions and team alignment.	1-2 Week	PM + CE
Build confidence on your spec strategy	Align with UXR, Content, Eng, and DS leads, and begin internal dogfooding. Ensure early agreement on the problem and path forward.	< 1 Week	PM
Identify the right eval strategy	There’s no one-size-fits-all eval. The structure depends on your specific use case and the model’s current performance. As performance improves, expect to evolve the eval to match new user needs.	<1 Week	PM + Eng
Prototype w seed examples yoursel	Create seed examples yourself and test the eval via dogfooding. Confirm it captures the behaviors you want to promote or avoid. Use early testing to build a failure-mode taxonomy.	1 Week	PM + Eng
Build the eval	Define clear guidelines and collaborate with vendors to build the eval. A good eval should reflect both current product traffic and anticipated future scenarios.	2-3 Week	PM + CE + Eng
Validate the Judge	Use golden human labels to validate the judge's accuracy. Target ≥70% precision and recall to ensure the model-based judge can reliably scale evaluation.	1-2 Week	Eng
Calibrate against Human Eval	Continuously monitor the judge's agreement with human evaluations, especially as model behavior or prompting evolves. Refine the eval pipeline as needed.	Ongoing	PM + CE+ Eng

Deep Dive starting Step 4+ Step 5⇒ Identify the right eval strategy + Prototype w seed examples yourself

1. Objective Capabilities with Prompt-Specific Behavior

(e.g., Instruction Following)

For capabilities where the model behavior changes based on the prompt, such as instruction following, a rubric-based eval is highly effective. The structure of the rubric depends on the nature of the instructions:. The rubric will  be adapted per prompt, where each prompt has different instructions to follow

2. Objective Capabilities with Consistent Behavior

(e.g., Citation Recall, Hedging)

These capabilities have uniform behavior expectations across prompts, allowing for a fixed rubric.

3. Subjective or Experience-Driven Capabilities

(e.g., Completeness, Personalization)

For capabilities that are more subjective or user-experience-driven, such as completeness or personalization, eval design becomes more nuanced.

To build a strong eval:

• First, decompose the capability into clear, observable sub-dimensions (e.g., relevance, tone, specificity)

• Use rubric-based evals to capture these core attributes

• Then complement with preference-based evals to assess nuanced aspects of satisfaction and overall user experience

Deep Dive Step 6⇒Prototype with seed examples.

There are multiple ways to test out the seed examples

1. Human Review  using google search, competitive analysis etc

2. One can use ChatGPT or other AI tools available to quickly test if the rubric is able to catch the behavior that the evaluation is trying to promote and/or avoid. Here are some quick steps you can try on the ChatGPT web UI yourself.

Here is the instruction you can use on Chatgpt/Claude/Gemini to judge your model's response quality based on defined rubrics

"This is User Prompt “X”, This is the response model generated “Y”, Judge this response on these  rubrics “Z”"

Here is an example of how I used it A- is the response from Meta AI, B- Rubrics from evals I build C- GPT out put when using above instruction

Deep Dive Step 7⇒Build the eval

1. Share a clear set of guidelines with the vendor, including a few seed examples to illustrate how you want the evaluation to be conducted.

2. Request a pilot with a few samples to verify whether it aligns with the provided guidelines. If there are discrepancies, update the guidelines to clarify them and work with the vendor to ensure alignment and a clear understanding of the requirements.

3. Once the pilot meets expectations, scale the evaluation. If it's a rubric-based eval, it's important to verify the rubric created by the vendor. You can verify rubrics using AI tools online. Here are some quick steps to do so using the GPT Web UI:

   1. Provide the prompt to GPT to see how it responds. Use the AI tool that aligns with the product feature you are developing.

   2. Once GPT generates a response, use this instruction: "Judge this response based on rubric 'X' and respond with Pass/Fail per the rubric." Replace 'X' with your rubric.

   3. If the rubrics are passing, it indicates that they are the right set of rubrics. However, it's possible that your product specifications differ from GPT's, in which case all the rubrics may not apply, or GPT may pass/fail on these rubrics. Use your own judgment along with GPT's results to assess whether the GPT response is accurate. Here is one example:- 

This was a FDD prompt so we also needed to give post context to GPT along with User prompt 

Step 1: This is the post'...............'

This is the prompt '....................' 

Generate a response