Every role is different, and so is the definition of a good answer. Canditech’s custom AI Auto-Scoring agents let you score open-text, video and coding questions according to your own hiring needs and standards.
You build your custom agent by adding scoring rules that define what matters for the role, such as specific knowledge or the right approach to a scenario. The AI then evaluates every candidate’s answer against those rules.
The result is scoring that reflects how your team evaluates candidates, applied consistently to every answer, while eliminating the need for hours of manual review.
In this article, we'll explore:
How Do You Create a Custom AI Auto-Scoring Agent?
Open the question you want to score and select the AI Auto Scoring tab.
Click Build custom agent.
Enter your first scoring rule in the text box. Describe one specific requirement the candidate’s answer should meet.
Click Add rule to add it to your custom rule.
Repeat this process for each additional rule, clicking Add rule after entering each one.
Review your rules, then click Save to save the custom agent for that question.
What Are Scoring Rules in a Custom AI Auto-Scoring Agent?
Your custom agent is made up of scoring rules. Each rule is a short statement describing one thing you’d expect to see in a good answer. Together, these rules define what the agent looks for when evaluating candidates’ answers.
For example, a scoring rule might be:
“The candidate recommends at least one way to reduce customer wait times.”
For each rule in your custom agent, the AI reads the candidate’s answer and decides whether it passes or fails.
How Does Your Custom Agent Calculate the Score?
The score is the percentage of rules the candidate passed:
For example, if a question has 5 rules and a candidate passes 4 of them, their score is 80%.
A few things to keep in mind:
Each rule is pass or fail, with no partial credit. If a rule asks for three examples and the candidate gives two, the rule isn’t met.
Every rule contributes equally to the score. No single rule is worth more points than another.
How Does Your Custom Agent Interpret Scoring Rules?
The AI evaluates answers based on exactly what each rule says. It can't guess what you meant or fill in missing information. That's what makes it consistent, but it also means how you phrase your rules is crucial.
In practice:
Vague rules are too easy to pass. A rule like “The candidate shows good judgment” can be met by almost any answer, because even a weak answer may demonstrate some judgment. A more specific rule, such as “The candidate escalates the refund request to a manager because it exceeds the $500 limit”, sets a clearer standard.
Specific requirements must be met in full. If your rule asks for “at least three steps”, an answer that gives only two steps won't pass, even if both are excellent.
The AI only has the information you provide. It can see the question and the rule, but not any other materials. If a rule depends on information that isn't included in the question, you need to include it in the rule. (More on this below.)
How Should You Structure a Scoring Rule?
Each scoring rule in your custom agent can be as simple as a single sentence. If it needs more explanation, we recommend using the following three-part structure:
1. Statement: Describe what the candidate needs to say or demonstrate to pass the rule. Be as specific as possible about what counts as a correct answer.
2. Context: Provide any relevant background information, such as details from the scenario or attached materials, that the AI needs to understand why the answer is correct.
3. Examples: Give one or two examples of answers that would pass the rule and, if helpful, one that wouldn't. Make it clear that these are just examples, so other correct answers can still pass.
Here's an example of what a complete rule might look like:
Enter all three parts together in the same rule box, then click Add rule to add the complete rule to your custom agent.
How Do You Write Effective Rules for Your Custom Agent?
Start with What the Question Asks
Identify what the question asks candidates to do, then write rules to assess each part. For example, a question asking candidates to “identify the problem and recommend a fix” would naturally lead to at least two rules: one for identifying the problem and one for recommending a fix.
Make sure your rules assess only what candidates were asked to do. For example, if the question doesn't ask candidates to consider risks, including a rule about risks would penalize them for something they weren't asked to address.
Set the Bar at a Good Candidate
Write rules that a qualified, capable candidate would be expected to pass. A full score should be achievable by a good candidate, not reserved for an exceptional or perfect one. Avoid rules that require unusually specific details or exceptional insights that go beyond what you'd reasonably expect from someone qualified for the role.
Aim for One Idea per Rule
Since each rule is either pass or fail, combining multiple requirements into a single rule means a candidate could fail it even if they meet some of them.
❌ Instead of one rule:
“The candidate suggests a solution AND considers its risks.”
✅ Use two separate rules:
"The candidate recommends a specific solution.”
“The candidate identifies at least one risk associated with that solution.”
Describe the Specific Behaviors or Details You Want to See
Words like good, strong, or effective are subjective, and the AI may interpret them too broadly. Instead, describe the specific behaviors or details you expect to see in the candidate's answer.
❌ Instead of:
“The candidate shows empathy toward the customer.”
✅ Use:
“The candidate acknowledges the customer’s frustration before offering a solution.”
❌ Instead of:
“The candidate presents a good project plan.”
✅ Use:
“The candidate’s plan includes a deadline for each step.”
Include the Facts the Rule Depends On
The AI reads the question and the candidate’s answer, but it doesn’t have access to the section introduction or attached files. If a rule relies on information from these sources, make sure to include the relevant details in the rule itself.
Example:
Imagine a question that asks candidates to review an attached return policy and decide whether a customer who purchased an item 45 days ago is eligible for a refund.
You might write a rule like:
“The candidate correctly determines whether the customer is eligible for a refund.”
But there's a problem: The AI hasn't seen the return policy, so it has no way of knowing whether the candidate's decision is correct.
Instead, include the relevant policy information directly in the rule:
“According to the return policy, refunds are only allowed within 30 days of purchase. The candidate states that the customer is not eligible for a refund because the purchase was made 45 days ago.”
Clarify Which Answers Are Accepted
Sometimes, more than one answer can be correct. When your rule includes a list of acceptable answers, make sure to specify whether only the answers on the list are accepted, or whether other valid answers can also be accepted.
If only the listed answers are accepted, make that clear:
“The candidate identifies at least one of the following causes of the delivery delays: the warehouse staffing shortage, the broken sorting machine, or the new courier contract. No other causes are accepted.”
If the list contains examples rather than all possible correct answers, make that clear too:
“The candidate suggests at least one specific way to reduce customer wait times, such as adding chat support, hiring seasonal staff, or offering a callback option. Other practical, specific suggestions are also accepted.”
If you want to award more points for identifying more correct answers, you can create a series of rules with increasing requirements:
Rule 1: “The candidate identifies at least one of the following causes: …”
Rule 2: “The candidate identifies at least two of the following causes: …”
Make sure to repeat the full list of acceptable answers in each rule. The AI evaluates each rule independently, so referring to “the same causes as above” won't work.
How Do You Write Scoring Rules for Coding Questions?
The same principles apply to coding questions. Rules should focus on specific, observable aspects of the candidate's code.
✅ Good examples
The solution uses recursion.
The solution does not use a built-in sorting function.
The function returns an empty list when the input is empty.
The solution handles a user ID that doesn't exist.
❌ Too vague or subjective
The code is elegant.
The solution is efficient.
If you want to assess something like efficiency, specify exactly what you're looking for. For example: “The solution iterates through the list only once.”
What Should You Check Before Saving Your Custom Agent?
Before saving your custom agent, review its scoring rules using this checklist:
✅ Does each rule focus on one idea?
✅ Is it clear what a candidate needs to say or demonstrate to pass each rule?
✅ If a rule lists acceptable answers, is it clear whether other examples are also accepted?
✅ If a rule relies on information from attachments or the section introduction, are the relevant facts included?
✅ Would a correct answer still pass if the candidate expressed it differently?
✅ Would a qualified, capable candidate be able to pass most of the rules?
Tip: After your first few candidates complete the assessment, review the AI's explanations for why each rule passed or failed. If a rule isn't evaluating answers as you intended, try making it more specific or adding an example.
Ready to get started? Head over to your question editor, go to AI Auto Scoring and click Build custom agent to empower your hiring process with precision!
Note: AI Auto-Scoring is available on Enterprise plans. If you're interested in enabling it for your account, please contact our team.