Canditech’s custom auto-scoring agents allow you to automatically evaluate open-text and video questions using a scoring rubric you define. This enables you to consistently assess responses without relying solely on manual review.
Using the "Build custom agent" option, you can create a custom scorecard that clearly defines what the agent should look for in a response. This may include specific concepts that must be mentioned, required skills or knowledge areas, expected depth of explanation, or other criteria tied to the question itself.
Custom agents can be applied across a wide range of use cases, from more general responses to highly specific or niche skill sets. They are particularly useful when you want scoring to reflect your internal evaluation framework, rather than a generic assessment approach.
Once configured, the agent applies your custom rubric consistently across all candidate responses, making it easier to compare results at scale while maintaining alignment with your scoring standards.
What is a "Custom Agent" (in Canditech terms)?
When we talk about building a "custom agent" for AI auto-scoring on Canditech, you're not coding an AI from scratch. Instead, you're defining a set of clear, objective rules that our underlying AI agent will follow to evaluate candidate responses. Each rule acts as an independent "check" for a specific aspect of the candidate's answer, resulting in a simple "Pass" or "Fail" for that particular point.
How is the Score Calculated?
The final score represents the percentage of rules the answer successfully satisfied. For example, if you defined 5 rules and a candidate met 3 of them, they will receive a score of 3 stars.
This means you set the evaluation standards, and our AI ensures they're applied consistently and fairly to every candidate.
Why Go Custom? The Power of Tailored Evaluation.
While our pre-built agents cover a wide range of roles and skills, building your own custom agents allows you to:
Save Valuable Time with Auto-Scoring: Automatically evaluate candidate responses, freeing hiring managers from manual reviews and accelerating decision-making.
Address Unique Business Challenges: Evaluate how candidates would tackle scenarios directly relevant to your company's operations.
Boost Accuracy & Fairness: Minimize human bias and guarantee every candidate is scored against the same precise, objective criteria.
Align with Company Culture: Ensure candidates' responses reflect your organization's values or preferred approaches.
Your Guide to Crafting Effective Auto-Scoring Rules
Creating effective scoring rules is straightforward. Follow these principles to build a custom agent that delivers accurate, insightful scores:
Principle 1: One Rule, One Clear Check
The Goal: Each rule should measure a single, distinct aspect of performance. Avoid combining multiple ideas into one statement.
Why it Matters: Our AI agent evaluates each rule independently. If a rule has two parts, and a candidate partially meets it, the AI might struggle to give a definitive "Pass" or "Fail."
Good Example ✅
"The candidate identified at least one negative trend, efficiency gap, or campaign outcome that fails to meet the target KPIs."
"The candidate acknowledges potential risks in their solution."
Avoid ❌
"The candidate suggests an innovative solution and considers all its potential impacts." (This is two checks: innovation + impact assessment.)
Principle 2: Focus on Observable Behaviors or Content
The Goal: Rules should check for things the AI can actually "see" or "read" in the candidate's written response (specific words, phrases, concepts, or structural elements).
Why it Matters: The AI interprets text. The more concrete and explicit your rule, the better the AI can apply it consistently. It can't infer emotions or intentions not directly expressed.
Good Example ✅
"The response includes a specific target audience for the marketing campaign."
"The candidate uses professional and respectful language throughout the response."
Avoid ❌
"The candidate demonstrates motivation." (“motivation” is too abstract).
Principle 3: Include All Necessary Context
The Goal: Ensure the rule contains all the information the AI needs to make a judgment, without requiring it to "remember" details from the original question prompt or previous rules.
Why it Matters: Each rule is evaluated independently. The AI doesn't carry context from outside the specific rule text. If your rule references something, it must be explicitly included or fully described within that rule.
Good Example ✅
(If the question mentioned a client named "Acme Corp"): "The candidate addresses 'Acme Corp' by name in their email draft."
Avoid ❌
"The answer gives 3 reasons for the chrome indicators presented above." (The AI won't know what "chrome indicators presented above" refers to because that context isn't within this specific rule.)
Principle 4: Align with Competencies & Question Intent
The Goal: Each rule should connect directly to the specific skills, knowledge, or judgment you intend to assess with that open-text question.
Why It Matters: Well-aligned rules help ensure that you’re evaluating what truly matters for the role, and nothing else. When scoring criteria drift away from the question’s original intent, you might unintentionally measure unrelated traits, which can make results less reliable and less consistent.
Example: If your question is designed to assess prioritization skills, a rule like “The candidate identifies the most critical task first” keeps the focus on the intended competency. Adding unrelated elements (like writing style or tone) could unintentionally skew results and reduce the accuracy of your evaluation.
Principle 5: Include Examples for Clarity
The Goal: Provide illustrative examples of what a "Pass" looks like, so any grader (human or AI) understands exactly what constitutes meeting the rule.
Why it Matters: This is crucial for consistent, objective, and fair scoring. Your examples help the AI understand the intent of the rule, not just the literal phrasing.
Example: "The candidate clearly explains why their chosen approach is superior to alternatives. (e.g., 'This method is faster because...', 'It's more cost-effective due to...')."
Principle 6: Avoid Subjectivity and Vague Terms
The Goal: Be precise. Replace general or interpretive terms like "good," "strong," "appropriate," or "effective" with specific, measurable indicators.
Why it Matters: Subjective terms are open to different interpretations, making consistent AI scoring difficult. Define what you expect in concrete terms.
Good Example ✅
"The candidate provides at least three distinct steps in their proposed action plan."
Avoid ❌
"The candidate gives a good answer."
Principle 7: Calibrate for Proficiency, Not Perfection
The Goal: Write rules that a solid, qualified candidate will pass. A score of 100% (5 stars) should indicate a "Strong Hire," not necessarily a rare, one-of-a-kind genius.
Why it Matters: If your rules are too strict or look for "extraordinary" nuances only, excellent candidates might receive average scores. The rules should cover the essentials of a good answer, ensuring that anyone who meets your standard for the role receives a high score.
Good Example ✅
"The candidate mentions at least one risk mitigation strategy." (A standard expectation).
Avoid ❌
"The candidate provides a highly unique and never-before-seen mitigation strategy." (Too high a bar; implies only outliers get full points).
By following these guidelines, you're not just creating a checklist; you're building a sophisticated, tailored evaluation framework that ensures your skill assessments collect deep, objective insights you need to pinpoint and hire the best talent for your organization.
Ready to start building your custom rules? Head over to your question editor and click "Build custom agent" to empower your hiring process with precision!
Note: AI Auto-Scoring is available on Enterprise plans. If you're interested in enabling it for your account, please contact our team.