Canditech's Code Project Questions give candidates a real, multi-file development environment where they can build, run, debug, and test code. With VS Code, a terminal, and optional built-in AI assistants in one workspace, you can evaluate how candidates approach the kinds of tasks they would encounter on the job.

This article walks you through creating a Code Project Question, configuring its environment and scoring, and reviewing the candidate's work afterward.

What Are Code Project Questions in Canditech?
How Do You Create a Code Project Question?
Which Tech Stacks Are Available for Code Project Questions?
How Do You Set Up the Project Files and Instructions?
How Are Code Project Questions Scored?
How Do You Manage AI Assistance in Code Project Questions?
What Is the Candidate Experience in a Code Project Question?
How Do You Review a Completed Code Project Question?
How Do You Configure Advanced Code Project Question Settings?
Learn More

What Are Code Project Questions in Canditech?

A Code Project Question lets candidates work on a complete coding project within a Canditech skills assessment. You choose a tech stack, customize the starting project, and write the task instructions. Candidates then work in VS Code directly in their browser, without setting up a local development environment.

Depending on the selected stack, the workspace can include a live browser or phone preview and a database. You can also make Claude Code and Codex available to evaluate how candidates use AI alongside their own coding and problem-solving skills.

After submission, Canditech can run your tests and (on Enterprise plans), apply custom AI scoring rules. You can review the final code, replay the candidate's progress, and read their AI assistant conversations.

Note: Code Project Questions are currently in Beta. If they are not yet available in your account, click the Code Project card to open the Request access form.

How Do You Create a Code Project Question?

Step 1 - Add a Code Project question: In the assessment builder, add a question to a section and select Code Project. Click How it works if you would like to view a short walkthrough.
Step 2 - Choose a tech stack: Choose a tech stack and click Create. Search tech stacks finds one by name, and each card shows the stack's version. The tech stacks are listed in the next section.

The tech stack picker for a new Code Project question

Step 3 - Edit in IDE: Wait while Canditech prepares VS Code in the browser, which can take a few minutes. It opens with a starter project for your stack.
Step 4 - Customize the project: Build the project candidates will start from. Edit the starter files, drag in your own files and folders, or use the built-in AI assistant to help prepare the project. Write the task instructions in GETTING_STARTED.md and configure the environment and scoring in .canditech/config.json. See the project files and advanced settings sections below for details. Once you are ready, click Next.

The Edit in IDE step with the starter project open

Step 5 - AI Scoring (Enterprise plans): Review the AI scoring rules defined in .canditech/config.json. To change them, return to Edit in IDE and update the configuration. Click Next.

The AI Scoring step showing the AI scoring rules read from .canditech/config.json

Step 6 - Validate: Canditech checks the project's required files, its configuration and file sizes. When you set a scoring command, it also starts the environment, injects your hidden tests and runs them. Each test and rule shows its share of the score.
Step 7 - Save the question: Click Save. If validation is still running, the button reads Save when validated and saves once every check passes.

The Validate step with every check passed

Note: Editing sessions are time limited. The time left is shown at the top of VS Code, so click Save before it runs out.

This is how the finished question appears in the assessment:

A finished Code Project question in the assessment, with VS Code and Claude Code

Which Tech Stacks Are Available for Code Project Questions?

Choose the stack your role uses. Each one comes with a starter project, a terminal and pre-configured run, debug and test tools.

Backend: TypeScript (Node.js), Python, Java, C#, Go, Rust, Ruby, PHP, Kotlin, Scala, Swift, C, C++, Elixir, Clojure, F#, Dart, Bash, R, Perl, Haskell, Lua, Groovy, Erlang, CoffeeScript, Julia and PowerShell.
Frontend: React, Vue, Angular, Next.js, and HTML, CSS & JavaScript, each with a live browser preview. Svelte is coming soon.
Fullstack: Express, Next.js, ASP.NET Core, Spring Boot, Flask, Django, FastAPI, Laravel, NestJS and Ruby on Rails. Each has a web front end, a live browser preview and a PostgreSQL database.
Mobile: React Native, with a live phone preview.
Database: MySQL, PostgreSQL and IvorySQL (Oracle-compatible).

How Do You Set Up the Project Files and Instructions?

Every project needs three files: GETTING_STARTED.md, .gitignore and .canditech/config.json.

GETTING_STARTED.md contains the task instructions and opens first for candidates. Clearly describe the goal, requirements, and expected deliverables. This file is read-only by default, unless you override the read-only settings.
.canditech/config.json controls the question. It sets hidden and read-only files, the scoring command, test weights and AI scoring rules. It also sets up the AI assistants, the files that open first and the dev server preview.
The .canditech folder is hidden from candidates. It also contains README.md, which explains the available settings, and can hold your AI scoring rule files.

See "How Do You Configure Advanced Code Project Settings?" below for configuration options and examples.

Note: Projects can hold up to 3,000 files and 100 MB, with no single file over 5 MB. A section can have one Code Project question, an assessment up to three, and a section that has one can run up to 12 hours.

How Are Code Project Questions Scored?

Hidden tests: your scoring command runs after the candidate submits, with your hidden tests put back in place. Each test case that passes earns its weight; by default every test counts equally.
AI scoring rules (Enterprise plans): an AI agent marks each rule pass or fail. It reads the candidate's project files and their AI assistant conversations. Rules and tests share one scale.
AI rules score only when at least half of the test cases pass. Otherwise every rule is marked failed.
If no test result can be scored, or the AI agent cannot complete its evaluation, the question requires manual scoring.

To learn more about the AI Auto-Scoring options available in Canditech, click here.

For more information on writing Custom AI scoring rules, click here.

How Do You Manage AI Assistance in Code Project Questions?

You can make Claude Code and Codex available to candidates inside VS Code. You set what they can use in the aiAssistant part of .canditech/config.json:

Turn the assistants off for the question.
Set the AI budget the assistants share for each candidate.
Limit which AI models candidates can choose.

Candidates see their AI budget above VS Code, with an AI rates button listing each model's price. The status bar warns them as the budget runs low, and the assistants stop answering once it is spent.

See "How Do You Control the AI Assistants?" in the Advanced Settings section below for configuration details.

What Is the Candidate Experience in a Code Project Question?

Step 1: While the environment starts, the candidate sees Preparing your development environment and can click Watch walkthrough. This usually takes about a minute and doesn't count against their time.

The candidate's Preparing your development environment screen with the Watch walkthrough button

Step 2: VS Code opens on GETTING_STARTED.md. Fullscreen gives it the whole screen, with the timer still visible.
Step 3: The candidate writes code, uses the terminal, installs packages and runs tests. Work saves automatically, and the Source Control view shows their changes.
Step 4: If working in fullscreen, they click Exit fullscreen, then Submit assessment when they are ready to submit the assessment. When time runs out, their work is saved and submitted.

Note: If the environment can't start, the candidate is told their time hasn't started and can refresh to try again. Canditech also retries on its own.

How Do You Review a Completed Code Project Question?

Open the candidate's report and scroll to the question.

Choose Final Answer to see the project as submitted, or Play Progress to replay how the candidate worked. Initial → Final shows what they changed.
Open Chat to read the candidate's conversations with the AI assistants.
Expand Test Cases and AI Results to see each test and rule passed or failed, with the AI's reasoning.
Click the download button, Download code and AI transcripts, to get the project and every AI conversation as a zip file.

A candidate's Code Project question in the candidate report

For more on reviewing a candidate's report, click here.

Note: Code Project Questions can also be used in Live Coding Interviews.

How Do You Configure Advanced Code Project Question Settings?

Use .canditech/config.json to customize file access, scoring, AI assistance, and the development environment. Starter projects include settings for their selected tech stack, which you can adjust in the Edit in IDE step.

What Is .canditech/config.json?

.canditech/config.json is the file you edit to configure a Code Project Question in your project. It lives in the project you build in VS Code in the browser, in the Edit in IDE step. Canditech reads it whenever the question is validated, and again for every candidate session.

The whole .canditech folder is hidden from candidates. They never see your settings, rules or reference files.
Every starter project also has .canditech/README.md, which opens beside your project and describes each setting. The sections below explain these settings in more detail.
The Validate step checks the file. A misspelled or unknown setting fails validation, and the message names the setting and what is wrong.

The Edit in IDE step with .canditech/README.md open in VS Code

Each starter project comes with settings for its tech stack. A Python project, for example, can hide its hidden tests and score them with pytest:

{
  "hiddenPaths": ["tests/hidden/**"],
  "autoScoring": {
    "command": "cd ~/project && pytest --junitxml=results.xml || true",
    "outputs": [{ "format": "JUnit", "path": "~/project/results.xml" }]
  },
  "aiAssistant": { "isEnabled": true }
}

A setting you leave out uses its default, described in its section below.

How Do You Hide Files from Candidates? (hiddenPaths)

hiddenPaths lists files and folders removed from the candidate's workspace before the session starts. Use it for hidden tests and anything else candidates should not read.

"hiddenPaths": ["tests/hidden/**"]

Paths are relative to the project root. * matches any characters except /, and ** matches any characters including /.
Hidden files are put back at scoring time, so your hidden tests still run against the candidate's code.
The .canditech folder is always hidden, whether or not you list it.
GETTING_STARTED.md, .gitignore, .vscode/settings.json and .vscode/launch.json can't be hidden.
Paths can't contain ...

How Do You Make Files Read-Only? (readOnlyPaths)

readOnlyPaths lists files candidates can open and read but not change. The tab shows a lock, and VS Code refuses to save an edit.

"readOnlyPaths": [
  "GETTING_STARTED.md",
  "init.sh",
  "pre-init.sh",
  "src/api/contract.ts"
]

Without the setting, GETTING_STARTED.md, init.sh and pre-init.sh are read-only.
Setting it replaces that default list. Include those files again if they should stay locked, or use [] to make every file editable.
It uses the same patterns as hiddenPaths. A pattern that matches the whole project, such as **, fails validation, so list specific files.
It guards against accidental edits. A candidate can still change a file from the terminal, so use hiddenPaths for anything they must not see.
It applies to candidate sessions only. While you build the question, every file stays editable. It doesn't affect scoring.

How Do You Configure Automated Test Scoring? (autoScoring)

autoScoring sets the command Canditech runs after the candidate submits, and where that command writes its test results.

"autoScoring": {
  "command": "cd ~/project && pytest --junitxml=results.xml || true",
  "outputs": [
    { "format": "JUnit", "path": "~/project/results.xml" }
  ]
}

command is the shell command that runs your tests inside the candidate's workspace. Set it to null to turn off test scoring.
outputs lists the result files to read. Each entry has a format, "JUnit" or "XUnit", and a path to the XML file, starting with ~/project/.
When a command is set, outputs needs at least one entry.
End the command with || true so failing tests still produce a results file instead of stopping the run.
The Validate step runs the command against your own project and lists each test case it finds, with its share of the score.

See "How Are Code Project Questions Scored?" above for how test results and AI rules contribute to the score.

How Do You Set Test Case Weights? (autoScoring.testCaseWeights)

By default every hidden test case counts the same. testCaseWeights makes some count more or less than others.

"autoScoring": {
  "command": "cd ~/project && pytest --junitxml=results.xml || true",
  "outputs": [
    { "format": "JUnit", "path": "~/project/results.xml" }
  ],
  "testCaseWeights": {
    "tests.hidden.test_api::test_concurrent_booking": 3,
    "*::test_smoke*": 0.5
  }
}

Weights are relative. A test case at 3 counts three times as much as one at 1, and 0.5 counts half.
Any test case you don't list has a weight of 1.
A weight is a number greater than 0 and at most 100. Decimals are allowed.
The key is the test case's name as the results file reports it: classname::name, or just name when there is no class. This can differ from what the test runner prints. For example, pytest prints tests/hidden/test_api.py::test_concurrent_booking but reports tests.hidden.test_api::test_concurrent_booking.
The Validate step lists the exact names it found, so copy keys from there.
* in a key matches any characters, including /. An exact name wins over a pattern, and the first matching pattern wins over later ones.
A key that matches no test case fails validation, so a renamed or mistyped test is caught.

How Do You Add AI Scoring Rules? (autoScoring.aiRules)

AI scoring rules grade what tests can't check, such as structure, naming, error handling or how the candidate approached the task. After the candidate submits, an AI agent marks each rule pass or fail. It reads the candidate's project files and their AI assistant conversations.

"autoScoring": {
  "command": "cd ~/project && pytest --junitxml=results.xml || true",
  "outputs": [
    { "format": "JUnit", "path": "~/project/results.xml" }
  ],
  "aiRules": [
    { "ruleText": "Endpoints follow REST conventions" },
    { "ruleFile": ".canditech/rules/error-handling.md", "weight": 3 }
  ]
}

Each rule has exactly one of ruleText, the rule itself, or ruleFile, the path of a Markdown file that holds it.
For longer rules, use a separate Markdown file to make the instructions easier to write and maintain.
A rule file must be inside the .canditech folder, so candidates can't read it. A path anywhere else fails validation.
A plain string is the same as ruleText. "Endpoints follow REST conventions" and { "ruleText": "Endpoints follow REST conventions" } are the same rule. A string is always read as the rule, so a rule can mention a file name.
weight is optional and works like test case weights: greater than 0, at most 100, and 1 by default. Rules and test cases share one scale, so a rule at weight 3 counts as much as three test cases at weight 1.
The AI Scoring step of the question shows these rules, read from this file.

Note: AI scoring rules are part of Enterprise plans. They score only when at least half of the test cases pass; otherwise every rule is marked failed.

For tips on writing AI Auto-Scoring rules, click here.

How Do You Control the AI Assistants? (aiAssistant)

Candidates can use Claude Code and Codex in VS Code when enabled. aiAssistant turns them on or off, sets their budget and limits the models candidates can choose.

"aiAssistant": {
  "isEnabled": true,
  "allowedModels": ["openai/gpt-5.4", "anthropic/claude-sonnet-5-5"]
}

isEnabled is required. Set it to false to remove AI assistance from the question.
usdBudget is optional. It sets how much the assistants can spend for each candidate, in US dollars. Without it, the default budget applies. The Validate step reports a value outside the range your account allows.
allowedModels is optional. Without it, candidates can use every supported model. With it, they can use only the models listed, and any other model is refused, even when typed in by hand. An assistant with no allowed model is removed from the workspace.
A model ID combines the provider and model name. The Supported model ids list in .canditech/README.md always shows the current models.
When a model is retired, its id leaves that list and validation rejects it. A session whose allowed models are all retired starts without AI.
Claude models (anthropic/…) are available when Claude models are turned on for your account.
allowedModels applies to candidate sessions and previews. While you build the question, you can use every model.

Candidates see their remaining budget above VS Code, and the assistants stop answering once it is spent.

How Do You Choose Which Files Open First? (defaultOpenFiles)

defaultOpenFiles lists the files that open as tabs when the candidate first enters the workspace.

"defaultOpenFiles": ["src/App.tsx", "GETTING_STARTED.md"]

GETTING_STARTED.md always opens first, in the focused tab, whether or not you list it.
Paths are relative to the project root. They can't start with / or contain ...
You can list up to 20 files.
A file hidden by hiddenPaths can't be listed.

How Do You Configure Dev Servers and Live Previews? (devServers)

devServers lists servers that start automatically when the workspace loads, after dependencies are installed. Each runs from the project root. Leave it as [] for stacks without a server.

"devServers": [
  { "command": "npm run dev", "previewUrl": "http://localhost:5173" }
]

Each entry has these fields:

command (required) starts the server, such as npm run dev or ng serve.
previewUrl (optional) marks the server the preview pane shows. Canditech waits for it before opening the preview.
readyUrl (optional) is a URL that scoring waits on before it runs the tests. Any response below 500 counts as ready. Set it on a backend without a preview, especially a slow-starting one such as Java or .NET.

A server with neither previewUrl nor readyUrl starts, but nothing waits for it.

Step 1 - Bind the server to all interfaces: The server must listen on 0.0.0.0, not the default localhost. Otherwise the preview never opens, even though the server is running. For example, use -b 0.0.0.0 for Rails, or server.host: '0.0.0.0' (or npm run dev -- --host) for Vite.

Step 2 - Accept the preview's host name: The preview loads through a Canditech address in the candidate's own browser. Frameworks that check the host name refuse it with a 403 error until you allow it:

Vite: server.allowedHosts: true.
Angular: the same key under serve.options in angular.json.
Rails: config.hosts.clear in development.

Step 3 - Reach the backend through a relative path: In the candidate's browser, localhost is their own computer. Call the backend through a relative path that the dev server forwards, such as /api/… with Vite's server.proxy. This also removes the need for CORS.

For a fullstack project, list each server. Give the frontend a previewUrl and the backend a readyUrl, so both are running before scoring starts:

"devServers": [
  {
    "command": "cd backend && mvn spring-boot:run",
    "readyUrl": "http://localhost:3000/"
  },
  {
    "command": "cd frontend && npm run dev",
    "previewUrl": "http://localhost:5173"
  }
]

Only the previewUrl server is reachable from the candidate's browser. The frontend reaches the backend through the dev server's forwarding, such as /api/… to localhost:3000.

How Do You Add a Database? (database)

database starts a database next to the workspace, reachable from the candidate's code at localhost. Leave it as null when the project doesn't need one.

"database": { "engine": "postgres" }

engine is "postgres" (PostgreSQL 16), "mysql" (MySQL 8.4) or "ivorysql" (IvorySQL 5.6, PostgreSQL that also accepts Oracle syntax).
Every session gets a fresh database, and scoring gets another fresh copy. Anything the candidate writes is discarded before scoring.
Each database starts with the same sample e-commerce data: the tables customers, products, orders and order_items. Write your task and tests against these tables.
Candidates connect as the app user, with full access to the app database only.

The connection details are already set in the workspace, so most frameworks pick them up on their own:

postgres: DATABASE_URL=postgresql://app:app@localhost:5432/app, plus PGHOST, PGPORT, PGUSER, PGPASSWORD and PGDATABASE.
mysql: DATABASE_URL=mysql://app:app@localhost:3306/app, plus MYSQL_HOST, MYSQL_TCP_PORT and MYSQL_PWD.
ivorysql: DATABASE_URL=postgresql://app:app@localhost:1521/app, plus the same PG… variables. Port 1521 accepts Oracle syntax and PL/SQL, and port 5432 on the same database accepts plain PostgreSQL.

Your autoScoring command connects the same way, for example with a driver that reads DATABASE_URL.

Learn More

Learn more about Canditech's Coding Project Questions.