Originally published on the Brokee engineering blog (September 2023, as part of a monthly company update). The brokee.io domain was later sold and now redirects elsewhere; this copy was recovered from the Internet Archive, trimmed to the technical content, and lightly edited. Screenshots of the Terminal History and Asciinema Player UI did not survive recovery.

Brokee is a platform for technical skills assessments: candidates get access to live IT infrastructure and have to fix a broken application to demonstrate their skills with a specific stack. Previously, we gave reviewers a text-based history of the candidate’s activity — which commands they ran while troubleshooting configuration, environment, or security issues.

Recording of User Activity

The command history was useful for a quick glimpse of how a candidate approached a problem — where they got stuck for 5–20 minutes, when they needed to check documentation or search for a fix. But it lacked depth: we couldn’t see what happened while a candidate was editing a file, which made it hard to tell whether a change was relevant or the candidate was just staring at a file without making progress. Our automated evaluations already confirm whether the application ends up fixed, but knowing how it got fixed still matters.

So we integrated asciinema into the testing environment. Now the full session can be replayed with one click.

Nobody wants to spend 30–60 minutes rewatching a candidate’s terminal activity, though, so we suggest reviewers do this instead:

• Take a 1-minute look at the command history
• If something looks interesting or unusual, jump to that timestamp in the recording to see what was actually happening

That’s enough to understand how a candidate solved a specific problem without the full replay.

AI-Generated Summary of Candidate Performance

We also started using AI to summarize evaluations. Command history and session replay help technical reviewers understand how a candidate worked, but less technical members of a hiring team often can’t parse raw terminal activity — so a video or command log alone isn’t enough for them to make an informed call.

The fix: generate a structured, plain-language summary from the same session data. It gives every reviewer — technical or not — a shared understanding of which tools the candidate used and what for (networking, file editing, security configuration, etc.), what changes they made, and which configurations they touched. Since there are usually multiple valid ways to fix the same problem, even experienced engineers can pick up something new from the summary if a candidate used an unfamiliar tool.

This makes the automated evaluation double as both a performance report and, incidentally, educational material for the team reviewing it — a useful way to compare how different candidates solved the same problem.

Reporting Recap

Where this leaves our reporting, for now:

• Test completion status (passed/failed)
• Time taken to complete the assessment
• Number of completed subtasks
• Automated checks
• AI-generated summary of user activity
• Command history
• Terminal activity recording

More advanced reporting is on the roadmap, but this is where things stand.