How to Create a Take-Home Coding Challenge Developers Love

How do you create a take-home coding challenge that developers love?

We’ve been part of hundreds of technical hiring processes. And since our challenges have a 92% completion rate, we’ve perfected take-home coding challenges so that they produce the best results for your hiring team while keeping candidates engaged. Here’s how, why, and when to design a coding challenge that helps you attract and identify the right talent to your team.

Ready to learn more? Read on!

Take-home coding interviews

Is there a reason why hiring teams should use take-home coding challenges instead of, say, some other type of coding assessment test? And why do technical hiring teams even need a coding skill test? Here’s what you need to know:

Why hiring teams need to use a coding interview

The reason hiring teams need a coding skill test is to assess developers’ skills and make the best hire for your team.

It’s hard and expensive to hire developers. The wrong hire could potentially cost you 30% of that person’s first-year salary. CareerBuilder reports that companies lose $14,900 on every bad hire. That number goes up for high-earning devs.

Technical hires are some of your most expensive team members.

Let’s say you pay $70,000/year for a junior developer (plus benefits). You would shell out at least $21,000 for a bad hire. Not to mention your indirect costs, like lowered employee morale and lost team productivity.

At the same time, 80% of technical hiring managers say there’s a lack of talent and 41% say it’s becoming more difficult to attract technical talent.

It’s your job as a member of the hiring team to optimize the hiring process so that you choose the right people AND you offer a great candidate experience. Otherwise, the best talent will find a job elsewhere. It’s really that important.

What is a take-home coding challenge?

A take-home challenge is a skill assessment tool that lets you fairly and accurately test new developers. As plenty of research shows, well-designed work assignments correlate with job performance. That’s also why they’re so powerful for hiring the right candidate.

The process is simple:

Developers complete the task at home and you then assess that task.

Now you might ask, “what if candidates cheat?”

And the answer is yes, if you choose a tool with good plagiarism detection. For example, at CodeSubmit, we’re really focused on using safe plagiarism protection features in our product. At the same time, developers can use their own coding environment to complete the task, so they simulate a real-world coding task.

But take-home coding challenges aren’t the only options you have when hiring developers. Why are they so much more effective than alternatives like whiteboarding tests? That’s what we’ll look at next.

What are the alternatives?

There are different types of technical assessments and developer skill tests out there.

Let’s quickly run through the most common options:

Whiteboarding - Algorithmic quizzes that candidates complete in front of the hiring managers, usually on a whiteboard (and yes, that means that devs are handwriting their code). Whiteboarding is meant to reveal a candidate’s thought process, preparedness, and ability to communicate their ideas, but often results in a lot of stress and anxiety for the interviewee.

Live-Coding Interviews (e.g. CoderPad interviews) - The digital twin of whiteboarding, but it has almost the exact same format. A candidate is asked to solve a problem while the hiring manager watches and asks them questions along the way. These are better than whiteboarding because candidates write code on their computer (which is always better than on a whiteboard), but they’re still terrible for candidates when it comes to the stress and anxiety accompanying these interviews.

Pair Programming - Some sort of happy medium is achieved here in pair-programming, where a member of the hiring team and a candidate work on a task together. Either the candidate writes code and the interviewer reviews the code or they analyze a codebase together. When done well, these aren’t a bad option. When done poorly, they’re also terrible. It often comes down on the interviewer to provide a great experience.

Early-stage Screening Quizzes (e.g. HackerRank, Codility, etc.) - Coding quizzes, usually algorithmic (like reverse a binary tree), which are sent to candidates early in the interview process and are designed to filter people out. They’re also often timed. Literally nobody likes doing these. They are a necessary evil for Big Tech to set a minimum standard for the thousands of candidates that apply via their career portals. If you’re a startup, you definitely do not need these.

Okay, so here’s the massive problem with these types of coding tests:

Scientific studies (along with dozens of anecdotes from Tech Twitter) have shown that the aforementioned methods of technical assessment only measure how long people have prepared for the interview and how well they perform under pressure -- not their actual technical skills.

The phenomenon is further evidenced by the huge market for developer interview practice tools and guides. When it comes to these types of interviews, the candidate who spent the most time practicing algorithms on Leetcode and memorizing “Cracking the Coding Interview” front to back, wins.

Regardless of their actual technical ability or that of others interviewing for the same role, the most prepared person gets the job.

Interestingly, a lot of the platforms who offer these types of interviews also offer a developer tool to make money off of the folks prepping for their platform (looking at you, HackerRank, CoderByte, and CodeSignal.)

And that’s why we’re so bullish on take-home assignments - because they measure real candidate coding ability and they test what matters without all the anxiety and without all the bias.

Take-homes have the potential to offer a great candidate experience if used in the right way, which we’ll talk more about below. But first, some social proof. Here are just a few things our candidates have said about take-homes:

“The whole assessment process and hiring experience is just so much better for our hiring managers and candidates."

“A great way to test candidates. Very hands on approach and it is quite easy to see how relevant it is towards the job.”

What’s more, good take-homes facilitate diversity, equity and inclusion and make it easier to discover top developers with unconventional backgrounds. If your hiring team has a great take-home, then they don’t need to rely on signals like a Big Tech or Ivy League background to determine who to hire.

The world's first language learning app, Babbel, used take-homes when assessing their latest cohort of Babbel Neos, a program that champions diversity and is aimed at self-taught career-changers, graduates of developer boot camps, and recent computer science university graduates.

Aimee, the Neos Program Manager at Babbel says, “The feedback that we received from the candidates about their experience was overwhelmingly positive - specifically regarding the ease of the process, as well as the personal and engaging experience the process offered.”

Take-home challenges can be completed at candidates’ own pace and in the comfort of their own homes, making them more accessible to remote talent, talent with disability, or other special circumstances.

And because there’s nobody looking over your shoulder while you’re coding (except maybe your cat) take-home’s are much less stressful than their in-person alternatives.

So now you know why take-home challenges are a great way to assess your talent. But how do you use them in your hiring process?

That’s what we’ll look at next.

Integrating take-homes into your hiring process

Take-home challenges can make your hiring process much more effective and equitable. But for them to do so, your hiring team needs to use them in the right way. Here’s how.

Send take-homes at the right time

Sending a take-home too early in the hiring process is a kiss of death to any “better candidate experience” you may have been hoping to provide.

Technical interviews -even short ones- require investment from the candidate. Sending any type of work assignment as Step 1 in your hiring process, before you’ve ever talked with the person, is a terrible idea. It reflects poorly on your company and indicates an insensitivity and lack of respect for their time. Don’t do that.

For startups and SMEs with shorter hiring cycles and fewer candidates to sort through, a good hiring process may look like this:

Like I said, take-homes are a time investment for candidates. The most sought-after talent will not complete a time-consuming take-home if you send it to them before they know anything about your company.

That’s why your hiring team should also be invested in the candidate before you ask them to complete a take-home assignment. Your team should spend time discussing the role and laying out the entire interview process with candidates before they ask for work samples.

But what if you have the champagne problem of “too many” candidates? What if you need to filter out candidates before you spend time calling them up to discuss next steps?

If that's your team’s reality, then your best bet is to use a filtering tool before or after a phone screen. Remember that doing so may sacrifice the candidate experience. Your company must be an otherwise prestigious place to work, with a strong employer brand, for top-tier talent to be willing to put up with an arduous hiring process.

Once you’ve filtered out some candidates, you could move the promising ones onto a hiring manager interview and then request a (short) take-home challenge.

(Tip: CodeSubmit also comes with a filtering tool - but we’ve made our quizzes resemble daily programming tasks, which are more fun and more relevant than reversing a binary tree. They’re called CodeSubmit Bytes.)

Let's take a quick look to Basecamp as an example. Basecamp sends out their take-homes to fewer than 20 candidates at later stages of the interview process. They do have a problem with too many candidates. Over 1,000+ candidates apply each time they open a new position. They’ve got a stellar reputation as being a super cool place to work, receive thousands of applications, and still find the time to invest in their candidates and provide a great hiring experience with take-homes.

It requires work and intention, but it’s definitely doable.

Let's move on to setting the interview up.

Design a structured assessment process

Software is eating the world, but the world of HR, that's a good thing. Now that you know when to integrate take-home challenges into your hiring process, let's discuss the how.

With the right technical interview software, communication and collaboration is suddenly easy. We know that hiring is a team sport, so at CodeSubmit, everyone on your team gets a seat, irrespective of how large your hiring team is).

CodeSubmit integrates with Greenhouse and Slack and a whole slew of other great HR software via Zapier to make the interview process seamless for you and candidates.

Then of course, you might be wondering if you need to manually evaluate candidate submissions.

The answer is yes: someone on the hiring team needs to review the submitted assignments.

This is a non-negotiable best practice. But with the right features, the review process becomes easy and fun!

If that seems like a lot of work, think about it this way:

You’re hiring for a key role at your company. And because the take-home is the late-stage technical assessment, you’ve already filtered out a lot of candidates. The candidates who remain in the running are your top talent, but to identify the right one for the job, you need to assess their coding skill, the way they approach and work through problems, and their way of communicating and collaborating.

This is the optimal time to invest in your candidates. With their submission in hand, your hiring team has everything they need to make a data-driven, informed hiring decision.

So invest the time; it's an investment that pays back dividends.

How to design your take-home coding challenge

Alright, this is important: How do you design a take-home challenge?

First, here’s how not to design a coding challenge.

One of the main objections candidates have to take-home challenges is that they are perceived as free labor. Common mistakes that hiring managers make include tests that are too long and interviews that don’t provide feedback.

The problem is not just that a sloppy take-home provides a bad candidate experience. It also means that many of the best candidates won’t bother spending hours on your coding challenge, so you lose out on a lot of the top talent. Some will even leave the encounter with a bad taste in their mouth and complain about the experience on glassdoor. Nobody wants that.

So take-home coding challenges must be done the right way. We’ve found the right way - Our challenges have a 92% completion rate. What does it look like?

A great take-home challenge...

• Has defined evaluation criteria
• Is short (2-4 hours long)
• Is relevant to the role (but not unpaid labor)
• Is followed up with an interview or a feedback session

1. Define evaluation criteria first

Having defined evaluation criteria does a few things. First, you have an easier time evaluating people’s work. You also limit bias.

For example, here’s how Slack defines their criteria:

“The exercise is graded against a rigorous set of over 30 predetermined criteria. We’re looking for code that is clean, readable, performant, and maintainable. We put significant effort into developing these criteria to ensure that, regardless of who grades the exercise, the score is an accurate reflection of the quality of the work.”

Shoutout to Slack for shining a light on the fact that these criteria take a lot of effort to develop, but they’re worth it.

All that said, there shouldn’t be unnecessary limitations. As an example, Atlassian lets candidates use whatever services or code libraries they’d like.

2. Decide if you need a time limit

Another big reason developers object to coding challenges? They can be extremely time consuming (20-40 hours or even more).

Most developers aren’t applying to a single job, so imagine that they have your challenge, plus 2 or 3 others. It’s often a huge time investment to do these.

There are advocates who say you should pay candidates as freelancers for these challenges because few people have time for this much free work. This is a great idea, but also not doable for some companies. Alternatively, just keep the assignment short. Also keep in mind that the longer the assignment is, the more time your team should spend reviewing it.

At CodeSubmit, we recommend 2-4 hours as a good time limit. None of our challenges take longer than 6 hours to complete. Atlasssian has an 8-hour time limit and Basecamp estimates that their assignments take 3-5 hours to complete.

In some cases, your team might prefer to prescribe a range of time, rather than a hard time limit. Coding is a creative pursuit and a hard time limit can be distracting or stressful for some candidates. To make sure that candidates use a comparable amount of time on the challenge, your team can ask them afterward to disclose how long they spent on it. Most folks will be candid.

3. Choose a task that is relevant to the job

Pinterest uses challenges in technical interviews that are closely related to the role they’re interviewing for:

“We look for problem solving skills that show candidates can tackle real-world technical challenges like those they may encounter in their daily work.”

And it makes sense: it’s highly unmotivating for candidates to work on a task that they would normally never come across in the actual job.

On the flip side, YOU don’t get a complete picture of your candidates’ real-world skills unless the task matches the job description.

So make sure your task is relevant to the open role.

4. Follow-up after the interview

Last, discuss the challenge after the interview. Not only can you learn valuable things about your candidates (how they work on a problem, their communication skills, teamwork, etc.), but a great candidate experience requires it.

There’s nothing worse than professional ghosting. If your candidate has spent hours on a task, it is incredibly disappointing to get rejected without any further explanation. Or worse: never hear anything back at all.

By providing candidates with feedback, you ensure that they walk away understanding your decision and learning something for their next application. Why burn bridges, right?

Pro tip: Use the right tool

You can implement a take-home process without a software tool, but it’ll be a lot more work for your team to do it that way. Instead, choose a tool with the features you need for your take-home challenges.

Do you have your own coding assignment?

Or do you want to use a done-for-you question?

And what language or framework are you testing for?

Do you need integrations? What about the support that your coding assignment tool provides?

Make sure you understand your needs before you sign up for a tool. (Here’s our guide on the 20 best tools.)

CodeSubmit is great because it was designed for technical hiring teams with take-home challenges in mind.

Your team is able to create coding assessments specific to each unique open role. Every assessment contains evaluation criteria related to the work your engineers face in their daily work. Assignments can be fully customized to your engineering team’s stack. And collaborative review is suddenly a whole lot easier.

If you already use a take-home, you can easily upload it to CodeSubmit and start benefiting from a software-driven process. Think of the integrations! We offer a lot of them, as well as support at every step of the way, unlimited seats to your entire hiring team, and more.

One of the core aspects of CodeSubmit is that candidates can demonstrate their skills using the same tools they’ll be working with on the job (git, their own code editor or IDE, their favorite libraries). You assess their real-world skills and get to know what they’re capable of.

Some of CodeSubmit's features include:

• Library of coding challenges
• 60+ languages and frameworks (see the full list here)
• Free trial
• Flexible monthly billing
• Integrations
• Built-in plagiarism detection
• Screening tests (CodeSubmit Bytes)
• Candidates use their own tools and workflows
• Unlimited seats on all subscriptions

Examples of great take-home assignments

Curious to know what a coding challenge can look like?

Here are a few examples of CodeSubmit Challenges.

This one shows one for JavaScript using React:

And here’s one to assess a WordPress developer:

As you can see, there’s a clear task description and a recommended time limit. Plus, the tasks are fun and feel like assignments candidates would get on the job.

Over to you! There you have it. Now you know how to design and integrate a take-home coding challenge, when and why you should, to create an effective and candidate-friendly technical hiring process.

Is your team looking for the right take-home assignment tool?

Try CodeSubmit for FREE for 7 days (no credit card required).