An Explanation For Non Programmers On Why Programming Tests Are A Bad Idea

An Example Of Why Coding Are Tests Are Bad.

An imaginary programming test question
The problem with a programming test is that it usually becomes an exercise in does the Person doing the test write code in the same way as the Person setting the test.

With an added twist of did the person taking the test understand the unspecified parts of the test!

Here I am looking at a short test, something that might take 20-30 minutes and is done on the employer's site before or after an interview rather than one that takes 2-10 hours and is done at home.

Recently I was talking to a recruitment agency who mentioned that they had sent multiple candidates to an employer and they had all failed a programming test and all for the same reason.

They had answered the question asked in a sensible way but the company wanted it answered in an unreasonably complex way that might have been reasonable in the real world.

Programming Or Coding Tests as Part Of The Interview.

At first glance a programming test seems like a reasonable part of the recruitment process and why would anyone object to taking one?
The primary problem with programming tests is that it is almost impossible to create a good one that takes less than a day to complete.

As an example on this page I have shown what sounds like a reasonable short test question and provided three answers.

As this page is aimed at non programming recruiters it is not necessary to fully understand the example answers, although you probably will end up with a rough grasp. The important point is how different the answers are and how do you as the recruiter gain any useful information from the answers?

The second problem stems from the first, many good candidates simply won't take tests because they see them as a waste of time because the test can not tell the recruiter anything useful.

So any company that uses them is not one that they want to work for.

As a recruiter you may hate this but it does reflect the fact that the demand for good programmers has outstripped the supply for many years, COVID may have altered this balance but not the belief.

So a company that uses tests could be starting a self reinforcing loop, as a company that tests can only recruit from the pool of candidates that will accept testing then it is possible that it is mainly testing those less suitable for job.

This may lead to poor selections creating the view that more testing is needed to weed out the potential wrong choices, but this extra testing reduces the candidate pool even further.

My view is that if you want to do a useful test then you really need it to take a day or so. If a test is that long will a candidate have the time to take it and are you willing to regard that as time that should be paid for?


Answer My Comments
The "Best Answer" This is what I wanted to see, code done the way that I would do it, but the answer does a lot not required by the question.
The "Worst Answer" Answers the test question thinking that a minimal answer is what is wanted, which doesn't really tell the tester much.
The "Modern Answer" The question assumes too much reflecting the lack of experience by the test creator, the only possible answer is the way that I would do it. It is genuinely possible that the candidate is a massively experienced C# programmer who has never needed to populate a DataTable using an SQL datareader.


Remember COBOL?

ACOBOL compiler
Many tests still don't understand the difference between a programming langauge and run time libraries.

Back In The 1970s And 1980s

A long time ago programming languages were quite simple so if you were looking for a programmer with COBOL, FORTRAN or BASIC those skills could be tested for in a fair and reasonable manner.
Whilst there were many versions of each language, each hardware maker would have their own version, they all had quite limited capabilities and were all based on standards, except for where they weren't.

So if someone said they could program in say COBOL then there was a core part of COBOL that they must know regardless of their background and only one reasonable way to do something.

A competent test would also allow someone answering a question using IBM COBOL for a job writing ICL COBOL to pass.

At the same time there were also languages like C which are different in that there is a core language, again every C programmer would know most of the core and also optional libraries that aren't part of the language but could be viewed as extending the C language.

Most C language implementations include a library called STDIO which handles very basic reading and writing to the screen or disc. To repeat the above the library is not part of the language.

So it doesn't follow that a C programmer would had ever used STDIO, he may have worked somewhere where STDIO was replaced with XYZCorpIO which does the same job in a completely different way.

Hence it is not possible to infer anything from testing a programmer's knowledge of C by asking questions about a library such as STDIO unless he explicitly claims to be competent with that library.




C++ and assembler, "it's part of the language" as the compiler compiles it!

A bit of C++ and assembler
This was perfectly acceptable C++ in Visual Studio.

Programming Languages Nowadays.

The two key concepts described above no longer apply, this is especially true if you are recruiting developers to work with Microsoft Technologies.
There is no longer a core of a language that everyone should know, the C# and VB languages have expanded to the extent that they contain commands that a programmer could legitimately never use.

LINQ is a great example, it has become a "Marmite" part of the C#/VB languages, some people love it and some hate it.

Also the idea of a library being separate from the language has gone and the two have merged into one. In this example question many people would say that a datatable is part of the language even though it is part of a library.

Any C# test that includes LINQ, which is part of the language not a library, has the implicit assumption that the person taking the test has been using C# in an environment that supports its use.

The reverse is also true, an environment that uses LINQ may not use the technology that the test setter has set a question for.

Failing someone for never having used LINQ doesn't tell you that they can't learn to use it. Nor does failing them for not knowing the non LINQ way of doing it mean that they can't learn that way.

If you really want to discard a candidate for needing a couple of days familiarisation with a handful of new commands then that is your right.

Just remember that this is what doing tests probably implies to many candidates, so it may reflect badly on you as an employer.

So when you are creating or purchasing your tests you need to be aware that the concept of their being one correct answer no longer applies.

I have used LINQ as an example quite a lot because it is easy to do an internet search is LINQ good and see many postings that are easy, sometimes less easy, to read and understand.

Again it doesn't matter if you fully understand the arguments in these posts, the fact that there are so many highlights the possible issues.


Should A Test Reflect The Company's Business?

An imaginary programming test question
A real business working in the financial transactions arena used the game Lights On as a test.

A Real World Example Of A Test

Clearly the company knew what they were expecting but do they realise that they may have got the test wrong?
Full details of the test are here, whatever they were looking for my answer wasn't what they wanted.

Which I suspect puts the test into the third category, they were making assumptions and hadn't specified them in the test.


Other Interesting Pages On Some Of My Sites.
Car Insurance Premiums Explained
Generic car picture Car Insurance Premiums Explained This site aims to explain how car insurance premiums are usually calculated in response to repeated questions on various internet fora such as;

I added my mum as a driver and the premium went down.Why is someone quoting £300 and someone else £1200? Why is my renewal premium higher than my first year's premium?
Buy A Ghost
Picture of a cat with yellow star Buy A Ghost is a light hearted source of one page ghost stories aimed at pubs and restaurants as talking points.

Print them off and put them on the tables or walls and you have an instant talking point.

Aimed at regulars who have run out of things to say or new groups or couples as an ice-breaker.
Weight Loss Calculator
Screen shot of weight loss calculator Weight Loss Calculator Tries to simulate the body and report the effects of food and exercise in periods as short as 15 minutes.

This level of detail highlights the effects of a run, bike ride or chocolate bar.

For those who are new to exercise, weight fluctuation due to glycogen usage is made much clearer.
Please note that these links do not use any tracking cookies or similar technology.