Always fun to read your posts! It made me think deeply 😉, not only about what the exams are testing, but also about the alternatives.
As you say, exams are a proxy for something, be it general intelligence, ability to memorise and retrieve information fast, or some other types of skills/abilities. But proxies are just that - proxies. Someone that aces all the exams and does well in a job interview may not necessarily turn out to be a star employee.
One way to deal with that could be to create exam/interview questions based on situations, I.e. describing the situation and asking what they would do or how they would solve that problem in that particular situation. That opens possibilities for a range of different answers that may engage both fast and slow thinking. Although the evaluation of those answers would be more labour intensive and subjective. Also, what we say we would do and what we actually do is not always the same either. I may say that I will leave work on time, go to the gym, and cook myself a healthy dinner, but actually I stay late, pick up McDonalds on the way home and watch Netflix the whole evening.
If we went a step further, we could create simulated (or real) environments and observe how people behave in those environments. For example, I took a driving test last week, which involved me having to perform various parking manoeuvres and drive around the city while an eagle-eyed examiner sitting next to me recorded all the things that I didn’t do properly. However, this type of evaluation is also not perfect as there are many outside variables that can influence the outcome of the test. Things like traffic and weather conditions, the behaviour of other drivers and pedestrians on the road, the examiner’s mood, and many others can play a major role. You might say that you should be able to handle all those difficulties to earn a driving license, which I agree with. But at the same time there will be a number of not so good drivers that may get favourable conditions and pass the test.
So I guess the best thing to do is to always be mindful of what it is that we want to evaluate and choose the most appropriate method for that. At the same time, every measure will only be a proxy, so we should not take the results as gospel. Context matters.
Thank you for the kind words & the thoughtful reply.
I really like your framing of tests as "proxies." I wish I'd used that in the post! While all proxies are imperfect, some will be better than others. But, like you say, better proxies will typically be more expensive.
For example, the final stage of recruitment at Vinted is often a tryout, during which the candidate spends a full day in the office working on a real-life task. That will test deep thinking a lot more than a 1-hour interview or even a 4-hour homework task. But it's a lot more expensive. Trial or probation periods are even more accurate predictors of long-term performance but more expensive still for everyone.
Indeed! It’s much less costly to make 100 students sit an exam than to supervise each one of them individually on a full day work trial. Can’t wait for your next post!
Always fun to read your posts! It made me think deeply 😉, not only about what the exams are testing, but also about the alternatives.
As you say, exams are a proxy for something, be it general intelligence, ability to memorise and retrieve information fast, or some other types of skills/abilities. But proxies are just that - proxies. Someone that aces all the exams and does well in a job interview may not necessarily turn out to be a star employee.
One way to deal with that could be to create exam/interview questions based on situations, I.e. describing the situation and asking what they would do or how they would solve that problem in that particular situation. That opens possibilities for a range of different answers that may engage both fast and slow thinking. Although the evaluation of those answers would be more labour intensive and subjective. Also, what we say we would do and what we actually do is not always the same either. I may say that I will leave work on time, go to the gym, and cook myself a healthy dinner, but actually I stay late, pick up McDonalds on the way home and watch Netflix the whole evening.
If we went a step further, we could create simulated (or real) environments and observe how people behave in those environments. For example, I took a driving test last week, which involved me having to perform various parking manoeuvres and drive around the city while an eagle-eyed examiner sitting next to me recorded all the things that I didn’t do properly. However, this type of evaluation is also not perfect as there are many outside variables that can influence the outcome of the test. Things like traffic and weather conditions, the behaviour of other drivers and pedestrians on the road, the examiner’s mood, and many others can play a major role. You might say that you should be able to handle all those difficulties to earn a driving license, which I agree with. But at the same time there will be a number of not so good drivers that may get favourable conditions and pass the test.
So I guess the best thing to do is to always be mindful of what it is that we want to evaluate and choose the most appropriate method for that. At the same time, every measure will only be a proxy, so we should not take the results as gospel. Context matters.
Thank you for the kind words & the thoughtful reply.
I really like your framing of tests as "proxies." I wish I'd used that in the post! While all proxies are imperfect, some will be better than others. But, like you say, better proxies will typically be more expensive.
For example, the final stage of recruitment at Vinted is often a tryout, during which the candidate spends a full day in the office working on a real-life task. That will test deep thinking a lot more than a 1-hour interview or even a 4-hour homework task. But it's a lot more expensive. Trial or probation periods are even more accurate predictors of long-term performance but more expensive still for everyone.
Indeed! It’s much less costly to make 100 students sit an exam than to supervise each one of them individually on a full day work trial. Can’t wait for your next post!