Key points
- Most attempts to measure risk tolerance fail in at least one crucial way, be it confusing the measurement, confusing the audience, or thinking guesswork is a good enough replacement for rigorous psychometric science.
- The difference between a faulty test and a good one is often in the testing – in scientific terms, how reliable and valid are the items selected for inclusion, and how well does the set work as a whole?
- Beware of measurements that, mix in traits other than risk tolerance, measure financial understanding, or provide inconsistent, unstable outputs.
How Not to Measure Risk Tolerance
The need to assess investors' risk tolerance has been law for a long time. Sadly, not long enough for measuring it at all to evolve into measuring it at all well, save for a select few cases.
This is a problem. Measuring risk tolerance badly is as bad as not measuring it at all. Most attempts fail in at least one crucial aspect. A random selection of instruments can't claim to be an orchestra. Close enough isn't good enough. And if 'everybody else is doing it' isn't good enough for your mother, it's not good enough for the regulators either, however well-intentioned or well-meaning. As H.L. Mencken put it, “there is always a well-known solution to every human problem – neat, plausible, and wrong.”
As with investing in general, assessing risk tolerance well is mostly about avoiding silly mistakes.
Though statements that pass the audition for inclusion in an assessment should be basic, the process by which they are chosen definitely shouldn't be. Too often, assessment-designers get this the wrong way around: bodging complex statements together with little statistical rigour to justify the bodging or the complexity.
Even among those that understand the importance of psychometric tests where investors can't reliably isolate the factors that affect how they think about risk at any one time (separating those that should shape a long-term portfolio from those that shouldn't), too few grasp that not all psychometric tests are created equal. Poor design can distort what you meant to ask into something quite different. You may think you're asking “How much risk are you willing to trade-off for better returns in the long-term?”, but end up eliciting answers to the question: “How do I feel about taking investment risk this morning?”
How do you know you're doing it right? How do you know you're measuring the right thing – and only the right thing?
You've got to test it. Even the experts – partly by virtue of being experts – don't know if a given set of questions does what it set out to do until it's been sent through a scrupulous statistical sieve. Does a statement represent only the single trait it's supposed to, rather than a combination? Is it consistent across market cycles and cultures? Does the set of statements work together to provide stability without suffocation?
Getting it right is a science. Getting it wrong is serious.
Risk tolerance is a psychological trait. And like other psychological traits, say, one's degree of extraversion or introversion, if an environment is not aligned with a psychological preference – like a strongly introverted person in a busy networking event – you've a recipe for discomfort and a person wanting to get out of their environment as quickly as possible. The consequences of this in personal finance can be extremely costly.
The biggest problems are caused by accidentally measuring something other than risk tolerance. Sometimes that something may be worth measuring… separately. Other major mistakes are about questions that are either too hard to understand, or too hard to answer accurately enough to be helpful… or too fun and trivial to be accurate.
Here's how not to do it:
Confusing the measurement
Don't confound risk tolerance with other attitudes, behaviours, or personality dimensions – Short-term emotional responses to markets should be understood in order to be controlled, not to be baked into a recommended solution. For example, don't ask about an investor's feelings about investment decisions after they've been made (when emotions are likely to be high, and unstable), or the confidence with which they were made (confidence can be useful, e.g. to calibrate decision-making processes, but it doesn't affect core psychological preferences). Know also that risk is domain-specific: even other financial domains, like income, insurance, or gambling, give limited insight into one's willingness to trade off investment risk and reward.
Don't confound risk tolerance with investment objectives – What an investor is aiming to achieve with their investments is independent of their willingness to take on the risk of worse outcomes in order to achieve those aims. The way one trades off goals against each other can be an expression of risk tolerance; the extent of the dreams should not be an input into the risk-tolerance score itself. Goals are part of the context, not the calculation. A suitable portfolio can change when circumstances change, but that's because of a change in risk capacity, not risk tolerance.
Don't confuse hypothetical choices with optimal actions – Responses to choices between hypothetical gambles, lotteries, or portfolios result neither in stable outcomes nor absolute levels that can be meaningfully used; a forced choice from a limited menu may not reflect investors' reality.
Don't confuse past behaviours with optimal actions – Preferences 'revealed' through actions are not necessarily fundamental 'preferences' in a psychological sense. Importantly, what's not revealed is whether these actions were sensible. To use past actions as a foundation for risk tolerance means entrenching possibly bad behaviours, not encouraging better ones.
Don't trivialise risk tolerance into 'games' – Gamification is great for engagement; but gimmicky games trivialise risk tolerance, they do not test it. Form should follow function, not replace it; if you're not measuring what you're supposed to be measuring, the playfulness of your polish doesn't matter.
Confusing the audience
Don't require numerical calculations or probabilistic reasoning – This is a psychology assessment, not a maths test. Any percentage sign is usually a bad sign. In truth, numbers on the whole should be avoided – numbers cause stress, and stress can cause psychometric assessments to be too much psycho and not enough metric.
Don't use potentially ambiguous or context-dependent statements – What shows statistical reliability in one culture or time period may not do so in others. Questions with a social-status subtext can lead us to lie to ourselves. Some contexts can change the way a statement is read, e.g. asking about the risks of banks before versus after a financial crisis. And some questions, e.g. those that contain multiple clauses, are as much a test of an ability to not get confused as they are risk tolerance.
Don't require knowledge of investing – We want to test risk tolerance, not knowledge. Questionnaires – especially ones not thoroughly tested – can be cursed with their designers' knowledge, a failure to understand what it's like to not understand even 'simple' concepts like the difference between shares and bonds. Knowledge and experience are important, and low K&E should restrict investment options, but not systematically and forever, which is what happens when you bake it into a risk-tolerance score. K&E is a temporary constraint, more akin to short-term liquidity issues than to a long-term psychological preference.
Don't reference or require knowledge of current market conditions – We need to assess long-term willingness to trade-off risk against return, not current attitudes to risk and markets. We want to measure an internal tolerance for risks, not the current state of the external risks themselves.
Favouring guesswork over science
Don't require respondents to assess their own future feelings – For example, responses to “How would you feel after a 10% drop in portfolio value?” are dreadful predictors of one's future emotional state. An investor's psychology is shaped by a lifetime of experience; it should be measured on that experience, not on speculation about what might happen in future, especially in reaction to a new event.
Don't elicit returns expectations – Asking an investor what return they expect to experience creates an unreasonable expectation where none existed before. Return expectations are better 'owned' by the adviser, and brought up after risk tolerance has been assessed.
Use only statements that discriminate effectively between individuals – Risk tolerance is a relative measure – we need people to be well-distributed across a range. If the majority of responses fall within a narrow band, the question is about as helpful as an abundance of straight-A scorecards is to a university admissions tutor.
The ideal solution dodges falling into these traps, and dodges unwittingly failing the investor in the process.
Originally published in New Model Adviser on 25/9/2019.
