I have been a follower of Philip Tetlock, Dan Kahneman, Richard Nisbett, Thomas Gilovich etc for over 20 years and devoured Tetlock’s recent Superforecasting book on its release. Tetlock has led a team of forecasters using a combination of crowdsourcing, cognitive bias training, performance feedback and other techniques to outperform a suite of other teams at forecasting geopolitical events in a controlled multi-year forecasting tournament. The tournament was run by IARPA (Intelligence Advanced Research Projects Activity including FBI, CIA, NSA etc), which is the Intelligence community’s equivalent to DARPA, as part of their Analysis and Anticipatory Intelligence streams.
IARPA invited a range of teams from multiple elite US universities to forecast hundred’s of important geopolitical questions to gauge their performance against professional intelligence analysts’ forecasts (the professionals had access to classified information but not the teams). In the first year Tetlock’s team beat all their competitors handsomely but also outforecast IARPA’s forecast accuracy goals, not only for the first year but also IARPA’s initial goals for the second and third year of the tournament. The next year Tetlock’s team vastly improved again, vastly outperforming all the new goals (and competitors) such that IARPA decided to cancel the tournament and bring Tetlock’s team into to do all future IARPA forecasting. In many cases Tetlock’s team of Superforecasters, with access only to publically available information, significantly outperforms the intelligence analysts with access to classified information.
But what makes Tetlock’s team of Superforecasters so good at predicting the future? Tetlock, being an academic, took the opportunity to identify what habits and characteristics his team of forecasting geniuses had that other’s don’t. Here’s the attributes he found:
- Cautious – they predicted outcomes with less personal certainty than others, they always kept “on the other hand” in mind
- Humble – they didn’t claim that they knew everything or that they fully understood all aspects of a problem and would readily change their mind when new evidence came in
- Nondeterministic – just because something has happened they don’t ascribe a “Monday’s quarterback” explanation for its occurrence. They keep in mind that events may have turned out differently for potentially unknown reasons.
- Actively open-minded – they’re always testing their own beliefs and see them as hypotheses to be tested, not dogma to be protected
- Naturally Curious with a need-for-cognition – love solving problems and adding new ideas and facts to their own knowledge
- Reflective, introspective and self-critical – they are constantly re-evaluating their own performance and trying to uncover and correct the errors they themselves made
- Numerate – tend to do back-of-the-envelope calculations, comfortable with numbers
- Pragmatic (vs Big Idea) – not wedded to any one ideology or worldview, prefering reality and facts over opinions and untested theories
- Analytical (capable of seeing multiple perspectives) – break problems up into logical parts and consider many different views of a problem
- Dragonfly-eyed (value multiple perspectives) – value the inputs of viewpoints that are new and different to their own and can handle assessing differing theories at the same time.
- Probabilistic – see events as likely or unlikely (as opposed to will or won’t happen) and are comfortable with uncertainty
- Thoughtful updaters – willing to cautiously adjust their previous assessments as new information comes in
- Good intuitive psychologists (aware of and able to compensate for common biases) – they understand the various cognitive biases us humans have when thinking about problems and put in the effort to overtly overcome the shortfalls the biases cause
- Personal Improvement mindset – always trying to learn more and get better at whatever they are doing
- Grit – simply don’t give up until they feel they can’t improve their assessment any further
When I reviewed this list of personal attributes and habits it struck me just how similar these Superforecaster attributes were to the attributes ascribed to truly excellent data scientists. A quick review of articles (e.g. Harvard Business Review, The Data Warehouse Institute, Information Week etc) about the differences between good Data Scientists and great ones turned up this aggregated list (a ✔ indicates an overlap with Superforecasters and ✗ being unique to excellent Data Scientists ) :
- Humble (before the data) ✔
- Open-minded (will change their mind with new evidence) ✔
- Self-critical ✔
- Analytical (breaks problems down, does quick back-of-the-envelope calcs) ✔
- Persistence/grit ✔
- Comfort with uncertainty/able to hold multiple theories at once ✔
- Cognitive/Innate Need-for-understanding ✔
- Pragmatic (as opposed to theoretical) ✔
- Interested in constantly improving ✔
- Creative (able to generate many hypotheses) ✗
- Has innate understanding of probability/statistical concepts (conditional probability, large numbers etc) ✔
- Business Understanding ✗
- Understands Databases and scripting/coding ✗
This also works in the other direction with only a couple of Superforecaster specific attributes that aren’t covered by the Great Data Scientist attributes list. Tetlock has studied his Superforecasters with academic rigour whereas the Data Scientist list is likely to be more opinion and untested hypotheses (so I believe the Superforecasters won’t like this article :-), however one cannot but be impressed by the significant overlap of the two lists.
Is it possible that the big successes provided by Data Science to date have actually not been due primarily to the massive data crunching capability and the ubiquity of IoT and social media data collection, but is actually primarily the result of applying the personal habits and attributes of Superforecaster-like Data Scientists to business problems? If so, this says we could get a lot of business value out of applying Superforecaster-like people and approaches to many business problems, especially those that may not have a lot of data available.
Do you know of any applications where a Superforecaster approach might help your organisation?
List of articles:
https://hbr.org/2013/01/the-great-data-scientist-in-fo
https://infocus.emc.com/william_schmarzo/traits-that-differentiate-successful-data-scientists/
http://www.cio.com/article/2377108/big-data/4-qualities-to-look-for-in-a-data-scientist.html
http://www.boozallen.com/insights/2015/12/data-science-field-guide-second-edition
https://upside.tdwi.org/articles/2016/06/13/five-characteristics-good-data-scientist.aspx
Note that I had to edit the aggregated list due to the fact that some articles were comparing Data Scientists to general IT people, or general executives, as opposed to less effective Data Scientists. This led to some articles concentrating on the very technical capabilities required by every Data Scientist as opposed to the personal characteristics and habits which resulted in excellence.