Category Archives: science

The best way to get expert judgement is not to ask the expert?

In our personal and public lives, we are often faced with the possibility of events that are unlikely to happen but will have catastrophic consequences if they do; we have heart attacks, collapse of banks, earthquakes, Space Shuttle disasters, and invasive species precipitating ecological meltdown, just to name a few. We cannot ignore these events, and it is important to make the best estimates of their probabilities of occurrence in order to formulate judicious and ethical risk-management policies. This is a formidable challenge because the rarity of these events makes it impossible collect enough data within a reasonable time horizon to arrive at statistical estimates, especially if the potential consequences are grave enough to necessitate urgent preventive measures. Furthermore, empirical investigation may be altogether impossible unless we make happen the very catastrophe we are trying to avert.

When we are thus stuck between a rock and a hard place, expert judgement may have to serve as a substitute for quantitative data; this was the focus of two seminars given by Mark Burgman, Professor of Environmental Science at the University of Melbourne and Director of the Australian Centre of Excellence for Risk Analysis, at the Cambridge Conservation Seminar series and Cambridge Public Policy seminar series.

One of the first messages to hit home from Mark’s empirical research with human subjects is that there is no correlation whatsoever between how expert a given person is perceived to be by his or her peers in a given discipline and how good his or her actual judgement is of a given problem in that discipline. People assess the expertise of their peers based mainly on age, appearance and number of publications, but there is no correlation of one’s judgement with any of those criteria. Indeed, there is no known way to pre-segregate people who would deliver good and bad judgements of a given problem.

Mark’s studies also show that groups containing a mixture of experts and non-experts collectively make judgements with lower bias and higher precision than individuals, as long as the group is not too large. Equal weighting should be given to everyone’s contribution; although the perceived experts may have more experience and context-awareness, they also have a higher tendency to be overconfident. The empirical results show that their judgements are no less error-prone than those of the novices. Furthermore, it was found that no individual, expert or otherwise, performed consistently across different problems within the same discipline.

Such teamwork can be a powerful tool for practitioners in diverse disciplines, all dealing with problems embedded within complex dynamic systems from ecosystem conservation to business dynamics, where numerous factors interact with one another through a convoluted topology of causal feedback loops. One can only hope to arrive at the best possible solution when all stakeholders in the problem at hand partake in the discussion from beginning to end. It is because this allows the problem to be tackled from diverse and complementary perspectives and provides ample opportunity to reconcile conflicting objectives, in order to arrive at the best possible understanding of the problem and a consensus that everyone truly believes in. This, you would probably agree, is preferable to a situation where disputes cannot be resolved and have to be put to the vote. There are numerous voting systems but, as Mark highlights, none of them is completely satisfactory and there is no known perfect solution. True democracy is an illusion, although it doesn’t work too badly in practice.

Mark walked us through the basic nuts and bolts of group-based expert judgement using a very simple but highly illustrative example of estimating, or judging, the number of beans in a jar. Unlike a real-world rare extreme event, this setup was amenable to experimental manipulation. A judgement is first obtained from each individual in the group. He advocates four subsidiary questions for the individual:

1. What is your highest estimate of the number of beans?
2. What is your lowest estimate?
3. How confident are you that the actual number lies within those limits?
4. What is your best estimate of the number of beans?

This four-step process has been found to elicit the most accurate estimates from individuals—Mark welcomes ideas on how to do even better.

Next, one can improve the estimates by obtaining a consensus from multiple individuals. This is done by incorporating the innovative four-step process above into what is known as the Delphi Method. After each member of the group has independently given his or her answers to the four questions, the answers are collated and shown to the whole group. Each member is then asked to revise his or her personal answers given the collated figures, and the whole sequence may then be repeated again. This has been empirically shown to yield even more accurate estimates. Furthermore, the variation in inputs from multiple individuals with different cultural and educational backgrounds gives an idea of how robust and reliable the estimate might be. This is reminiscent, albeit tenuously, of a ‘sensitivity analysis’ of the robustness of mathematical model predictions in the face of parameter uncertainty. Despite the real-world importance of determining the degree of sensitivity of predictions to parameter uncertainty, sensitivity analysis is rarely done or required even by peer-reviewed ecology journals.

Conservation could benefit from group-based expert judgement, since ecological data are often scarce and incomplete and the nonlinear behaviour of complex ecosystems makes low-probability high-risk events inevitable. Mark’s take-home message is that expert judgement should be accorded the same reverence as scientific data, but it should be harnessed in a systematic way that minimizes bias and maximizes precision rather than being a haphazard question-and-answer session. The complementarity of quantitative data and expert judgement in turn feeds nicely, I think, into the system dynamics framework for simulating and solving complex nonlinear problems in many disciplines.

I am grateful to Mark Burgman for permission to communicate his research and hope it will be of value to my readers.

Related topics
Info-gap decision theory
How banking benefits invasion ecology
Horizon scanning in nature conservation

Advertisements

Theoretical Photography of Black Holes

I recently met Rob Hocking, a Canadian mathematician who is passionate about visualizing theoretical physics in photography. He showed me some fascinating images of how everyday objects would appear if a black hole were in the vicinity—a predicament one would think twice about wanting to experience in real life. Those images had been computed using conventional photographs as raw data. ‘Light rays from the object are bent by the black hole en route to the observer, resulting in a distorted appearance which can be artistically pleasing’, says Rob. The bending effect is called gravitational lensing, a term photographers may find appealing.

Illustration: Rob Hocking

Gravitational lensing. Illustration: Rob Hocking

In the figure above, light rays from different directions are deflected to varying extents by a black hole before reaching the observer’s eye, including rays (example in blue) from behind, making it possible to see objects not normally within the human field of vision. In order to accommodate this, a photograph that sees in all directions must be used to generate the hypothetical image of ‘environmental’ black holes. This photograph, known as a spherical panorama, is a composite of photographs covering all points on a sphere with the camera in its centre. The number of photographs needed to construct the panorama depends on the focal length of the camera lens and the desired resolution.

Spherical projection. Photo: Rob Hocking

Spherical projection. Photo: Rob Hocking

This is a spherical panorama of Rob and his yoga instructor Kevin Elander, which stretches the floor and ceiling in the same way that Antarctica and the Arctic are projected on many world maps. The panoramic tripod head that Rob uses can be seen in the mirror at far left. It has been programmed to rotate the camera through 40 different positions while keeping the optical centre fixed. The only direction it cannot look is straight down; that view is covered by an additional, handheld photograph.

There is an ingenious device, patent pending at press time, for capturing spherical panoramas. This consists of a ball studded with cameras pointing in all directions. The ball is thrown into the air. When it reaches maximum altitude, the internal motion sensor sets off all the cameras at once. I assume it survives the impact of return to earth.

A work of art: one of the equations used to add black holes to a spherical panorama. Source: Rob Hocking

Abstract art: one of the equations used to add black holes to a spherical panorama. LaTeX source: Rob Hocking

Given the panoramic photograph, the next step is to calculate the trajectories of the deflected photons using the geodesic equations from Einstein’s general theory of relativity. This information is then used to remap the photographic pixels. The following diagram shows four calculated examples of the infinitely many possible trajectories of photons around a black hole on their way from a given point to the observer. Photons may orbit the black hole any number of times before shooting off at a tangent.

Illustration: Rob Hocking

Some calculated photon trajectories. Illustration: Rob Hocking

The remapping of the yoga photograph is shown below, with two black holes situated above the yoga and math gurus. Because of the infinite number of photon paths, there are in fact infinitely many secondary images of the gurus around each black hole if we assume infinite photographic resolution; only one of the images is visible for each black hole, the rest being too small to be resolved on the digital display. This example is intriguing as it incorporates two interacting black holes.

Meditating on black holes. Photo: Rob Hocking

Meditating on black holes. Photo: Rob Hocking

Here is another of Rob’s spherical panoramas featuring the Great Wall of China, along with one of his signature black hole photos derived from it. The odd patch in the sky is not a renegade black hole—it is the consequence of camera misalignments during the production of the composite photographs. Rob had to handhold the camera, not having a panoramic tripod head at his disposal at the time.

The Great Wall. Photo: Rob Hocking

The Great Wall of China. Photo: Rob Hocking

Photo: Rob Hocking

Hole in the wall. Photo: Rob Hocking

Check out more of Rob’s thought-provoking photographs here. I’m grateful to Rob for telling me about his work and for his valuable assistance with the editing of this post.

Rob Hocking is a PhD student at the Faculty of Mathematics, University of Cambridge. It’s great to meet someone who is able to unify the photographic and relativistic meanings of the slogan where Photons meet Black Holes. See more of Rob’s fascinating stuff at his blog.


%d bloggers like this: