An Interview with Dr Anca Hanea PhD
Interviewed by Paul Vorbach F.ISRM
Paul Vorbach F.ISRM
Welcome to this conversation with Dr Anca Hanea PhD, Associate Professor at the University of Melbourne. Associate Professor Anca focuses on work through the University of Melbourne’s Centre of Excellence for Biosecurity Risk Analysis. She specialises in probabilistic modelling, Structured Expert Judgment (SEJ), and decision making under uncertainty, with applications across biosecurity, infrastructure and public policy. Anca has been designing and running SEJ since 2009, advancing both theory and practice by combining applied probability, optimisation, psychology and the social sciences.
Her career spans roles at the University of Melbourne and TU Delft, with teaching experience across three countries. She holds a PhD in applied probability and a Master’s degree in risk and environmental modelling cum laude from Delft University of Technology, plus a Bachelor’s degree in mathematics from the University of Bucharest.
I have observed what seems to be a ravine between the qualitative assessment methodologies and the mathematical, data driven assessment methodologies in risk management. When I came across Anca and her area of specialisation, I was keen to explore this topic, being suspicious that subjective judgment was not sufficiently rigorous, but also understanding that the preconditions for quantitative analysis are not often present either. Her field of expertise presents an opportunity to address that gap.
Most risk professionals are familiar with expert opinion, but structured expert judgment, or SEJ, is a different beast. How do you explain the fundamental difference between an informal expert consultation and the scientific rigour of SEJ?
Dr Anca Hanea PhD
I have a quantitative background and foundations are important to me. We are being generous when we call other methods informal or unstructured. Some of them are unscientific. What we are trying to do is bring scientific rigour into the discussions we have with experts, because they are present in every step, from the definition of the problem to the modelling to the quantification and parameterisation.
What we can do is make that collection of expert data as rigorous as any other data collection in the field. That means following scientific principles: making it reproducible, repeatable, and documented. The process needs to be planned and accountable, like any other scientific process.
Empirical control, whenever needed, is akin to calibrating measuring instruments when you collect data from the field. But it also has the extra layer of working with humans, so we need to ensure there is fairness in the process and that we mitigate, or at least are aware of, the potential cognitive and motivational biases.
There is a lot to consider when you design an application. There is no way to do that without a structured protocol and clear steps to account for those considerations.
Paul Vorbach F.ISRM
You use the word elicitation, which is not a term commonly heard in risk workshops. It seems you are more precise in the way you look to elicit expert judgment. Would that be the right characterisation?
Dr Anca Hanea PhD
We are striving to be as precise as possible. But the qualitative part of eliciting knowledge, as well as numbers, rationales and reasoning, is important too. We are trying to elicit as much information as possible, and if we can put it in quantitative terms, then that is what we need to do, because it feeds into models that need parameter estimates.
Paul Vorbach F.ISRM
In biosecurity or emerging AI threats, we often lack historical data. Why is SEJ the gold standard for quantifying risks where traditional statistical models fall short?
Dr Anca Hanea PhD
I would not say they fail, but they are insufficient because they depend on having data. Considering structured expert judgment as a complementary tool is the key. We still use statistical methods; it is just that the context does not justify the assumption that what we are modelling can be based on historical data.
We often have to predict something that has never happened before, so historical data would not fit the context or the assumptions in the modelling. But we still build probabilistic models. We still start with the context, the conceptual model, and the probabilistic and statistical model. In this process we often identify data gaps, and those are the ones we fill with expert judgment, because the knowledge is there.
I met someone recently who was not using the term “expert” but “knowledge holders” instead, and I liked that. There is knowledge in most of the problems we deal with. The data is not there either because it concerns the future, so we have not had the chance to collect it, or we did not have the resources to do so.
We are trying to harness that knowledge and put it in numerical format to complement the statistical methods we have based on available data.
Paul Vorbach F.ISRM
Is the role and necessity for SEJ going to increase as future environments become more complex?
Dr Anca Hanea PhD
I think so. We are seeing that in the biosecurity settings I work in now, trying to prepare for the future under different climate scenarios and changes in trading and other factors. The uncertainty comes from many more moving parts than usual.
The need for SEJ will increase. I do not know if acceptance of it as a scientific tool will increase at the same rate, but we will have to acknowledge that there are many more connected and independent uncertainties that we do not have the data to deal with.
Paul Vorbach F.ISRM
You were instrumental in developing the IDEA protocol, which stands for Investigate, Discuss, Estimate, and Aggregate. For a risk manager on the ground, what is the secret sauce in this four step process that helps neutralise groupthink and overconfidence effects?
Dr Anca Hanea PhD
There are a few ingredients of the secret sauce sprinkled throughout all stages of this protocol, from pre-elicitation through to the aggregate stage.
In the pre-elicitation part, we strive to get diverse experts. We do not know exactly what that means, so we use proxies for diversity in terms of world views, gender, age, and experience. We are trying to get as many views in the room as possible.
Having an independent facilitator helps, not with groupthink as such, but with the group biases that occur with dominating personalities. I am often that independent facilitator, and it is beneficial that I do not know who is perceived as the best expert in the room. I can be fair with everyone and draw out all the voices, and understand whether the quiet ones are quiet because they feel intimidated or for other reasons.
The first step, the Investigate phase, is there to mitigate against cognitive biases and to support the discussion. By asking individuals to put effort into that first independent estimate, they do their exercise on their own, drawing on their own resources, not talking to the other experts. That makes them invested in the numbers and their reasoning.
Once the discussion starts, they need to hear solid arguments to change their mind. They do their first round estimate, then we bring them together to discuss the de-identified numbers. We present feedback plots showing all the numbers from the group without associating numbers with people.
They do not need to identify themselves on those plots. They talk about the reasons behind numbers being low, high, or not intersecting. Because they have thought hard about the numbers, they need a good reason to change them. After discussing, sharing reasoning, and exploring counterfactual scenarios, they go back and, without anyone knowing, change their mind or not.
The second estimate stage is again independent and anonymous. Someone in the room may have nodded vigorously because their boss said something they do not agree with, and then independently they do not need to change anything. No one would know.
Even if there is a dominating personality that could distort the estimate, by having a second round that no one knows the details of, we mitigate against that. We have seen exercises where people choose not to change their mind, even after significant discussion. They were not convinced.
The last ingredient is that we aggregate mathematically. The discussion does not need to go on forever. We do not strive for consensus. We do not ask anyone to agree on a specific set of numbers. They pick up information from other knowledge pools, from experts with different backgrounds, different reading, and different experiences. At the end, we aggregate mathematically, pulling all the numbers together. What you get is one set of estimates that does not fit what any one person thinks, but is a group estimate that incorporates all the uncertainties and all that was discussed.
Paul Vorbach F.ISRM
There is a lot of behavioural science in structuring these processes. How important is trust between the group members and the facilitator? You mentioned independence and not knowing who the senior figures are. Why is that important?
Dr Anca Hanea PhD
In my role as facilitator, it is important because I do not get intimidated by knowing who the big expert is. I do not need to know that. I can treat everyone fairly, and I can tell them to sit down and listen if they talk too much.
I have no skin in the game. I am not from that community. That allows the other people to speak up. My background is in mathematics. The only social sciences training I have is through this work and through colleagues who advise on the steps of these protocols.
The group dynamic matters. When the group is cohesive and respectful, the discussions are wonderful. I learn with every elicitation. It makes a full day of discussing dozens of questions a breeze, and at the end, everyone is happy they participated.
I have been in groups with tensions. Being independent helps, not for the tension to disappear, but for it not to manifest. At the end, people are often happy they heard the others’ judgment and reasoning. They do not necessarily agree, but they see where the others are coming from. That mutual understanding makes the community more cohesive.
Paul Vorbach F.ISRM
How important is it to be explicit upfront about your lack of awareness of who has what seniority or status within the group? Is that part of your positioning?
Dr Anca Hanea PhD
It is. It also allows me to ask naive questions. I can throw in numbers that make no sense, because it is easy for people to say, “That is ridiculous, that can never happen,” rather than justify their own numbers. It serves as a conversation starter. Once they get going, it is often a rich and respectful discussion.
Paul Vorbach F.ISRM
A unique aspect of your work involves scoring experts based on seed variables or calibration questions. Can you explain why we should weight expert opinions differently, and how we can measure if someone is a good probability assessor?
Dr Anca Hanea PhD
This is a tricky one. Only a subset of the community advocates for seed variables. I am part of that subset. These variables are important and bring value to an elicitation. I do not always have the chance to include them, but if I can, I do. They are slightly misunderstood.
Calibration questions are about calibrating the measurement instrument, like a microscope when you are measuring something. But it is easy to see them as grading the experts. We have had people say they felt they were being graded like a student, and that is not what we are doing. Because of that tension, part of the community does not use them.
There are two main points where seed questions make the elicitation better. In the IDEA protocol, individual experts give their estimates, then have a discussion, and then have the chance to change their mind. Between those two rounds, we claim that the discussion makes estimates more accurate, more calibrated, and more informative.
One step is to verify that the protocol steps are doing what we think: improving the estimates. Another contribution is at the end, when we aggregate and use a group assessment, we can check if that group assessment improves when we use differential weights based on performance rather than equal weights. We can justify that the aggregation we choose is the best we can have.
What we measure are questions for which we either already know the answers (the experts do not), or questions where the answers will become available after the elicitation. We acknowledge uncertainty, so experts give not just a point estimate, but also a lower and upper bound, forming an interval for each question.
In a good probability assessor, we look for someone who captures the true value between the bounds on the long run, and does so with informative answers. Non-informative answers are where the bounds are so spread they contain no useful information. We look for calibrated answers that are also informative. We measure both, combine them, and assign weights proportional to the score.
We can do the same before and after the discussion. Every time we check, after the discussion experts get more calibrated and more informative. That gives us confidence the steps are doing what they are meant to.
More often than not, when experts change their mind, they move in the direction of the truth. The uncertainty decreases, intervals become narrower, because ambiguity got resolved and conditioning factors were clarified. At the end, on the same questions we know the answers for, we can identify the aggregation that performs best. As we would think of it as a virtual expert in the textbook, that is the one that represents the group.
There is value in having calibration questions. They are hard to find, and they are based on the assumption that they trigger the same thought processes and mental models as the questions of interest. If they are not representative, the criticism is that you do not know if they predict future performance. There is no theorem that proves this, at this stage. There is work at TU Delft where studies are collected into a dataset with validation on those results. Every time someone examines it, the calibration questions appear to add value.
Paul Vorbach F.ISRM
Is there transparency around the weighting decisions back to the experts?
Dr Anca Hanea PhD
When we have calibration questions, the experts know they will be there. Sometimes they may be identified, sometimes not, so experts do not necessarily know which ones. If the questions are based on past events, they will identify them. They always know, and we tell them that individual scores are known only to whoever calculates them.
Whoever wants to know their scores is welcome to ask. In my experience, about 20 per cent of experts ask, and the reactions are positive. If they score well, they are happy. If not, they are happy that the group aggregation scored well, which is the right attitude. Most have it. They may start out nervous, but I have never heard of an expert who was unhappy at the end.
Paul Vorbach F.ISRM
I can imagine in some organisational contexts, competitiveness would loom large.
Dr Anca Hanea PhD
Many start out nervous. They do not all feel comfortable with the approach, but by the end, they recommend it to others. Going through the process makes them understand we are not interested in individual scores. It is not a test of knowledge or expertise. It is about getting the best out of the group.
Paul Vorbach F.ISRM
You noted that some in the field are not advocates of the seed variable approach, perhaps because they are not convinced the questions used to determine weightings are sufficiently representative. Preparing the right questions must be an important part of the process.
Dr Anca Hanea PhD
It is. It requires more work than the questions you do not have the answers for, because you want questions that are meaningful for the questions of interest, and you want them hard to find so no one games the system.
We usually have a few experts who do not participate in the elicitation but help us find seed questions, or provide unpublished experiments. We need insiders, because otherwise we cannot find the good ones.
In some applications there is a catalogue of seed questions because they have been used repeatedly. In volcanology, Aspinall has been running expert elicitations for years, the longest running elicitation in that field. Aspinall and his team often publish the questions, so volcanologists have access to a catalogue that helps develop new ones. If that became standard practice, it would be valuable. It is starting to happen. Calibration questions are increasingly published alongside the target questions.
Paul Vorbach F.ISRM
Your work with CEBRA, the Centre of Excellence for Biosecurity Risk Analysis, focuses on biosecurity, an area with high stakes and high uncertainty. What is one insight or risk that SEJ uncovered in biosecurity that data driven methods might have missed?
Dr Anca Hanea PhD
I do not think SEJ on its own can do that much, but the biosecurity centre I work for, the people I work with, and their willingness to use it as one tool in the toolbox, that is the key. The value in the biosecurity projects we deal with is understanding the value of future actions and management options under finite resources.
Resource allocation for management actions to mitigate future threats is hard to evaluate. A good decision avoids the risk, and then it is hard to measure whether it was your decision or other factors. For resource allocation, if we are interested in a pest or disease we want to keep outside Australia from neighbouring countries, or keep outside Tasmania if it is on the mainland, we simulate scenarios where the pathways for those pests and pathogens are described and modelled.
Then we look at what we can mitigate. Can we invest pre-border, helping neighbours mitigate, which would reduce the likelihood of the pest reaching Australia? Or invest in more inspections at the border? Or in surveillance protocols for early detection?
We compare those options, and there are different moving parts in the modelling. Some can be informed by data, but for some there is no data. That is where expert judgment matters. The willingness of government officials, border officers, and people who know about these problems to answer questions through structured protocols is the key to having a complete picture of what could happen and what would be affected by an incursion.
Paul Vorbach F.ISRM
Even the smartest experts are prone to cognitive biases like anchoring. How does the discussion phase of your protocol target these mental shortcuts without letting the loudest voice take over?
Dr Anca Hanea PhD
Anchoring is one of the strongest individual cognitive biases we know. The first thing we do is make the experts aware of it. Some think about it, but some do not realise how often it happens or how strong it can be. Because we ask them for hard things like thinking under uncertainty, they rely on rules of thumb and mental shortcuts.
We have a meeting with everyone explaining why we do things the way we do: why we need diversity, why we ask the questions in a particular way. For example, we ask for a best estimate then a lower and upper bound. We found that because they anchor on the best estimate, the interval around it tends to be too narrow. They appear overconfident.
One way to mitigate this is to ask them to first consider the counterfactuals. This gives them two anchors, but at least they do not settle on a best estimate with a small variation around it. By thinking about the worst and best that can happen, and putting everything in balance, we mitigate their own anchoring.
Then we use anchoring to our advantage. By asking them to do the first estimates as individual work, they anchor on their numbers so that during the discussion, they need to hear something convincing to move those answers.
Availability bias is another that comes up often. When answering questions, we all use what first comes to mind. By bringing experts together, we bring those availability biases, like pieces of a puzzle, onto the same table, and they complement and enlarge the knowledge base.
For group biases, an independent facilitator helps. The second round of estimates, where no one needs to know if they changed their mind, also helps. Not striving for consensus allows them to finish when no one has anything else to add, and move on.
There are studies that assess whether these mitigation strategies work. We cannot prove they do, but that is what we are striving for.
Paul Vorbach F.ISRM
The honesty and directness of the approach, helping experts understand why biases are important to address upfront, must be constructive for some as they confront hidden biases more explicitly.
Dr Anca Hanea PhD
It is often well received. People bring up mental models they have never verbalised or explained to someone else, so it helps their own thinking. But the deliberate biases, the motivational ones, are hard to mitigate.
I have not encountered that myself, because it is usually technical experts being asked about parameters. We try to separate their values and advocacy. The way to mitigate motivational biases is to ask for numbers and parameters that are inputs into models, so experts cannot calculate the output or manipulate those numbers.
Paul Vorbach F.ISRM
From a board level, risk professionals often struggle to convince executives or directors of subjective probabilities. How does the transparency and traceability of SEJ help a risk officer defend their assessments in a corporate or governmental setting?
Dr Anca Hanea PhD
I am still on the naive side of science, thinking that good scientific arguments should be convincing enough for people to take action. From that perspective, a transparent methodology and a traceable process should be sufficient.
But a more convincing argument is that we often need to answer crucial questions. A risk manager needs to take a decision, and there are few options without expert judgment. You can ignore the question because you lack data for a precise answer.
Or you can use this methodology for a more approximate answer, acknowledging the uncertainties. That may not appeal to many risk managers, but you can frame it in terms of best and worst case scenarios, and reach an informed decision.
Paul Vorbach F.ISRM
We often see organisations rely on a single subject matter expert for critical decisions. They might be within the organisation, or consultants held in high regard. What are the hidden risks of relying on individual expertise versus structured group aggregation?
Dr Anca Hanea PhD
There are many. Overconfidence would be the first. Not having the full picture. Not having the chance to discuss rationales with people holding a different world view. Not thinking of counterfactuals or having anyone challenge your views. It leads to confident, and often wrong, answers. It is a real danger.
Paul Vorbach F.ISRM
I can also see the potential conflict of incentives, particularly with a paid subject matter expert whose credibility depends on presenting themselves as the expert.
Dr Anca Hanea PhD
Those motivational biases are impossible to control for with only one expert.
Paul Vorbach F.ISRM
As we move into an era of large language models and AI driven analytics, do you see SEJ risking obsolescence, or does it become more critical as a way to ground or validate algorithmic outputs?
Dr Anca Hanea PhD
I do not think SEJ is in danger of becoming obsolete. AI algorithms can be capable, but they are trained on existing data. That is the reason we need structured expert judgment: we do not have the historical data to train our models or inform our estimates. Even though AI can process more information, you still would not have the future prediction we need.
A combination of the two would be essential. At the preparation stage, we can use AI to prepare background documents for experts, ensuring everyone starts with the same basis and anchors on what we know to be true. Using AI to ensure we did not miss anything could help.
Having AI summarise discussions, especially for asynchronous elicitations documented on a platform, a tool that summarises the main points and triggers counterfactual thinking would be useful. A combination is the way forward.
Paul Vorbach F.ISRM
For a risk manager reading this who wants to move away from gut feel but does not have a PhD in mathematics or statistics, what is the first step to start integrating structured expert judgment into their risk framework?
Dr Anca Hanea PhD
If they are open to that, they have made the first step. If they are confident that structured expert judgment and the protocols developing since the nineties are a scientific method, that is significant.
There are papers, books, blogs, and other resources that make these tools accessible. The beauty of the main principles is that they are simple and elegant. You can make it as complex as you want with more sophisticated aggregation tools and measures, but you do not need to. The first principles are simple.
We are a friendly bunch. Just reach out. Everyone in this community would be happy to help because we believe this is a good tool that everyone should have access to. If done properly, it advances the modelling, estimation, and predictions. That would be my advice: just reach out.
About Dr Anca Hanea PhD
Dr Anca Hanea is an Associate Professor at the University of Melbourne’s Centre of Excellence for Biosecurity Risk Analysis (CEBRA). She is a probabilistic modeller and applied mathematician with a strong interest in risk analysis and decision making under uncertainty.
Anca specialises in structured expert judgment (SEJ), a rigorous method for obtaining accurate, calibrated and informative quantitative estimates where experimental data are sparse, unreliable, or unavailable. She has been designing and running SEJ elicitations since 2009, advancing both theory and practice by combining applied probability, optimisation, psychology, statistics, and the social sciences. Her work spans biosecurity, infrastructure and public policy.
Her collaborations with theoreticians and practitioners worldwide have resulted in edited books, special issues, peer reviewed publications, research and networking proposals, book chapters, technical reports, and lectures designed for industry, as well as dozens of talks for international audiences.
Teaching is a longstanding passion. Anca chose pedagogy as her track during her studies at the Faculty of Mathematics, with developmental psychology and pedagogy forming part of her curriculum. Throughout her career she has taught across three countries.Anca holds a PhD in Applied Probability and a Master’s degree in Risk and Environmental Modelling (cum laude) from Delft University of Technology, and a Bachelor’s degree in Mathematics from the University of Bucharest. Her career spans roles at the University of Melbourne and TU Delft.
About Paul Vorbach F.ISRM
Paul Vorbach is the Founding Managing Director of AcademyGlobal and Vice President of The Institute of Strategic Risk Management (ISRM) Australia and New Zealand. He is a Fellow of ISRM (F.ISRM), Fellow of the Australian Institute of Company Directors (FAICD), a Fellow of the Chartered Governance Institute (FCG) and a Fellow of the Governance Institute of Australia (FGIA).
His career spans over 30 years of corporate experience, including executive roles at Deloitte and Citi. His professional interests include corporate governance, risk management, financial management, corporate turnaround and restructuring.
Paul teaches governance, risk management and strategic risk management and is Adjunct Faculty at the Australian Graduate School of Management (AGSM), University of New South Wales (and previously Honorary Professor at the UTS Faculty of Law). Through AcademyGlobal, he has designed and delivered executive programs across 20 countries on five continents.
He is a Non-Executive Director and NSW Councillor of the Australian Institute of Company Directors, a Trustee and Council Member of the Chartered Institute of Public Finance and Accountancy (CIPFA), and Chair of Green Templeton College, Oxford University, Oceania Alumni Chapter.
Paul holds a Master of Commerce from the University of Sydney, a Master of Laws from the University of New South Wales, an MBA from the University of Technology Sydney. He has completed the Advanced Management Program at Green Templeton College, University of Oxford, and the Public Leadership Credential at Harvard Kennedy School of Government.
