An introduction to Reasoning Management

David Blayney KC explains how using Associo for "Reasoning Management" enables disputes lawyers and other professionals to develop arguments, conclusions and predictions more efficiently and reliably

The penny dropped as I worked night and day in a leading “hub” role in the defence team in the RBS Rights Issue Litigation. We were engaged in a complex reasoning project and, though experts in our field, we lacked the tools and techniques to do the job efficiently. We lacked them because they simply didn't exist. I realised that this was an extreme instance of a problem that has irked me throughout my professional life, on small cases as well as big ones.

After the RBS case settled at the court door in 2017, I resolved to develop tools and techniques to improve the process of investigating issues and developing arguments, conclusions and predictions. I call the process "Reasoning Management". I recruited a skilled team of designers and software engineers to create a software platform, "Associo", to make the process easier.  In this article I explain what Reasoning Management involves and who can benefit from it.

"Reasoning" is the process of thinking about something in a logical way to form an argument, conclusion, or prediction. Reasoning is typically communicated by describing the analysis in words, for example in a legal opinion.

Many of us do this for a living, including lawyers, accountants, tax advisers and compliance staff. We have clients or employers who need us to provide arguments, conclusions or predictions on matters of importance to them. Do we comply with this regulation? What are the prospects of winning this case? What is the value of this asset or liability?  Can we verify this statement?  In answering these questions, we must make conclusions and predictions.

Improving the reasoning process with new technology and techniques benefits the whole spectrum of professional services, in much the same way that the introduction of spreadsheets transformed the delivery of services requiring numerical calculations.

Structure is the key.  All good reasoning has an underlying logical structure.  Typical reasoning workflows concentrate too heavily on the (often imprecise) words that describe that structure, and pay too little attention to the structure itself. The Reasoning Management approach rectifies that, focusing attention on the logical structure, and using the structure as a framework on which to hang all relevant information so that it can be easily found again when required.  At the heart of this new approach is a method of visualisation that I call a “reasoning graph”.

Let’s start with a simple example involving the application of a legal rule.  Stripped to its essence, a rule might say, for example, “If A is true, and either of B or C is true, then D is true”.  A to C in this example are propositions describing criteria relevant to the application of the rule, whilst D describes a legal outcome.

If you needed to give guidance to colleagues or clients on how to apply this rule, you could provide them with a flowchart as depicted in Figure 1, or you could build them a simple software application acting as an interactive questionnaire leading them through the same steps.

Figure 1

In simple instances involving a series of straightforward choices, the flowchart / questionnaire approach can work very well.  Software platforms such as Bryter and Neota Logic are excellent for this, and many simple business processes follow this approach.

But on issues of nuance or complexity, the flowchart / questionnaire approach breaks down.  This is because flowcharts and questionnaires assume that there is only a single valid path from each point in the analysis.  Each question has a definite answer that takes you down a resultant route.  But most reasoning tends to deal in shades of grey rather than black-and-white conclusions.  Both the intermediate conclusions and the ultimate conclusion need to be expressed as probabilities.

This problem can be overcome by using a reasoning graph. This allows you to build the logical structure of the rule, and then apply “confidence scores” to represent competing arguments and probabilistic conclusions.  Figure 2 shows the same rule, expressed as a reasoning graph with scores applied.

Figure 2

The uncertainty is now managed within the structure.  And by applying different sets of scores, you can view people’s differing judgements within the same analysis. You can also (as we see later) connect evidence to each point, so that all the relevant information is at your fingertips.

As the analysis grows in complexity, you divide it into filtered “Views”, each leading to an important sub-conclusion within the overall analysis.  Because each proposition in the analysis has a single entry in the Associo database, you don’t have the duplication and risk of inconsistency that arises when trying to develop analysis in a long document.

Using reasoning graphs provides an efficient and enjoyable way of applying the kind of rigorous step-by-step analysis that sets apart the most persuasive and highly regarded leaders in their fields. 

Working for 30 years with colleagues at Serle Court and solicitors at leading law firms, I have had the pleasure of engaging with some of the finest minds in the legal profession. The best advisers and advocates are particularly good at the following: (i) holding complex logical structures in the forefront of their minds, (ii) assessing the likelihood of each proposition within the structure being upheld, and (iii) amending the structure and probability assessments when presented with relevant new information.

The use of reasoning graphs goes a long way towards democratising the ability to work with structures in this way. The innate ability to hold complex analysis in your head becomes less important when you can see the structure on the screen.  Amending and adding to the structure, and revising probability assessments, are all much easier on the screen than in your head. Doing this on screen also has the great advantage that the structure is visible in real-time to colleagues, making meetings and calls more productive. And being able to see the analysis makes it much easier to spot mistakes and gaps in the reasoning.

The use of reasoning graphs also facilitates a crucial shift from imprecise and often ambiguous qualitative assessment to clear and precise quantitative assessment.  The importance of this shift, and the proof that it can be made, is compellingly articulated in “Superforecasting, the Art and Science of Prediction” (Tetlock and Gardner, 2015) and “Noise, a Flaw in Human Judgment” (Kahneman, Sibony and Sunstein, 2021). 

The techniques described in Superforecasting and Noise substantially overlap with the approach traditionally adopted by leading lawyers, but with some critically important refinements that have solid evidential backing. The common belief that judgemental assessments need to be expressed in vague terms because quantification is spurious is shown to be wrong by the results of the research described in “Superforecasting”.  Using the right techniques, massive improvements in predictive accuracy can be achieved.

The Superforecasting/Noise recommendations that have yet to be widely adopted by the legal profession (but should be) include: 

• the invocation of “wisdom of crowds” effects by getting individual team members to score each proposition independently, understanding these viewpoints, then looking carefully at the average of those scores before making a final team judgement; and

• the continuous updating of conclusions throughout the analysis as new information becomes available.

Both of these things are very difficult to do when following the traditional approach of developing analysis mentally or in narrative form, but can be done easily using Associo. We teach the full range of these techniques as part of our Reasoning Management training.

In short, there are solid grounds to conclude that the use of Reasoning Management techniques and Associo will enable human teams to make fewer analytical errors and make more consistently accurate predictions than traditional techniques.

This improved precision and accuracy is especially beneficial in contexts where the predictions that are being made relate to assets, liabilities or risks that need to be valued or quantified. Too often, the utility of carefully modelled calculations is undermined by ill-considered assumptions, for example as to the likelihood of an outcome relevant to the calculation.  Using Reasoning Management in combination with financial modelling produces more reliable numerical conclusions.  In my own field, this is particularly useful for providing clients and other stakeholders with regularly updated valuations of their litigation exposures.

The other big advantage of adopting Reasoning Management is increased efficiency, particularly in complex or document-heavy matters.  There are two main routes by which this increased efficiency is achieved.  Both work by making use of the structured approach to developing the analysis.

The first mechanism by which efficiency is increased is by reducing the time spent searching for relevant information, where that information has already been seen by the team but not recorded in a way that makes it easy to retrieve. In the disputes context, this avoidable search for previously seen relevant information contributes to much of the pain experienced when pulling information together in the run-up to trial, and lengthens the drafting time required for every document describing the analysis. It is especially acute where team members leave, holding in their heads an extensive understanding of where relevant information is to be found, which then has to be recreated.

The solution to this problem lies in systematically attaching information to proposition(s) to which it is relevant, at or shortly after the information is found. The information can then be found easily, directly from the proposition that it affects, without any need for searching. In Associo, this is done by maintaining tables of relevant information (documents, chronologies, things that witnesses can say) and connecting the individual pieces of information within these tables (or sets of information, such as a mini-chronology) to the propositions to which they are relevant.  Figure 3 provides an illustration of this.

Figure 3

The second mechanism by which efficiency is increased is by adding focus to the investigation process. It is much easier to find relevant information when you can quickly see the propositions that need to be supported or countered.  Tasks can be allocated to team members directly inside the analysis. 

Structuring analysis opens the door to clever, targeted uses of AI. GPT-4 can already understand code output from Associo that describes reasoning structures. It will be a small step from this to generating first drafts of written advice, statements of case, witness statements, sections of letters etc, straight from reasoning graphs maintained by human experts in Associo.  This will rightly put more emphasis on the substance of the analysis, with less time spent choosing the words to express it. Reasoning graphs can also be generated by AI from narrative inputs (e.g. the text of a statue). 

In this way, professionals will unlock great benefits from AI while maintaining visibility and control of the core activities of building analysis and applying judgement to predict outcomes.

Having devoted much of my time over the last five years to developing this new toolset, I am now able to provide services as a barrister in powerful new ways.  By focusing my time on high-impact tasks such as supervising the development of the analytical structure, reviewing the key judgement-calls, and helping with case valuation calculations, I can add more value per minute than was previously possible, while simultaneously helping to increase the productivity of more junior members of the team.

Colleagues in chambers and professional services firms large and small are starting to see the competitive opportunity that this presents for the delivery of their own services. It is early days, but momentum is building behind this new approach.