How accurate are litigators at predicting case outcomes?
Research has found that an attorney’s estimate of the probability of success is the most crucial variable in shaping decisions whether to litigate or settle a case in controversy (see, for review, Goodman-Delahunty et al., 2010; Jeklic, 2023).
Unlike parties to a case — who frequently exhibit an overconfidence about prevailing in court — attorneys are expected and trained to be more objective in their predictions. Said differently, attorneys are expected not to have a ‘myside’ bias.
How realistic are attorneys’ case outcome predictions? Do attorneys forecast outcomes accurately or do they exhibit a myside bias not unlike the parties to the dispute?
Much research has been done on this issue, spanning years, types of attorneys, adjudicators, and issues being decided.
Research Study 1 – Defense Attorneys
Loftus and Wagenaar (1988) conducted two studies about case outcome predictions of defense attorneys.
In one of the studies, the ability to predict an actual case outcome was assessed for 182 trial attorneys, 93 practicing primarily criminal defense and 99 primarily civil defense. These attorneys were asked to think of a case that would be going to trial soon, provide their minimum goal for the case outcome, and then estimate their chance of obtaining that goal. Verdicts then were compared to attorney estimates of success. Attorneys were found to be overconfident in their chances of winning, especially so in cases in which they had been highly confident to begin with.
In the other study, 84 criminal defense attorneys were asked to predict whether trial lawyers would be overconfident, well-calibrated or underconfident in their estimates of their chances of winning. Over half of the attorneys incorrectly predicted that lawyers would be well-calibrated or underconfident in their predictions.
Research Study 2 – Civil & Criminal Attorneys
Goodman-Delahunty and colleagues (2010) investigated realism in case outcome predictions of 481 trial attorneys across the United States, including 182 civil plaintiff and 155 civil defense trial attorneys, and 79 prosecutors and 65 criminal defense trial attorneys. The trial attorneys specified a minimum goal to achieve in one of their cases set for trial and estimated the chance of meeting the goal. The civil cases included tort cases (57%), contract cases (19%) and workplace disputes (9%), with the balance involving family dissolutions, wills and trust, copyright, trademark and patent law, bankruptcy, taxation and administrative matters. The criminal cases involved crimes against persons (64%) such as homicide, rape, robbery, assault and battery as well as victimless crimes and crimes against property (36%) such as controlled substances and misdemeanors. After the cases were resolved, case outcomes were compared with predictons and the attorneys were interviewed.
The trial attorneys were overconfident in their predictions, and calibration did not increase with years of legal experience. The majority of trial attorneys expressed confidence estimates that exceeded 50% (mean and median estimates of 64% and 70%, respectively), a logical impossibility. Lawyers making case outcome estimates exceeding 65% exhibited myside bias, that is, were inaccurately overconfident. Female lawyers were slightly better calibrated than their male counterparts and showed evidence of less overconfidence. Asking attorneys to generate reasons why they might not be able to achieve their goal did not improve calibration. Estimates about cases with a trial date in the very near future were not more accurate than estimates about cases with a trial date farther out. Lawyers sometimes claimed they had achieved their goal even when an explicit comparison proved they had not. Fewer than one-fifth of the lawyers (18%) were disappointed with the case outcome, although more than twice as many (43%) actually failed to achieve their stated minimum goal.
Research Study 3 – Judicial Awards & Argument Quality
Jeklic (2023) investigated the prevalence of myside bias on estimates of judicial award and perceptions of the persuasiveness of legal arguments among 166 law students. The law students were given the exact same facts and law about a personal injury case in which liability was pre-decided, randomly assigned to a side they would be representing, told that their task was to develop arguments about damages, estimated the amount a judge would award the plaintiff, and then rated the persuasiveness of eight case-related damages arguments (four favoring each side). The law students also were asked whether the fact that they represented one side influenced their estimates and ratings, and what they thought the other side estimated a judge would award the plaintiff. The law students were then asked to assume their own award estimate was wrong and think of reasons why. Finally, the law students were asked to make a final estimate of what they thought the judge would award the plaintiff.
The predicted judicial awards of plaintiff representatives were 69% higher than the estimates of defendant representatives. Both the plaintiff and defense representatives thought the arguments supporting their side were 30% more convincing than the arguments supporting the opposing side. Being exposed to the arguments for both sides did not change the myside bias in the estimates of judicial award: the plaintiff’s representatives still expected the award to be significantly higher than the defendants’ representatives. Over 50% of the representatives reported that they felt they were not at all or only marginally biased in their award estimates and evaluations of argument persuasiveness by fact they represented one side.
Debiasing strategies reduced, but did not eliminate the representatives’ myside bias. Thinking of reasons why one’s estimate was wrong reduced the myside bias in representatives’ estimates of judicial award, but the difference between estimates of plaintiff and defense representatives persisted and, further, 19% of the law student represenatvies used this reason-generation procedure to arrive at award estimates that were even more in their own favor than before. The final estimates of plaintiff representatives were still 21% higher than those of defendant representatives.
Conclusion
Trial attorneys quite often are overly confident of their chances of prevailing in court; that is, trial attorneys exhibit a myside bias.
Trial attorneys with more practical experience are not better predictors of case outcomes than trial attorneys with less experience. Female trial attorneys are somewhat better at predicting case outcomes than male trial attorneys.
While myside debiasing interventions such as counter-argument generation, counter-argument exposure, reason generation for why failure might happen, etc. can reduce myside bias, the interventions also can backfire and overconfidence in case outcome prediction persists at a significant level.
SourceGoodman-Delahunty, J., Granhag, P.A., Hartwig, M. & Loftus, E.F. (2010). Insightful or wishful: Lawyers’ ability to predict case outcomes. Psychology, Public Policy, and Law, 16(2), pp. 133-157.
SourceJeklic, M.A. (2023). Can you trust your lawyer’s call? Legal advisers exhibit myside bias resistant to debiasing interventions. Journal of Empirical Legal Studies, 20(2), pp. 409-433.
SourceLoftus, E.F. & Wagenaar, W.A. (2022). Lawyers’ predictions of success. Jurimetrics Journal, 28(4), pp. 437-453.