Evaluating Arbitrators: You Be The Judge

In 2007/8, I set up a now long-since defunct website (disputesloop.com) with the lofty aim of introducing some transparency into the appointment process of ADR neutrals. At its core, the site offered a battery of neutrals’ CVs supplemented with free-form written feedback from users.

On one level, it aimed to substitute the kind of third-hand market gossip that makes some ADR careers and breaks others, with something substantial, first-person and, importantly, in the public domain. I was keen for this feedback repository to develop into a body of market knowledge to inform the uninitiated and to develop a framework for objective discussion of the competencies of mediators and arbitrators.

The hope was that users would become more discriminating in their choice of ADR professionals, more concerned to match skills and competences to the requirements of the case, and more articulate about what they wanted from ADR professionals. I also hoped that the site would guide the market’s invisible hand to push out those who overtraded and under-delivered, while rewarding the capable and conscientious.

So far, so naïve. 150 or so noble souls signed up (and paid!) to offer their services via the site, and to subject themselves to the market’s slings and arrows. The only constraint on clients was an insistence that comments be attributed. And over time, and with some prompting, feedback did appear on the site.

You’d be right to infer, however, from the site’s non-existence that the venture was not a roaring success. The trouble was that the feedback wasn’t terribly valuable and the market wasn’t terribly interested. With hindsight, the reasons are entirely predictable. The first problem with the site was lack of confidentiality leading to lack of candour. Being mostly British, the comments were mostly polite. Anodyne, one might say.

But so what? If an arbitrator was ‘well prepared’, ‘responsive’, ‘polite’ and the award was ‘well-reasoned’ and ‘promptly delivered’, is that not a good enough reason to consider him or her for the next appointment? Well, no, actually.

Pinpointing arbitrators to avoid

I went back to my market (private practice lawyers) to investigate their desultory interest in the site. The message was clear: the positive attributes were somewhat helpful, but the information they really valued what not ‘who’s any good’, but ‘who’s seriously dreadful’. Knowing who to avoid was, and remains, the aim of the game. A good arbitrator does not guarantee a good result, but a careless, disorganised, disreputable arbitrator does guarantee the destruction of a hard-won client relationship.

If you are imagining that my site’s ultimate demise came as a result of defamation claims from angry arbitrators, you’d be wrong. Nobody wants to criticise an arbitrator in a public forum for fear of reprisals in any future hearing. That, too, is unhealthy and sustains many a creaking practice.

However, feedback is not the ne plus ultra of transparency that some imagine it to be. It’s biased (consciously and unconsciously), flawed, and ultimately misleading. Mediators, particularly, can get a raw deal. In caucus sessions, parties don’t and cannot know what’s going on in the other room. Your mediator might be working hard with one party to achieve a shift in attitude; it might take hours, and it might ultimately fail. The other party, feeling abandoned, and with no settlement forthcoming might feel justified in giving poisonous feedback. And sometimes they do; it’s first-hand, objective, but also entirely unfair.

Lack of fairness, however, is not a reason to refuse to engage with feedback; rather, it is something that must be factored in. If one dismisses both the unmerited abuse and unjustified praise, somewhere in the middle are valuable insights. The mere fact that such feedback has been put in writing gives it a status and credibility above word-of-mouth recommendations, which exert disproportionate influence.

Rumour has it

Almost a decade ago, I had an illuminating experience of the shortcomings of word-of-mouth feedback. I shadowed one UK mediator whom I thought I knew well from feedback accumulated verbally over the years. I thought I had a good sense of his style and how he might approach the case and handle the parties. But to my great surprise the caricature assembled from multiple sources in no way resembled the reality. I was confounded on every front: where I’d anticipated an austere, and forbidding manner, he was levity, modesty and bonhomie. Where I’d anticipated the heavy slugging of evaluation, he tiptoed around the issues with finesse and grace. Where I’d expected an imposing ego, I found a man who, being shown the door, stood obligingly in the corridor for hours at a time.

Two further points arise from this encounter: first (tangentially), I’d been told by one law firm involved just days before that the mediator concerned was one that they would never use again. His name was inked in their black book under the heading ‘never more’. Patent nonsense. Secondly, whenever I’ve related this experience to people in the market they’ve refused to believe me: he was putting on an act for your benefit, they say. Such is the incontrovertible power of rumour.

You be the judge

Flawed or not, market feedback is here to stay. The old adage that you can’t improve what you can’t measure applies to ADR. And while we’re still in the early stages of understanding what and how to measure performance, it remains in my view critical that the market is given a voice and provoked into dialogue.

Written by Matthew Rushton.

This is an adapted version of the original article which was first published on the JAMS website, to view it please click here.



Matthew Rushton is the Deputy Managing Director of JAMS International. He is the founder of Empirical Publishing, an online legal publisher focused on ADR. Matthew was previously the publisher of The Mediator Magazine and later The Mediator Directory, as well as launching an ADR brokerage site, DisputesLoop.com. He is a regular speaker on ADR topics at conferences.

About The Author

2 thoughts on “Evaluating Arbitrators: You Be The Judge”

  1. Pingback: In Conversation With Michael McIlwrath – Global Pound Conference Blog

  2. Pingback: In Conversation With Michael McIlwrath - Global Pound Conference Blog

Leave a Comment

Your email address will not be published. Required fields are marked *

Shopping Cart
Scroll to Top
Scroll to Top