Adding a Dash of Statistical Analysis

Two individuals, Adam O’Boyle and Ben Metz, involved in the Foundational Thinking initiative have also been helping out with the Stephen Lloyd Awards, a new fund that remembers the pioneering lawyer Stephen Lloyd by supporting innovative initiatives with the potential for systems change.

Two individuals, Adam O’Boyle and Ben Metz, involved in the Foundational Thinking initiative have also been helping out with the Stephen Lloyd Awards, a new fund that remembers the pioneering lawyer Stephen Lloyd by supporting innovative initiatives with the potential for systems change.

The following approach was taken to manage the first round of the selection process. Led by Jim Clifford, a partner at Bates Wells and Braithwaite, who has an academic background, the process used some simple statistical analysis to harness the goodwill of the 40 people who offered to score applications. As such the approach deserves a write up on the Foundational Thinking website as it is a significant departure from the norm of approaches to project selection in the foundation world…

Adding a Dash of Statistical Analysis to the Stephen Lloyd Awards

The Stephen Lloyd Awards received 103 applications in April 2015. Each comprised a two page summary and one page of supporting materials.

The awards team designed a three stage process, the first stage of which is described here. The intention was to create an efficient and effective approach to selection that engaged as many members of the pro bono support network the awards has built up since its inception, as a way to engage them initially with a view to longer term and more substantive engagements.

An invitation to assess was put to the network. 40 individuals came back indicating they would be willing to score. A scoring system with five criteria, each with a five point scale, was developed. To this was added a comments field and a question as to whether assessors commended the application.

Assessors were sent out an average of 10 applications and were asked to score using the system. The majority of assessors did this within the timescale requested, leading to every application being scored three or four times.

Statistical analysis was undertaken to identify assessors with unusual scoring patterns. Three unusual patterns were identified, each where assessors scored unusually high or low on particular sets of the criteria. These were marked up as “outliers” in the scoring system.

Assessors scores were aggregated to find the average across all assessments. A number of analyses were conducted, including:

  • Averaging without adjusting to outlier assessors
  • Averaging to adjust for outlier assessors
  • Combining high scoring applications with applications that had more than 66% commendation rates.

These were then discussed over the course of a three hour teleconference to identify 19 organisations that went forward to the next stage.

The above outlined process adopts the approaches of academia in the use of statistical analysis and combines with a crowd sourcing approach to undertaking the actual assessment. As such it appears to be unique in the world of foundations and worthy of further consideration.