Scoring methodology for Our Hospital shortlisting report (FOI)

​Request

The Our Hospital site shortlisting report July 2020 describes the steps taken and criteria applied to reduce the number of sites selected for continued consideration. The report does not include the scoring methodology applied or the resulting scoring output.

Can this be provided for all relevant steps including Step 5?

How does this scoring correlate with the Outcome Matrix included as Appendix 4?

The Outcome Matrix rates both 'Yes' and 'No' responses as 'green' for St Saviours Hospital site for the following criteria:

• is there sufficient space around the hospital building to enhance and support patients, staff and visitors?

• is the site in a quiet location?

• is there enough dedicated car parking and is it suitable?

Can you explain the rationale for this assessment and how it has been incorporated into the scoring process and impacted on the scoring output?

Response

All information requested is contained in the Site Shortlisting Report which has been published on the gov.je website. The pages of the report relevant to each section of the request are set out below.

Scoring Methodology

The process for identifying a short list of sites is illustrated on page 3 of the Site Selection Report:

Site Shortlisting Report

The first four steps illustrated on page 4 describe the process and were not scored. Further detail on each step of the process can be found in the Site Selection Report.

Step 5 of the process involved using a Site Shortlisting Panel to apply the criteria developed by the Citizens’ Panel. This step was scored in line with HM Treasury – The Green Book: Central government guidance on appraisal and evaluation, P58 – long-list appraisal:

The Green Book: Central government guidance on appraisal and evaluation

Individual questions / tests of the Citizens’ Panel criteria were not weighted on a percentage basis.

Green Book guidance provides the following assessments:

• yes (site passes the question / criterion / test)

• no (site fails the question / criterion / test, and does not pass to the next question for appraisal)

• maybe (site passes the question / criterion / test with a compromise or mitigation)

The scoring output was recorded on the Outcome Matrix which is included in Appendix 4 of the Site Selection Report -Application of Citizens’ Panel Criteria.

‘Yes’ and ‘No’ Green Responses

The answers in the Outcome Matrix were colour coded green, amber and red to reflect whether the score is positive, relatively neutral or negative. In some cases, criteria were phrased in such a way that a ‘no’ answer could be a positive response. For example, for the question ‘Will the historic environment / assets be lost or harmed?’ – a ‘no’ would be positive and therefore assigned a green rating. This applied to:

• all criteria, including those identified by the requester

• all sites in the Outcome Matrix, not just St Saviour’s Hospital

Further details of how the Citizens’ Panel’s criteria were developed and applied to produce the site shortlist can be found on pages 9 and 10 of the Site Shortlisting Report.

Internal Review Request

I wish to request an internal review of the Freedom of Information request 231389829.

The basis for the review is set out below:

A

The Our Hospital site shortlisting report July 2020 describes the steps taken and criteria applied to reduce the number of sites selected for continued consideration. The report does not include the scoring methodology applied or the resulting scoring output.

Can this be provided for all relevant steps including Step 5?

The above question is not answered in full. The question required the scoring methodology applied and the scoring output for each step to be provided. The response sets out the scoring methodology only and does not provide the resulting output (i.e. the comparative scores or other ranking attributed to each site that have resulted in content of the Outcome Matrix).

The Site Selection Report (page 10) states:

What was the outcome?

The application by the Site Selection Panel of the sequential test - that had been developed by the Citizens’ Panel - reduced the list of sites under consideration from 17 to 5. Those shortlisted sites are:

Fields to the North of Five Oaks

Millbrook Playing Fields and fields to the north

 Overdale + nearby fields 

 People’s Park + additional nearby site

St Andrew’s Park, First Tower

The output of the Site Selection Panel meeting is an appraisal matrix of all the sites large enough and that would be available in the timescale. The matrix is attached in Appendix 4 – Application of Citizens’ Panel Criteria – Outcome Matrix.

Please provide the actual comparative scores for each site and any weighting applied to criteria evaluated.

B

How does this scoring correlate with the Outcome Matrix included as Appendix 4?

This question has not been answered. A description of the process is provided, but it stops short of explaining how the five shortlisted sites have been chosen from the Matrix Outcomes. It would be reasonable to expect that the sites with the greater number of ‘green’ responses and fewest ‘amber’ and ‘red’ responses would score higher and, therefore, be shortlisted, but this is neither stated nor can it be construed from the Outcome Matrix at Appendix 4.

Hence, the question requires an explanation as to how the five shortlisted sites were selected from the Outcome Matrix as there is no obvious correlation.

Please provide an explanation to show how the five shortlisted sites were selected from the Outcome Matrix.

The scoring of ‘yes’, ‘no’, ‘maybe’ is shown in the matrix.

C

The Outcome Matrix rates both 'Yes' and 'No' responses as 'green' for St Saviours Hospital site for the following criteria:

Is there sufficient space around the hospital building to enhance and support patients, staff and visitors?

Is the site in a quiet location?

Is there enough dedicated car parking and is it suitable?

Can you explain the rationale for this assessment?

These questions have not been answered. The reply explains that criteria may have a positive (green) outcome for either a ‘yes’ or ‘no’ response depending on the nature of the question. This is obvious and the reply borders on the condescending.

My question, which has not been addressed, is that there appears to be a number of discrepancies in the Outcome Matrix where the same response has been reflected differently for the same criteria.

For example, the criteria ‘Is the site in a quiet location?’ has the following conflicting responses:

These responses cannot all be valid. The question seeks and explanation as to why there are conflicting responses for some sites in the three criteria cited in the question.

Please explain why there are conflicting responses for the three criteria cited.

D

- and how it has been incorporated into the scoring process and impacted on the scoring output?

The question refers to the impact of this apparent conflicting scoring. The answer may become clearer if the scoring output requested in the first question is provided. My concern is that the conflicting responses may have been translated into erroneous ‘scores’ that could have impacted on the validity of the choice of five shortlisted sites.

On the other hand, these may be simply typographical errors that have no impact.

Please state if the conflicting responses have had any impact on the shortlisting of the five sites?

Internal Review Response

This review has been completed by two senior members of the Government of Jersey, independent of the original decision-making process.

The original response has been reviewed and assessed at each point to identify whether it had been fully answered. The following changes have been made:

A

Please provide the actual comparative scores for each site and any weighting applied to criteria evaluated.

The first four steps illustrated on page 4 describe the process and were not scored. This statement from the Original Response has been reviewed and is correct. The first four steps are detailed below:

Step 1. Call for sites: scoring was not relevant to this step because as stated on page 5 of the Site Selection Report ‘To maintain the integrity of the process, all sites that were suggested for inclusion on the long list were put through the site shortlisting process.’

Step 2. Clinical criteria for site assessment – site area: scoring was not relevant to this step because this step discounted sites which were not large enough to accommodate the minimum size requirement for the footprint of the new hospital.

Step 3. Clinical criteria for site timetable: scoring was not relevant to this step because this step discounted sites that could not be delivered by 2026.

Step 4. Criteria by the Citizens’ Panel: scoring was not relevant to this step because sites were not assessed in this step.

As noted in the final response:

Step 5 of the process involved using a Site Shortlisting Panel to apply the criteria developed by the Citizens’ Panel. This step was scored in line with HM Treasury Green Book guidance on Long-list Appraisal and Short-list Selection:

The Green Book: Central government guidance on appraisal and evaluation (P58)

In line with this guidance, the answers to each of the Citizens’ Panel’s criteria questions were scored either ‘yes’, ‘no’ or ‘maybe’ based on the evidence. Neither the Citizen’s Panel nor the Our Hospital team hold any other scoring or weighting and sites were not ‘ranked’ of weighted in any way for each question.

B

Please provide an explanation to show how the five shortlisted sites were selected from the Outcome Matrix.

The scoring of ‘yes’, ‘no’, ‘maybe’ is shown in the matrix.

C

Please explain why there are conflicting responses for the three criteria cited.

The answers in the Outcome Matrix were colour coded green, amber and red to reflect whether the score is positive, relatively neutral or negative. In some cases, criteria were phrased in such a way that a ‘no’ answer could be a positive response. The answer provided explained why both ‘Yes’ and ‘No’ responses were colour-coded ‘green’ which is how the question read at the time the response was provided

However, since the Site Shortlisting Report was published (and this review requested) a formatting error in the Outcome Matrix spreadsheet has been identified. The answers for St Saviour’s Hospital for the three questions cited by the requester contained the word ‘no’ instead of ‘yes’.

D

Please state if the conflicting responses have had any impact on the shortlisting of the five sites?

They were correctly colour-coded green and therefore the shortlisting outcomes remain unchanged. The Outcome Matrix has been updated in the Site Shortlisting Report with the original matrix underneath for comparison. This can be viewed at the following link:

Our Hospital Site Shortlisting Report