Quality metrics for management review: which indicators support decisions

ICH Q10 section 4.1 asks management review to assess performance indicators, and EU GMP 1.4 asks for a state of control shown through monitoring. Both assume the site has metrics and that the metrics mean something. Most sites have plenty of metrics. Fewer have metrics that can change a decision.

What the regulators expect to see measured

EU GMP 1.4 requires the pharmaceutical quality system to establish and maintain a state of control through effective monitoring of process performance and product quality, and to use the results of that monitoring in batch release, deviation investigation and preventive action. The chapter does not name the indicators, and neither does PIC/S PE 009, which carries the same text. What EU inspectors ask for in practice is a small, consistent set: batch acceptance rate, product quality complaint rate, invalidated out-of-specification rate, and the on-time rate of the PQR programme under 1.10. Those four are useful because each bears directly on the state of control that 1.4 describes, and because an inspector who has seen them presented well at other sites will expect to see something like them at yours.

ICH Q10 does not prescribe metrics. Section 3.2.1 describes process performance and product quality monitoring; section 4.1(b) lists the processes whose performance indicators should be assessed: complaint, deviation, CAPA and change management, feedback on outsourced activities, self-assessment, and external assessments. The choice of indicator is left to the site, which is a freedom and a burden. The freedom is that you can measure what matters at your site. The burden is that 'we count deviations' will not satisfy an inspector who asks what the count tells you.

A metric needs four things

1. A definition that does not move. What counts as a deviation, what counts as a complaint, what the denominator is. If the definition changes, the trend breaks and the history is lost. Write the definition down and change it only through change control.
2. A target or a limit. A number without a comparison cannot be judged. A target says what good looks like; a control limit says when the process has changed. Use whichever fits, but use one.
3. A trend long enough to see a pattern. Eight quarters is a reasonable minimum. Two data points is a line, not a trend.
4. A rule for when the review must act. 'Three consecutive periods above target' or 'any single period above the upper limit' turns a graph into a trigger and removes the discretion that lets an uncomfortable trend go unremarked.

Leading and lagging

Lagging indicators tell you what has already gone wrong: complaints, recalls, rejected batches, inspection findings. Leading indicators tell you what is likely to go wrong next: overdue CAPAs, overdue training, validation periodic reviews past due, QC backlog, supplier audits overdue, vacancies unfilled. A review that looks only at lagging indicators is reviewing history. A review that includes leading indicators can make a decision before the complaint arrives.

The most useful leading indicators are usually the dullest: ageing. Ageing of open deviations, of open CAPAs, of open changes, of overdue training, of overdue calibrations. Ageing shows where the system is losing control before the outcome metrics do, and it is almost always a resources conversation, which is exactly the conversation senior management is there to have.

A working set

There is no correct list, but a review that presents something like the following will meet ICH Q10 4.1 and will withstand a metrics-literate inspector.

• Lot acceptance rate, and rejected or reworked batches by cause.
• Deviation rate per batch by classification, with repeat rate by failure mode.
• Invalidated OOS as a proportion of all OOS, and time to complete OOS investigations.
• Complaint rate per units distributed, confirmed rate, and time to close.
• CAPA: on-time closure, overdue and extended, and effectiveness check pass rate.
• Change control: on-time closure and changes closed without effectiveness assessment.
• Audit and inspection findings open, by classification and age.
• Supplier: rejection rate, supplier-attributed deviations, audit programme status.
• PQR on-time completion under 1.10 and open PQR actions under 1.11.
• Training overdue by department, validation periodic reviews overdue, calibration overdue.
• QC release backlog and document review backlog, in days.
• Headcount against plan and vacancy duration in quality-critical roles.

How to present a metric so the review can act

One graph per metric, with the target or limit drawn on it, the trend rule stated, and, where the rule is triggered, a paragraph that explains why and a question for the review. Do not present a dashboard of thirty tiles in green, amber and red. A dashboard tells the review that everything has been looked at; it does not tell the review what to decide. The metric owner should present, not QA on their behalf, and the owner should already know what they are asking for.

Mistakes that turn a metric into decoration

• Changing the definition mid-series so the trend improves.
• Presenting counts when the volume has changed.
• Setting targets that are always met, so the review never has to act.
• Showing a metric with no owner, so nobody in the room can explain it.
• Reporting on-time closure without reporting effectiveness.
• Presenting the same graph every quarter with no commentary, so the review learns to skip it.

Module 3 of the course builds a metric set for a pharmaceutical site, tests each metric against the four requirements, and rewrites a real dashboard so that every tile ends in a question the review can answer.