Prescribing Assessment: News from the UK and Internationally

Published: 29 Jan 2018
Category:

Concerns about prescribing

Prescribing is a core activity for all healthcare systems and most of the doctors who work in them. The average member of the public in the UK will be in receipt of around 20 prescriptions annually which are intended to treat current illnesses and symptoms or prevent future disease. Recent studies have demonstrated significant numbers of prescribing errors and potentially avoidable adverse drug reactions both in hospital and general practice. While the causes are multiple, one important response has been to focus on providing better training pathways, especially for medical students. This has included the development of assessments that might serve as a predictive measurement of prescribing performance.

Why assess prescribing?

There are two principal reasons to consider assessing prescribing. First, there are important patient safety and clinical governance concerns. It is important that all independent prescribers have reached a basic level of performance sufficient to be able to work effectively and safely in the healthcare environment. The required knowledge and skills will be determined by factors such as the likely case-mix, time pressures and the supervision that will be available. Second, there are important educational reasons. For individuals, it is important that they have the opportunity to reflect on their performance in this vital aspect of their work. This can be aided by provision of feedback about specific areas of weakness. Assessments also make it possible for educational institutions (eg medical schools) to reflect on the success of their training programmes, which often vary considerably.

Variability in prescribing assessments

Prescribing assessments come in many forms and the following headings highlight some of the areas of variability:

  1. Coverage. Prescribing has been well defined as ‘A written order, which includes detailed instructions of what medicine should be given to whom, in what formulation and dose, by what route, when, how frequently, and for how long’.7 While prescribing assessments might focus only on this complex action, many reflect the fact that prescribers spend at least as much time supervising and directing the use of medicines in other ways. Therefore, many assessments also extend into other skills such as reviewing prescriptions written by others, communicating about medicines, calculating drug dosage, monitoring the effects of medicines and interpreting data. (table 1)

  2. External validity. Prescribing assessments are intended to serve as surrogate markers of likely performance in the real-world setting. Therefore, the extent to which candidates are faced with realistic challenges is an important factor that determines validity and interpretation. These may range from largely knowledge-based exercises undertaken as Multiple Choice Questions (MCQs) in an examination hall to much more contextualised exercises in either a high-fidelity simulation or even in a real clinical environment

  3. Delivery. The setting for the assessment may range from a remote location to a simulated or real clinical environment and it may be based on paper or an electronic system. The latter offers the opportunity for distributed delivery and automation of marking.

  4. Purpose. This may range from a summative high-stakes assessment that influences progression decisions to a formative exercise that serves to provide feedback, and encourage learning and reflection. For summative assessments there is a need to ensure that there is a suitable standard-setting process in place.

  5. Reliability. An ideal assessment will be reliable, meaning that there can be confidence that a candidate who performs well on one occasion would do so again if re-tested. The factors determining reliability are the number of items in the assessment (i.e. its length), the discrimination of the items between strong and weak candidates and the number of different skills that the assessment tests.

  6. Marking. This should ideally be objective and consistent. Whenever human marking is involved an element of subjectivity is introduced even if it is based on a welldeveloped rubric. Automated electronic marking provides objectivity and avoids the very resourceintensive exercise of human marking, but it can be extremely difficult to mark prescriptions that way.

Noting the considerable variability in structure and delivery, Table 1 offers a simple classification based primarily on the relationship of the assessment to the relevant workplace environment. A very simple analysis would suggest that higher validity (contextualisation) comes at the expense of a lower chance of psychometric reliability, more risk of confounding and a greater requirement for resource.

Table 1
Section Description Marks Question items
1 Prescribing (PWS) 80 8 items of 10 marks
2 Prescription Review (REV) 32 8 items of 4 marks each
3 Planning Management (MAN) 16 8 items of 2 marks each
4 Communicating Information (COM) 12 6 items of 2 marks each
5 Calculation Skills (CAL) 16 8 items of 2 marks each
6 Adverse Drug Reactions (ADR) 16 8 items of 2 marks each
7 Drug Monitoring (TDM) 16 8 items of 2 marks each
8 Data Interpretation (DAT) 12 6 items of 2 marks each
1 Total marks 200  

UK prescribing assessments

Prescribing Safety Assessment (PSA). The PSA has been developed by the British Pharmacological Society and Medical Schools Council Assessment (MSCA) as a national summative assessment of knowledge, judgement and skills related to prescribing and supervising medicines at a basic level in a modern healthcare system. Its primary purpose is to enable final-year medical students to demonstrate that they have achieved the necessary competence to prescribe and supervise the use of medicines at the standard expected of a foundation doctor in an NHS hospital. The PSA is based on the competencies identified by the UK General Medical Council in outcomes for graduates (2015). It is delivered to all UK final-year medical students as a 2-hour online assessment that is divided into 8 sections (Table 2). It is intended to assess, as far as possible within the confines of a virtual environment, complex skills including powers of deduction and problem solving that are relevant to the work of junior doctors in UK hospitals. The innovative delivery system enables around 60,000 prescriptions to be instantaneously assessed against a standardised marking scheme. In 2016, a total of 7,343 final-year students from 31 UK medical schools sat the PSA in 200 individual events held around the UK with an overall pass rate of 95%. There was significant variation in the performance of individual medical school cohorts. The mean Cronbach’s alpha across the four papers used was 0.75. While not quite achieving the nominal 0.8 considered as ideal reliability for high-stakes assessments, this compares favourably with expectations fora multi-domain assessment that is only 2 hours in duration. Prescribing rights have now been extended to other professional groups (e.g. nurse practitioners, pharmacists) and early pilot work has been started to explore the utility of the PSA in that setting.

Table two

Responsive table
Category Example Comment
Non-casebased Multiple-choice questions that require a knowledge of modes of action, indications, contra-indications, adverse effects and interactions of drugs but do not set this in the context of cases encountered in clinical practice. Prescription writing completely absent although some of the answer ‘options’ may be prescriptions. No clinical judgement required. External validity for the workplace largely absent. Content easy to develop. Easy to deliver. Amenable to automated marking. Psychometric reliability easy to achieve.
Low-fidelity simulation Question items have some relation to clinical cases but may not be rich in content (e.g. clinical and medication history, examination findings). Prescription writing may be completely absent although some of the answer ‘options’ may be prescriptions. Little clinical judgement required. Poor external validity for the workplace. Content relatively easy to develop. Easy to deliver. Amenable to automated marking. Psychometric reliability easy to achieve.
Mediumfidelity simulation Question items are based on well-described case content (i.e. clinical history, examination findings and investigations). Clinical experience and judgement very important. Not delivered in a clinical environment. Some external validity for the workplace. Question content challenging to develop because it requires authors with clinical experience who can develop more complex materials demanding judgements. Relatively easy to deliver. Automated marking of prescriptions difficult and often resource intensive. Psychometric reliability more difficult to achieve (because of the collection of skills being tested).
High-fidelity simulation Delivery in a physical space that may contain simulated or real patients. Objective and structured in design. Assessment largely subjective based on individual candidate marking by examiners. Relatively high validity. Resource intensive. Psychometric reliability more difficult to achieve (because of the collection of skills being tested). Automated marking not possible
Real world practice Observational. Based on performance in real world clinical practice. Delivery in a physical space that may contain simulated or real patients. Marking subjective based on assessment of work against pre-specified best-practice standards by individual raters. High validity. Observational. Unstructured. Potential for confounding. Resource intensive. Very hard to test the other skills demanded of prescribers. Reliability poor.

Other UK prescribing assessments

There are numerous local assessments in operation, mostly focusing on screening competence amongst new doctors. The largest has been developed by Kent, Surrey and Sussex local education and training board (LETB) which is based on the combination of a ‘long answer scenario’ and short answer MCQs. Local assessments have the advantage that they are able to provide additional validity in terms of assessing against local protocols using local documentation and decision support. However, they suffer from the significant disadvantage of being expensive to support and maintain and difficult to make reliable. For these reasons, they might be ideally seen as a supplement to a national screening assessment.

International prescribing assessments

There are no international reports of large scale prescribing assessments, like the PSA, that are being delivered widely on an annual basis across multiple institutions, although numerous smaller assessments have been described.16 Two recent international initiatives are worthy of note. The European Association for Clinical Pharmacology and Therapeutics (EACPT) education working group organised a cross-sectional comparison of prescribing-related knowledge among 895 final-year medical students in 26 medical schools from 17 European countries. Each was asked to complete a standardised online assessment (clinical scenarios and MCQs). The majority of prescriptions (55%) contained errors and there were differences between schools. A Dutch national prescribing assessment is currently being piloted, although there is currently no English language description. The successful development and implementation of the PSA in the UK has led to several international collaborations with medical schools in Ireland, Malta, Australia, New Zealand, Canada and India.

Future questions

Important issues for future research have arisen. The assessments above are intended to be surrogate markers of the likely performance of candidates in future clinical practice. The reliability of that prediction should be tested by research that links assessment performance to real world practice. However, addressing this question poses significant challenges. There are numerous other confounding factors that might influence performance measures in the workplace (e.g. workload, supervision, case mix). Furthermore, following up and assessing clinical outcomes for large cohorts over a prolonged period of time will be resource intensive. There is also a need to seek explanations for the variability in performance between cohorts from different institutions. This might help to understand which training programmes are most successful in providing firm foundations for future prescribers.

Comments

Comments
Ahmed Obaid
Prescribers require training, and normal population needs education...
09/02/2018 12:32:45
 

If you are a British Pharmacological Society member, please sign in to post comments.

About the author

Professor Simon Maxwell is Director of Clinical Pharmacology & Therapeutics (CPT) teaching at the University of Edinburgh, where he has been active in developing elearning strategies to support education in this area. He has led a number of national education initiatives on behalf of the British Pharmacological Society including developing a core curriculum for CPT teaching in UK medical schools, the Prescribe e-Learning project undertaken with Healthcare Education England, and the Prescribing Safety Assessment, a joint initiative with the Medical Schools Council to produce a national assessment of prescribing for all UK medical students.

Post archive

No data found


;function _(a){var r=t[a];if(void 0!==r)return r.exports;var n=t[a]={exports: