Systematic reviews related to this topic

loading
137 References (137 articles) Revert Studify

Primary study

Unclassified

Factors predicting doctors' reporting of performance change in response to multisource feedback.

Journal BMC medical education
Year 2012
Links Pubmed DOI

This article is included in 1 Systematic review Systematic reviews (1 reference)

Loading references information
BACKGROUND: Multi-source feedback (MSF) offers doctors feedback on their performance from peers (medical colleagues), coworkers and patients. Researchers increasingly point to the fact that only a small majority of doctors (60-70 percent) benefit from MSF. Building on medical education and social psychology literature, the authors identified several factors that may influence change in response to MSF. Subsequently, they quantitatively studied the factors that advance the use of MSF for practice change. METHODS: This observational study was set in 26 non-academic hospitals in the Netherlands. In total, 458 specialists participated in the MSF program. Besides the collation of questionnaires, the Dutch MSF program is composed of a reflective portfolio and a facilitative interview aimed at increasing the acceptance and use of MSF. All specialists who finished a MSF procedure between May 2008 and September 2010 were invited to complete an evaluation form. The dependent variable was self-reported change. Three categories of independent variables (personal characteristics, experiences with the assessments and mean MSF ratings) were included in the analysis. Multivariate regression analysis techniques were used to identify the relation between the independent variables and specialists' reported change in actual practice. RESULTS: In total, 238 medical specialists (response rate 52 percent) returned an evaluation form and participated in the study. A small majority (55 percent) of specialists reported to have changed their professional performance in one or more aspects in response to MSF. Regression analyses revealed that two variables had the most effect on reported change. Perceived quality of mentoring positively influenced reported change (regression coefficient beta = 0.527, p < 0.05) as did negative scores offered by colleagues. (regression coefficient beta = -0.157, p < 0.05). The explained variance of these two variables combined was 34 percent. CONCLUSIONS: Perceived quality of mentoring and MSF ratings from colleagues seem to be the main motivators for the self-reported change in response to MSF by specialists. These insights could leverage in increasing the use of MSF for practice change by investing in the quality of mentors.

Primary study

Unclassified

Evaluation of procedure-based assessment for assessing trainees' skills in the operating theatre.

Journal The British journal of surgery
Year 2011
Links Pubmed DOI

This article is included in 1 Systematic review Systematic reviews (1 reference)

Loading references information
BACKGROUND: Procedure-based assessment (PBA) is used within most UK surgical training programmes for assessing trainees' procedural skills in the operating theatre. All postgraduate assessment methods require evidence to support their implementation. The aims were to evaluate the validity, reliability and acceptability of PBA. METHODS: Eighty-one trainees in six surgical specialties were assessed performing common procedures; 749 PBAs were provided across 348 operations by 57 clinical supervisors and four independent assessors. RESULTS: Construct validity was demonstrated by correlation of PBA scores with measures of surgical training and experience. Reliability (G exceeding 0·8) of the adjusted total item score and global summary score for a given procedure was achieved using four and three assessor judgements respectively; assessing a mix of procedures required more cases/assessors because performance is procedure specific. The acceptability of PBA for assessment and feedback within surgical training was predominantly positive among clinical supervisors and trainees. CONCLUSION: PBA demonstrated good overall validity and acceptability, and exceptionally high reliability. Trainees should be assessed adequately for each given procedure.

Primary study

Unclassified

What is a good general practitioner (GP)? The development and evaluation of a multi-source feedback instrument for GP appraisal.

Authors Shepherd A , Lough M
Journal Education for primary care : an official publication of the Association of Course Organisers, National Association of GP Tutors, World Organisation of Family Doctors
Year 2010
Links Pubmed

This article is included in 1 Systematic review Systematic reviews (1 reference)

Loading references information
BACKGROUND: Although multi-source feedback (MSF) has been used in primary healthcare, the development of an MSF instrument specific to this setting in the UK has not been previously described. The aims of this study were to develop and evaluate an MSF instrument for GPs in Scotland taking part in appraisal. METHODS: The members of ten primary healthcare teams in the west of Scotland were asked to provide comments in answer to the question, 'What is a good GP?'. The data were reduced and coded by two researchers and questions were devised. Following content validity testing the MSF process was evaluated with volunteers using face-to-face interviews and a postal survey. RESULTS: Thirty-seven statements covering the six domains of communication skills, professional values, clinical care, working with colleagues, personality issues and duties and responsibilities were accepted as relevant by ten primary healthcare teams using a standard of 80 percent agreement. The evaluation found the MSF process to be feasible and acceptable and participants provided some evidence of educational impact. CONCLUSION: An MSF instrument for GPs has been developed based on the concept of 'the good GP' as described by the primary healthcare team. The evaluation of the resultant MSF process illustrates the potential of MSF, when delivered in the supportive environment of GP appraisal, to provide feedback which has the possibility of improving working relationships between GPs and their colleagues.

Primary study

Unclassified

Three methods of multi-source feedback compared: a plea for narrative comments and coworkers' perspectives.

Journal Medical teacher
Year 2010
Links Pubmed DOI

This article is included in 1 Systematic review Systematic reviews (1 reference)

Loading references information
BACKGROUND: Doctor performance assessments based on multi-source feedback (MSF) are increasingly central in professional self-regulation. Research has shown that simple MSF is often unproductive. It has been suggested that MSF should be delivered by a facilitator and combined with a portfolio. AIMS: To compare three methods of MSF for consultants in the Netherlands and evaluate the feasibility, topics addressed and perceived impact upon clinical practice. METHOD: In 2007, 38 facilitators and 109 consultants participated in the study. The performance assessment system was composed of (i) one of the three MSF methods, namely, Violato's Physician Achievement Review (PAR), the method developed by Ramsey et al. for the American Board of Internal Medicine (ABIM), or the Dutch Appraisal and Assessment Instrument (AAI), (ii) portfolio, (iii) assessment interview with a facilitator and (iv) personal development plan. The evaluation consisted of a postal survey for facilitators and consultants. Generalized estimating equations were used to assess the association between MSF method used and perceived impact. RESULTS: It takes on average 8 hours to conduct one assessment. The CanMEDS roles 'collaborator', 'communicator' and 'manager' were discussed in, respectively, 79, 74 and 71% of the assessment interviews. The 'health advocate role' was the subject of conversation in 35% of the interviews. Consultants are more satisfied with feedback that contains narrative comments. The perceived impact of MSF that includes coworkers' perspectives significantly exceeds the perceived impact of methods not including this perspective. CONCLUSIONS: Performance assessments based on MSF combined with a portfolio and a facilitator-led interview seem to be feasible in hospital settings. The perceived impact of MSF increases when it contains coworkers' perspectives.

Primary study

Unclassified

Is multi-source feedback (MSF) seen as a useful educational tool in primary care? A qualitative study.

Authors Owens J
Journal Education for primary care : an official publication of the Association of Course Organisers, National Association of GP Tutors, World Organisation of Family Doctors
Year 2010
Links Pubmed

This article is included in 1 Systematic review Systematic reviews (1 reference)

Loading references information

Primary study

Unclassified

The quality of written comments on professional behaviors in a developmental multisource feedback program.

Journal Academic medicine : journal of the Association of American Medical Colleges
Year 2010
Links Pubmed DOI

This article is included in 1 Systematic review Systematic reviews (1 reference)

Loading references information
BACKGROUND: Written feedback on professional behaviors is an important part of medical training, but little attention has been paid to the quality of written feedback and its expected impact on learning. A large body of research on feedback suggests that feedback is most beneficial when it is specific, clear, and behavioral. Analysis of feedback comments may reveal opportunities to improve the value of feedback. METHOD: Using a directed content analysis, the authors coded and analyzed feedback phrases collected as part of a pilot of a developmental multisource feedback program. The authors coded feedback on various dimensions, including valence (positive or negative) and whether feedback was directed at the level of the self or behavioral performance. RESULTS: Most feedback comments were positive, self-oriented, and lacked actionable information that would make them useful to learners. CONCLUSIONS: Comments often lack effective feedback characteristics. Opportunities exist to improve the quality of comments provided in multisource feedback.

Primary study

Unclassified

Republished paper: Assuring validity of multisource feedback in a national programme.

Authors Archer J , McGraw M , Davies H
Journal Postgraduate medical journal
Year 2010
Links Pubmed DOI

This article is included in 1 Systematic review Systematic reviews (1 reference)

Loading references information
OBJECTIVE: To report the evidence for and challenges to the validity of Sheffield Peer Review Assessment Tool (SPRAT) with paediatric Specialist Registrars (SpRs) across the UK as part of Royal College of Paediatrics and Child Health workplace based assessment programme. DESIGN: Quality assurance analysis, including generalisability, of a multisource feedback questionnaire study. SETTING: All UK Deaneries between August 2005 and May 2006. PARTICIPANTS: 577 year 2 and 4 Paediatric SpRs. INTERVENTIONS: Trainees were evaluated using SPRAT sent to clinical colleagues of their choosing. Data were analysed reporting totals, means and SD, and year groups were compared using independent t tests. A factor analysis was undertaken. Reliability was estimated using generalisability theory. Trainee and assessor demographic details were explored to try to explain variability in scores. MAIN OUTCOME MEASURES: 4770 SPRAT assessments were provided about 577 paediatric SpRs. The mean scores between years were significantly different (Year 2 mean=5.08, SD=0.34, Year 4 mean=5.18, SD=0.34). A factor analysis returned a two-factor solution, clinical care and psychosocial skills. The 95% CI showed that trainees scoring ≥4.3 with nine assessors can be seen as achieving satisfactory performance with statistical confidence. Consultants marked trainees significantly lower (t=-4.52) whereas Senior House Officers and Foundation doctors scored their SpRs significantly higher (SHO t=2.06, Foundation t=2.77). CONCLUSIONS: There is increasing evidence that multisource feedback (MSF) assesses two generic traits, clinical care and psychosocial skills. The validity of MSF is threatened by systematic bias, namely leniency bias and the seniority of assessors. Unregulated self-selection of assessors needs to end.

Primary study

Unclassified

User perceptions of multi-source feedback tools for junior doctors.

Journal Medical education
Year 2010
Links Pubmed DOI

This article is included in 3 Systematic reviews Systematic reviews (3 references)

Loading references information
CONTEXT: The effectiveness of multi-source feedback (MSF) tools, which are increasingly important in medical careers, will be influenced by their users' attitudes. This study compared perceptions of two tools for giving MSF to UK junior doctors, of which one provides mainly textual feedback and one provides mainly numerical feedback. We then compared the perceptions of three groups, including: trainees; raters giving feedback, and supervisors delivering feedback. METHODS: Postal questionnaires about the usability, usefulness and validity of a feedback system were distributed to trainees, raters and supervisors across the north of England. RESULTS: Questionnaire responses were analysed to compare opinions of the two tools and among the different user groups. Overall there were few differences. Attitudes towards MSF in principle were positive and the tools were felt to be usable, but there was little agreement that they could effectively identify doctors in difficulty or provide developmental feedback. The text-oriented tool was rated as more useful for giving feedback on communication and attitude, and as more useful for identifying a doctor in difficulty. Raters were more positive than other users about the usefulness of numerical feedback, but, overall, text was felt to be more useful. Some trainees expressed concern that feedback was based on insufficient knowledge of their work. This was not supported by raters' responses, although many did use indirect information. Trainees selected raters mainly for the perceived value of their feedback, but also based on personal relationships and the simple pragmatics of getting a tool completed. DISCUSSION: Despite positive attitudes to MSF, the perceived effectiveness of the tools was low. There are small but significant preferences for textual feedback, although raters may prefer numerical scales. Concerns about validity imply that greater awareness of contextual and psychological influences on feedback generation is necessary to allow the formative benefits of MSF to be optimised and to negate the risk of misuse in high-stakes contexts.

Primary study

Unclassified

Investigation of trainee and specialist reactions to the mini-Clinical Evaluation Exercise in anaesthesia: implications for implementation.

Journal British journal of anaesthesia
Year 2009
Links Pubmed DOI

This article is included in 2 Systematic reviews Systematic reviews (2 references)

Loading references information
BACKGROUND: The mini-Clinical Evaluation Exercise (mini-CEX) is a workplace-based assessment which may be useful in anaesthesia training. However, its value depends on how supervisors use it with their trainees. This study analyses experience with the mini-CEX after its introduction into anaesthesia departments in our institution. METHODS: We conducted surveys, focus groups, and interviews with trainees and specialists. Data were recorded, transcribed, and entered into NVivo 8. Themes were identified and data coded into these themes. RESULTS: We identified six themes: assessor factors included skills needed to perform the assessments, influences on scoring decisions, and effects on the specialist-trainee relationship; trainee factors related to impact on trainee performance and value at the different training levels; teaching and learning included the effect of focused observation on structuring workplace learning; feedback described how the mini-CEX changed feedback and what was considered useful; mini-CEX process included implementation, initiation of assessments and case selection; and use in assessment included comparisons with existing assessments and the ability to identify poor performers. CONCLUSIONS: Mini-CEX formalized the supervisory relationship, promoting educational interactions. During the observation period, trainees took responsibility for decisions, and specialists learnt more about their abilities. The structured format broadened the scope of feedback and made it easier to address performance gaps. We identified factors that facilitated or hindered implementation, or limited effective feedback and the ability to address poor performance. From this analysis, we propose strategies for the implementation of mini-CEX, and recommendations for assessor training to improve the quality and value of the assessments.

Primary study

Unclassified

Reflection: a link between receiving and using assessment feedback.

Journal Advances in health sciences education : theory and practice
Year 2009
Links Pubmed DOI

This article is included in 1 Systematic review Systematic reviews (1 reference)

Loading references information
PROBLEM STATEMENT AND BACKGROUND: Feedback is essential to learning and practice improvement, yet challenging both to provide and receive. The purpose of this paper was to explore reflective processes which physicians described as they considered their assessment feedback and the perceived utility of that reflective process. METHODS: This is a qualitative study using principles of grounded theory. We conducted interviews with 28 family physicians participating in a multi-source feedback program and receiving scores across the spectrum from high to low. RESULTS: Feedback, especially negative feedback, evoked reflective responses. Reflection seemed to be the process through which feedback was or was not assimilated and appeared integral to decisions to accept and use the feedback. Facilitated reflection upon feedback was viewed as a positive influence for assimilation and acceptance. CONCLUSIONS: Receiving feedback inconsistent with self-perceptions stimulated physicians' reflective processes. The process of reflection appeared instrumental to feedback acceptance and use, suggesting that reflection may be an important educational focus in the formative assessment and feedback process.