BACKGROUND: Multi-source feedback (MSF) offers doctors feedback on their performance from peers (medical colleagues), coworkers and patients. Researchers increasingly point to the fact that only a small majority of doctors (60-70 percent) benefit from MSF. Building on medical education and social psychology literature, the authors identified several factors that may influence change in response to MSF. Subsequently, they quantitatively studied the factors that advance the use of MSF for practice change.
METHODS: This observational study was set in 26 non-academic hospitals in the Netherlands. In total, 458 specialists participated in the MSF program. Besides the collation of questionnaires, the Dutch MSF program is composed of a reflective portfolio and a facilitative interview aimed at increasing the acceptance and use of MSF. All specialists who finished a MSF procedure between May 2008 and September 2010 were invited to complete an evaluation form. The dependent variable was self-reported change. Three categories of independent variables (personal characteristics, experiences with the assessments and mean MSF ratings) were included in the analysis. Multivariate regression analysis techniques were used to identify the relation between the independent variables and specialists' reported change in actual practice.
RESULTS: In total, 238 medical specialists (response rate 52 percent) returned an evaluation form and participated in the study. A small majority (55 percent) of specialists reported to have changed their professional performance in one or more aspects in response to MSF. Regression analyses revealed that two variables had the most effect on reported change. Perceived quality of mentoring positively influenced reported change (regression coefficient beta = 0.527, p < 0.05) as did negative scores offered by colleagues. (regression coefficient beta = -0.157, p < 0.05). The explained variance of these two variables combined was 34 percent.
CONCLUSIONS: Perceived quality of mentoring and MSF ratings from colleagues seem to be the main motivators for the self-reported change in response to MSF by specialists. These insights could leverage in increasing the use of MSF for practice change by investing in the quality of mentors.
Journal»Education for primary care : an official publication of the Association of Course Organisers, National Association of GP Tutors, World Organisation of Family Doctors
BACKGROUND: Although multi-source feedback (MSF) has been used in primary healthcare, the development of an MSF instrument specific to this setting in the UK has not been previously described. The aims of this study were to develop and evaluate an MSF instrument for GPs in Scotland taking part in appraisal.
METHODS: The members of ten primary healthcare teams in the west of Scotland were asked to provide comments in answer to the question, 'What is a good GP?'. The data were reduced and coded by two researchers and questions were devised. Following content validity testing the MSF process was evaluated with volunteers using face-to-face interviews and a postal survey.
RESULTS: Thirty-seven statements covering the six domains of communication skills, professional values, clinical care, working with colleagues, personality issues and duties and responsibilities were accepted as relevant by ten primary healthcare teams using a standard of 80 percent agreement. The evaluation found the MSF process to be feasible and acceptable and participants provided some evidence of educational impact.
CONCLUSION: An MSF instrument for GPs has been developed based on the concept of 'the good GP' as described by the primary healthcare team. The evaluation of the resultant MSF process illustrates the potential of MSF, when delivered in the supportive environment of GP appraisal, to provide feedback which has the possibility of improving working relationships between GPs and their colleagues.
BACKGROUND: Doctor performance assessments based on multi-source feedback (MSF) are increasingly central in professional self-regulation. Research has shown that simple MSF is often unproductive. It has been suggested that MSF should be delivered by a facilitator and combined with a portfolio.
AIMS: To compare three methods of MSF for consultants in the Netherlands and evaluate the feasibility, topics addressed and perceived impact upon clinical practice.
METHOD: In 2007, 38 facilitators and 109 consultants participated in the study. The performance assessment system was composed of (i) one of the three MSF methods, namely, Violato's Physician Achievement Review (PAR), the method developed by Ramsey et al. for the American Board of Internal Medicine (ABIM), or the Dutch Appraisal and Assessment Instrument (AAI), (ii) portfolio, (iii) assessment interview with a facilitator and (iv) personal development plan. The evaluation consisted of a postal survey for facilitators and consultants. Generalized estimating equations were used to assess the association between MSF method used and perceived impact.
RESULTS: It takes on average 8 hours to conduct one assessment. The CanMEDS roles 'collaborator', 'communicator' and 'manager' were discussed in, respectively, 79, 74 and 71% of the assessment interviews. The 'health advocate role' was the subject of conversation in 35% of the interviews. Consultants are more satisfied with feedback that contains narrative comments. The perceived impact of MSF that includes coworkers' perspectives significantly exceeds the perceived impact of methods not including this perspective.
CONCLUSIONS: Performance assessments based on MSF combined with a portfolio and a facilitator-led interview seem to be feasible in hospital settings. The perceived impact of MSF increases when it contains coworkers' perspectives.
Journal»Education for primary care : an official publication of the Association of Course Organisers, National Association of GP Tutors, World Organisation of Family Doctors
CONTEXT: The effectiveness of multi-source feedback (MSF) tools, which are increasingly important in medical careers, will be influenced by their users' attitudes. This study compared perceptions of two tools for giving MSF to UK junior doctors, of which one provides mainly textual feedback and one provides mainly numerical feedback. We then compared the perceptions of three groups, including: trainees; raters giving feedback, and supervisors delivering feedback.
METHODS: Postal questionnaires about the usability, usefulness and validity of a feedback system were distributed to trainees, raters and supervisors across the north of England.
RESULTS: Questionnaire responses were analysed to compare opinions of the two tools and among the different user groups. Overall there were few differences. Attitudes towards MSF in principle were positive and the tools were felt to be usable, but there was little agreement that they could effectively identify doctors in difficulty or provide developmental feedback. The text-oriented tool was rated as more useful for giving feedback on communication and attitude, and as more useful for identifying a doctor in difficulty. Raters were more positive than other users about the usefulness of numerical feedback, but, overall, text was felt to be more useful. Some trainees expressed concern that feedback was based on insufficient knowledge of their work. This was not supported by raters' responses, although many did use indirect information. Trainees selected raters mainly for the perceived value of their feedback, but also based on personal relationships and the simple pragmatics of getting a tool completed.
DISCUSSION: Despite positive attitudes to MSF, the perceived effectiveness of the tools was low. There are small but significant preferences for textual feedback, although raters may prefer numerical scales. Concerns about validity imply that greater awareness of contextual and psychological influences on feedback generation is necessary to allow the formative benefits of MSF to be optimised and to negate the risk of misuse in high-stakes contexts.
PROBLEM STATEMENT AND BACKGROUND: Feedback is essential to learning and practice improvement, yet challenging both to provide and receive. The purpose of this paper was to explore reflective processes which physicians described as they considered their assessment feedback and the perceived utility of that reflective process.
METHODS: This is a qualitative study using principles of grounded theory. We conducted interviews with 28 family physicians participating in a multi-source feedback program and receiving scores across the spectrum from high to low.
RESULTS: Feedback, especially negative feedback, evoked reflective responses. Reflection seemed to be the process through which feedback was or was not assimilated and appeared integral to decisions to accept and use the feedback. Facilitated reflection upon feedback was viewed as a positive influence for assimilation and acceptance.
CONCLUSIONS: Receiving feedback inconsistent with self-perceptions stimulated physicians' reflective processes. The process of reflection appeared instrumental to feedback acceptance and use, suggesting that reflection may be an important educational focus in the formative assessment and feedback process.
OBJECTIVES: Delivery of 360-degree feedback is widely used in revalidation programmes. However, little has been done to systematically identify the variables that influence whether or not performance improvement is actually achieved after such assessments. This study aims to explore which factors represent incentives, or disincentives, for consultants to implement suggestions for improvement from 360-degree feedback.
METHODS: In 2007, 109 consultants in the Netherlands were assessed using 360-degree feedback and portfolio learning. We carried out a qualitative study using semi-structured interviews with 23 of these consultants, purposively sampled based on gender, hospital, work experience, specialty and views expressed in a previous questionnaire. A grounded theory approach was used to analyse the transcribed tape-recordings.
RESULTS: We identified four groups of factors that can influence consultants' practice improvement after 360-degree feedback: (i) contextual factors related to workload, lack of openness and social support, lack of commitment from hospital management, free-market principles and public distrust; (ii) factors related to feedback; (iii) characteristics of the assessment system, such as facilitators and a portfolio to encourage reflection, concrete improvement goals and annual follow-up interviews, and (iv) individual factors, such as self-efficacy and motivation.
CONCLUSIONS: It appears that 360-degree feedback can be a positive force for practice improvement provided certain conditions are met, such as that skilled facilitators are available to encourage reflection, concrete goals are set and follow-up interviews are carried out. This study underscores the fact that hospitals and consultant groups should be aware of the existing lack of openness and absence of constructive feedback. Consultants indicated that sharing personal reflections with colleagues could improve the quality of collegial relationships and heighten the chance of real performance improvement.
INTRODUCTION: Receiving negative performance feedback can elicit negative emotional reactions which can interfere with feedback acceptance and use. This study investigated emotional responses of family physicians' participating in a multi-source feedback (MSF) program, sources of these emotions, and their influence upon feedback acceptance and use.
METHODS: The authors interviewed 28 volunteer family physician participants in a pilot study of MSF, purposefully recruited to represent the range of scores. The study was conducted in 2003-2004 at Dalhousie University.
RESULTS: Participants' emotional reactions to feedback appeared to be elicited in response to an internal comparison of their feedback with self-perceptions of performance. Those agreeing with their feedback; i.e., perceiving it as generally consistent with or higher than self-perceptions responded positively, while those disagreeing with their feedback; i.e., seeing it as generally inconsistent with or lower than self-perceptions, generally responded with distress. For the latter group, these feelings were often strong and long-lasting. Some eventually accepted their feedback and used it for change following a long period of reflection. Others did not and described an equally long reflective period but one which focused on and questioned MSF procedures rather than addressed feedback use. Participants suggested providing facilitated reflection on feedback to enhance assimilation of troubling emotions and interpretation and use of feedback.
CONCLUSIONS: Negative feedback can evoke negative feelings and interfere with its acceptance. To overcome this, helpful interventions may include raising awareness of the influence of emotions, assisting recipients to focus their feedback on performance tasks, and providing facilitated reflection on feedback.
OBJECTIVE: To determine whether augmenting standard feedback on resident performance with a multisource feedback intervention improved pediatric resident communication skills and professionalism.
DESIGN: Randomized controlled trial.
SETTING: Children's Hospital Medical Center, Cincinnati, Ohio, from June 21, 2004, to July 7, 2005.
PARTICIPANTS: Thirty-six first-year pediatric residents.
INTERVENTIONS: Residents assigned to the multisource feedback group (n = 18) completed a self-assessment, received a feedback report about baseline parent and nurse evaluations, and participated in a tailored coaching session in addition to receiving standard feedback. Residents in the control group (n = 18) received standard feedback only. The control group and their residency directors were blinded to parent and nurse evaluations until the end of the study.
MAIN OUTCOME MEASURES: Residents' specific communication skills and professional behaviors were rated by parents and nurses of pediatric patients. Both groups were evaluated at baseline and after 5 months. Scores were calculated on each item as percentage in the highest response category.
RESULTS: Both groups had comparable baseline characteristics and ratings. Parent ratings increased for both groups. While parent ratings increased more for the multisource feedback group, differences between groups were not statistically significant. In contrast, nurse ratings increased for the multisource feedback group and decreased for the control group. The difference in change between groups was statistically significant for communicating effectively with the patient and family (35%; 95% confidence interval, 11.0%-58.0%), timeliness of completing tasks (30%; 95% confidence interval, 7.9%-53.0%), and demonstrating responsibility and accountability (26%; 95% confidence interval, 2.9%-49.0%).
CONCLUSION: A multisource feedback intervention positively affected communication skills and professional behavior among pediatric residents.
CONTEXT: Multisource feedback (MSF) is a type of formative assessment intended to guide learning and performance change. However, in earlier research, some doctors questioned its validity and did not use it for improvement, raising questions about its consequential validity (i.e. its ability to produce intended outcomes related to learning and change). The purpose of this qualitative study was to increase understanding of the consequential validity of MSF by exploring how doctors used their feedback and the conditions influencing this use.
METHODS: We used interviews with open-ended questions. We purposefully recruited volunteer participants from 2 groups of family doctors who participated in a pilot assessment of MSF: those who received high (n = 25) and those who received average/lower (n = 44) scores.
RESULTS: Respondents included 12 in the higher- and 16 in the average/lower-scoring groups. Fifteen interpreted their feedback as positive (i.e. confirming current practice) and did not make changes. Thirteen interpreted feedback as negative in 1 or more domains (i.e. not confirming their practice and indicating need for change). Seven reported making changes. The most common changes were in patient and team communication; the least common were in clinical competence. Positive influences upon change included receiving specific feedback consistent with other sources of feedback from credible reviewers who were able to observe the subjects. These reviewers were most frequently patients.
DISCUSSION: Findings suggest circumstances that may contribute to low consequential validity of MSF for doctors. Implications for practice include enhancing procedural credibility by ensuring reviewers' ability to observe respective behaviours, enhancing feedback usefulness by increasing its specificity, and considering the use of more objective measures of clinical competence.
Multi-source feedback (MSF) offers doctors feedback on their performance from peers (medical colleagues), coworkers and patients. Researchers increasingly point to the fact that only a small majority of doctors (60-70 percent) benefit from MSF. Building on medical education and social psychology literature, the authors identified several factors that may influence change in response to MSF. Subsequently, they quantitatively studied the factors that advance the use of MSF for practice change.
METHODS:
This observational study was set in 26 non-academic hospitals in the Netherlands. In total, 458 specialists participated in the MSF program. Besides the collation of questionnaires, the Dutch MSF program is composed of a reflective portfolio and a facilitative interview aimed at increasing the acceptance and use of MSF. All specialists who finished a MSF procedure between May 2008 and September 2010 were invited to complete an evaluation form. The dependent variable was self-reported change. Three categories of independent variables (personal characteristics, experiences with the assessments and mean MSF ratings) were included in the analysis. Multivariate regression analysis techniques were used to identify the relation between the independent variables and specialists' reported change in actual practice.
RESULTS:
In total, 238 medical specialists (response rate 52 percent) returned an evaluation form and participated in the study. A small majority (55 percent) of specialists reported to have changed their professional performance in one or more aspects in response to MSF. Regression analyses revealed that two variables had the most effect on reported change. Perceived quality of mentoring positively influenced reported change (regression coefficient beta = 0.527, p < 0.05) as did negative scores offered by colleagues. (regression coefficient beta = -0.157, p < 0.05). The explained variance of these two variables combined was 34 percent.
CONCLUSIONS:
Perceived quality of mentoring and MSF ratings from colleagues seem to be the main motivators for the self-reported change in response to MSF by specialists. These insights could leverage in increasing the use of MSF for practice change by investing in the quality of mentors.
