|
Advertisement |
|
|
||
 |
|
|||||
|
|||||
|
 |
||||||
|
||||||
|
||||||
|
Originally published as JCO Early Release 10.1200/JCO.2008.20.3091 on February 17 2009 © 2009 American Society of Clinical Oncology.
Developing an In-Training Examination for Fellows: The Experience of the American Society of Clinical OncologyFrom the University of North Carolina at Chapel Hill, Chapel Hill, NC; American Society of Clinical Oncology, Alexandria, VA; Vanderbilt-Ingram Cancer Center, Nashville, TN; National Board of Medical Examiners, Philadelphia, PA; Cleveland Clinic Foundation, Cleveland, OH; and Baystate Medical Center, Springfield, MA. Corresponding author: Frances A. Collichio, MD, Division of Hematology/Oncology, University of North Carolina, Chapel Hill, NC 27599-7305; e-mail: fcollich{at}med.unc.edu.
The American Society of Clinical Oncology (ASCO) developed its own test—the Medical Oncology In-Training Examination (MedOnc ITE)—as a tool to assess trainees' knowledge of the clinical oncology subspecialty, establish consistency in educational standards across training programs, identify areas of strength and weakness in individual programs, and stimulate intraprogrammatic reading and discussion. The Accreditation Council for Graduate Medical Education Outcome Project provided additional incentive for ASCO to develop an ITE. The examination was developed in 4 years. The concept of the examination and the budget were approved by the ASCO governing board. The National Board of Medical Examiners was selected to work with ASCO. Fellowship programs were contacted to determine if they had the information technology support to hold the examination. A blueprint for the examination was developed. The test format, including the number of questions and the selection of case-based single best answers, was determined. Physician volunteers to write the questions were solicited from among program directors, various ASCO committees, and disease experts. A workshop was held to teach volunteers how to write proper case-based questions. From this pool, a smaller group of physicians was selected to develop the test and review all test questions. The final examination was developed and administered in February 2008, with scores provided to fellows and program directors in April 2008. Feedback received after the examination will be helpful for developing future MedOnc ITEs. The process ASCO went through to develop the MedOnc ITE serves as a model for other subspecialties interested in developing their own ITEs.
Graduate medical education is becoming increasingly complex, and tools to determine the proficiency of training programs and trainees are needed. Although emphasis has been placed recently on expanding assessment and demonstrating outcomes in proficiencies other than medical knowledge, evaluation of the depth and breadth of residents' knowledge base remains critically important, because this constitutes the foundation on which clinical competence is built, and is an essential requisite for the development of sound clinical reasoning skills.1,2 The American College of Physicians, Association of Program Directors in Internal Medicine, and Association of Professors of Medicine recognized the importance of assessing knowledge more than two decades ago and developed an in-training examination (ITE) for internal medicine.3,4 Residents have used their results to identify areas of deficiency that require additional study, compare their performance with that of their peers in training programs throughout the world, and track their improvement over time.4,5 Program directors have used the results to provide educational counseling to residents, develop teaching programs, and validate the need for changes in curriculum.3 The implementation and continued evolution of the Accreditation Council for Graduate Medical Education (ACGME) Outcome Project, which requires demonstration of educational outcomes in six competencies (ie, medical knowledge, patient care, practice-based learning and improvement, systems-based practice, professionalism, and interpersonal and communication skills), provide additional incentive for program directors to introduce high-quality assessment methods into their programs.1,5,6 Furthermore, the third phase of the ACGME accreditation standards now requires the introduction of external measures to verify resident and program performance.1 Based on these driving factors—the complex medical environment, the importance of a sound knowledge base and related competencies, and the need to use reliable and valid measures to assess resident competence—the American Society of Clinic Oncology (ASCO) developed its own test—the Medical Oncology (MedOnc) ITE—and included questions that would not only reveal the examinees' knowledge base but also assess other aspects of the medical competencies. This article describes the background for this endeavor, the step-wise process that was undertaken, the purpose of the test format, and future goals.
The goals of the examination were first, to identify areas of strength and weakness in individual programs; second, to assess knowledge (and other competencies, where appropriate) in the subspecialty; third, to establish consistency in educational standards across training programs; fourth, to provide a tool to improve training; fifth, to stimulate intraprogrammatic reading and discussion; and sixth, to provide feedback to individual trainees regarding their strengths and weaknesses in specific content areas and compare their results against national outcomes as delineated in the score reports. To meet these goals, several factors were considered. The examination had to test the application of knowledge in real-life situations, rather than the simple recall of facts. Questions and answers had to be accurate and logical. Ideally, in addition to assessing medical knowledge, the examination would provide information on other competencies, where appropriate. The examination had to be electronic, easy to grade, and reproducible, and contain questions that could be used in the future. Data related to the examination had to be assessable for research purposes.
The MedOnc ITE was administered electronically over 6 hours as a 200-question examination. This included breaks and orientation and tutorial time. February was chosen as the month to hold the test, because it balanced the need to allow for meaningful knowledge acquisition by first-year fellows and the need to provide enough time for third-year fellows to address areas for improvement while still in training. There were 2 days on which the test was offered, and each trainee chose one day. Security during the examination was tight to ensure accuracy of results and protect the repository of questions for future use. Written examinations can include several types of questions, such as true or false, matching, fill in the blank, and even essay questions. The case-based clinical vignette, with a single best answer format, was chosen for this examination because it represents a reliable and efficient means of testing knowledge related to the real-life experiences of physicians taking care of patients.2,7 In this format, a patient care case or situation is initially presented to provide a clinical context for the question. This is followed by a single-sentence question, called the lead-in question. Finally, a set of answers are shown. These answers must be grammatically correct, avoid cueing (ie, enabling the examinee to find the answer by reviewing the options),8 show numerical choices in a consistent order, and be straightforward. Appendix 1 (online only) summarizes how to write a proper case-based question.9 Appendix 2 (online only) shows an example of how to write a question based on initial ideas. Other types of questions were not chosen for this examination because of the variety of problems they can have. For example, true or false questions would have prompted a simple recall of facts rather than complex thinking. They could also have been ambiguous. Matching questions could have enabled the so-called test-wise examinee to use the process of elimination rather than demonstrate clinical thinking. The examination blueprint is summarized in Table 1, including the domains (content areas) covered in the test and the percentage of questions devoted to each domain. The primary purpose of the examination was to act as an educational tool for the fellows and a program assessment tool for program directors. A secondary goal was to provide fellows with a practice test before the American Board of Internal Medicine (ABIM) Examination in Oncology. As such, the blueprint of the MedOnc ITE was similar to but not identical to the ABIM test. The MedOnc ITE did not duplicate the ABIM blueprint, because its goal as an educational event was to assess programs and give trainees a broad view of where they stood. The allocation of topics reflected an attempt to test trainees on a range of diseases, clinical settings, basic science, clinical research, and portions of the ACGME competencies. In parallel to the ABIM examination, the distribution of many questions reflected the frequency of the types of cancers that are seen in an oncology practice. The MedOnc ITE included questions testing for competencies other than knowledge. Some questions focused on less common cancers but were included because they tested trainees on treatment for a curable cancer. Other questions were included because they tested trainees on the key concept of "do no harm."
The questions were also assigned one or more task codes (Table 2). Each question therefore belonged to a domain area and was categorized as a question that tested trainees on evaluation, diagnosis, management, or prognosis. These task codes were used to analyze trainees' proficiency in the subareas of the content.
Residency and fellowship programs are required to provide outcome data on six competencies—medical knowledge, patient care, practice-based learning and improvement, systems-based practice, professionalism, and interpersonal and communication skills. However, providing outcome data to review committees and trainees is challenging.10 Historically, assessments of the competencies other than knowledge have been based on a limited number of relatively brief observations of trainees by faculty.11 Other methods, such as the multisource observation (ie, the 360-degree observation), in which multiple individuals (peers, nurses, pharmacists, and social workers) are asked to assess trainees, are increasingly being used.12 Comprehensive assessment of the competencies requires the use of many different methods, which can be time consuming and resource intensive. The ACGME provides methods of assessment on its Web site.1 Resource limitations require that program directors apply assessment methods as efficiently as possible. Consequently, methods that cover more than a single competency are desirable. Although most of the MedOnc ITE addressed medical knowledge, the committee wrote questions that were intended to focus on other competencies. Questions that address aspects of patient care other than medical knowledge are difficult to write.13 For example, one element of systems-based practice is the knowledge and understanding of medical practice and delivery systems in different areas of the country, and how they differ from one another, including methods of controlling health care costs and allocating resources. A question addressing the examinee's mastery of this competency may require the examinee to recognize that disparity in health care access can lead to delay in diagnosis. It is not possible to assess professional behaviors with an ITE. However, as with the other competencies, the core of knowledge and set of principles underlying medical professionalism can be assessed with standardized examinations.14 A question that addresses professionalism may deal with medical ethics, cultural disparities, or physician burnout, which can affect a physician's ability to maintain compassionate care. Future work on the MedOnc ITE could involve comparing (and/or supplementing) the results of this small subset of non-knowledge–based questions with other tools that measure these competencies.
It took more than 4 years to move the Med Onc ITE from the idea stage to the reporting of results to the first set of examinees. The steps, which could be applied to any program, are listed in Table 3. The idea to develop an ITE originated with oncology program directors and was presented to the Oncology Training Programs (OTP) Committee in 2004. After the concept was endorsed by the OTP, it was submitted to the ASCO Board of Directors, which approved funding for the ITE as a new ASCO initiative.
ASCO recognized that a partnership with an experienced testing organization would be needed to meet the goals of the examination. ASCO staff created a request for proposals that outlined the scope of work and areas of expertise needed from a vendor, which included: first, expertise and experience; second, good collaboration skills; third, efficiency; and fourth, affordability. The vendor had to have experience with the ABIM blueprint, the capability to do editorial and psychometric reviews of the examination, the capability to assemble the examination and post-examination reports, and writers to support the physician volunteers. The vendor had to have the capability to conduct a question-writing workshop. ASCO staff sought input from physicians on the ASCO OTP Committee in choosing a vendor. Five vendors were considered and delivered on-site presentations. The National Board of Medical Examiners (NBME) was considered highly, because the Association of Subspecialty Physicians and the Alliance for Academic Internal Medicine had informed ASCO of the NBME. Ultimately, the NBME was chosen to work with ASCO. An ad hoc ITE steering committee was convened, comprising OTP Committee members involved in the process and ASCO's representative to the Association of Specialty Professors, which simultaneously was promoting the development of ITEs to the internal medicine specialties. The steering committee developed the blueprint for the examination. A business plan was written, and pricing for the examination was determined. A registration process was created. For ASCO, this included creating a Web application to allow programs to register their fellows electronically. ASCO staff worked with training program sites to ensure that they had the appropriate information technology infrastructure to hold the examination at their sites (Appendix 2). These institutions were informed about the time commitment and fees for the examination and the need for proctors. In October 2006, a half-day question-writing workshop was provided by NBME staff at an airport hotel in Chicago, IL, chosen for easy access. Fifty-five physician volunteers were solicited from among program directors, the ASCO Cancer Education Committee, and the Community Practice Committee, and by a broad e-mail to ASCO members. Participants were chosen on a first-come, first-served basis. ASCO covered the cost of participation in the workshop. Those who were interested in being writers attended the workshop, where they received instruction on how to write questions of good quality. At the end of the workshop, participants interested in writing examination questions were asked to sign up to participate in a question-harvesting process; 34 of these participants submitted questions for the test. The Test Materials Development Committee (TMDC) was chosen from this pool of examination writers in February 2007. Because the blueprint had been chosen before the TMDC was developed, and it required a range of expertise, ASCO chose people for the TMDC on the basis of their expertise as well as their ability to write questions and attend meetings. The proportion of questions was partially modeled on the ABIM Oncology Certification Examination. Each TMDC member was asked to disclose any conflicts of interest and sign a confidentiality agreement with both the NBME and ASCO. Each member was assigned to write 25 new questions and review questions that had been submitted by other attendees of the first workshop. These questions were among the 300 questions reviewed by the TMDC during a 2-day meeting at NBME headquarters in Philadelphia, PA, in August 2007. Each question was reviewed and assigned to a primary domain area (Table 1) and one or multiple task areas (Table 2). Informally, the questions were also graded A, B, or C. A questions were of excellent quality, tested key concepts, and would likely be used for the first ITE. B questions had minor flaws or repeated concepts that had already been addressed by A questions. None of the flawed or repetitive B questions were used. Those B questions with minor flaws were saved to be potentially rewritten in future years, if committee members were able to rework the questions to meet high standards of quality and make them relevant to the year of the examination. C questions had major flaws and could not be used. This peer review ensured that all questions were scientifically sound and had educational value. The final examination was created by members of the TMDC in partnership with the NBME. A subset of the TMDC met in November 2007 to review a 200-question examination form assembled by NBME staff that met the examination blueprint. A 10-question practice test was also assembled. The assembled examination was then reviewed in an electronic (Web-based) format in early February. A process for growth and retention of the principles of an ITE is necessary. ASCO has recommended a flexible term for the TMDC. Committee members will have to have ABIM Oncology Board Certification and experience teaching in an oncology training program. This experience can include a position as a current or former program director or associate director, or as a faculty member with an interest in teaching. All committee members will have to be trained at a question-writing workshop. ASCO plans to have a question-writing workshop every other year to develop new examination writers. With this process, the pool of questions should grow by 200 questions each year. Questions that become outdated or inaccurate over time will be discarded. Poorly performing questions will be dropped. The best written questions addressing critical topics will be retained and used in subsequent years. Peer review before and during the formal annual meeting by members of the TMDC will continue.
ASCO was responsible for selecting test sites and informing the NBME of the locations for each test site, providing all appropriate personnel with information concerning computer-related requirements necessary for Web-based delivery of the examination, and arranging for test sites to have the necessary computer equipment for Web-based delivery. The NBME provided ASCO with the technical requirements (Table A115, online only) for Web-based delivery several months before the examination date. ASCO worked with the NBME to develop and deliver a Web-based survey to all clinical oncology fellowship sites in the United States and Canada to determine whether each site met the minimum technical requirements. The NBME analyzed the results of the survey and provided ASCO with a list of sites that required follow-up. ASCO contacted each site that had not met the technical requirements; NBME technical staff worked directly with sites, as needed, to work through technical issues before administration of the examination. Approximately 6 weeks before administration of the examination, the NBME provided ASCO with proctor login information and a link to the workstation certification tool. ASCO forwarded the information to the chief proctor at each test center. Each proctor was asked to run the workstation certification on each computer to check hardware and software configurations and detect potential problems in the launch or display of the examination. This tool also downloaded the secure browser required for administration of the Web-based examination in a secure, locked-down environment.
The first examination was administered on 2 days in February 2008. The 2-day period was chosen to facilitate clinical coverage at the programs and help provide the necessary security for the test. Eleven examinees were not able to participate in February, so they took the examination in March 2008. The number of participating programs was 139, including 22 medical oncology programs (15 in the United States and seven in Canada) and 117 combined hematology and oncology programs. There were 1,093 examinees, of whom 318 were in their first year, 399 in their second year, and 376 in their third year. There were only a few examinees beyond their third year, and they were assessed with the third-year examinees. Not all fellows in training took the examination. There are several reasons why the number of fellows in training did not equal the number of examinees. First, participation was voluntary. Some programs decided that first-year fellows may have had too little experience to take this test 7 months into their 3-year fellowship program. Informal feedback has indicated that expense did decrease the number of examinees from some programs. We do not have data on programs that chose not to sign up because of technical issues, but we do have data indicating that several programs had technical concerns before the test took place, and all of these issues were resolved by examination time. Finally, some examinees dropped out because of health or maternity or paternity issues. There were no other ITEs for hematology and oncology fellows when this examination was administered, but beginning in 2009, there will be an ITE for hematology. The TMDC discussed the implications of two tests at its meeting in summer 2008, but a real analysis of the examinations will not be possible until after the first ITE for hematology is administered and analyzed. Standard statistical methods were used to evaluate the performance of the questions in order to eliminate questions that did not perform well and identify those that were useful. A question was selected for review if fewer than 30% of examinees answered it correctly, there was inconsistency in the results between the strong test takers and the weak test takers, the wrong answer was chosen more often than the keyed-in correct answer, the strong test takers chose a different answer than the keyed-in correct answer, or an alternative answer was frequently chosen. The analysis was based on the responses of 396 second-year fellows who took the examination under standard conditions. Twenty-three questions were identified. ASCO content experts and NBME staff reviewed these questions during a conference call, and as a result, four questions were deleted from scoring, and one question was rekeyed. Thus, 196 of 200 questions on the examination were scored. Feedback in the form of a written explanation of the examination was sent to program directors. Most program directors met individually with trainees to discuss their test results. Long before the examination was administered, it was decided that the scores of the various programs would not be made public. Security of scores was a high priority. The written results included a summary of the performance of the fellows in all programs on the entire examination as well as on certain areas of content and tasks. A histogram showed the distribution of mean scores for all programs, and the mean score of each particular program. For each examinee, the results show a total score and a score within each content area by year of training, including the SE of that score. Examinees could see where they stood relative to other examinees by year and topic. The total score was based on a reference group of all second-year fellows, and had a mean of 500 and a standard deviation of 100. The mean score for the total group was 494, with a standard deviation of 114 and a range from 100 to 800. Content area scores were calculated and scaled so that the base reference group had a mean of 70 and a standard deviation of 8. The base reference group score will serve as a benchmark for future examinations. The SE of measurement (SEM) is a statistic that provides information about the degree of uncertainty in the scores. In general, the more questions on which a score is based, the smaller the SEM.2 The average SEM for this year's examination was 3%, which is equivalent to 41 scaled score units. This means that if an examinee's true proficiency were 500, his or her examination score would likely fall between 459 and 541 (500 – 41 and 500 + 41). The feedback did not include results of individual test questions or provide topics from individual questions because this examination was not meant to have programs or trainees study detailed topics; instead, it was meant to provide an overall assessment of the competence of programs and trainees to date. At the summer 2008 meeting, the results of the 2008 test were discussed; the ITE was shown to have performed statistically well, with a trend of improvement by year of fellowship. The TMDC discussed whether results of individual topics or questions should be sent to examinees but decided this would detract from the purpose of the test, tempting programs to teach to this test or trainees to study to the test. An optional post-examination survey was provided, to which 86% of examinees responded. Seventy-nine percent said that the examination contained the appropriate number of questions. Seventy-eight percent said that the questions were written clearly, and 6.6% said that they were not. Twenty-eight percent said that breast cancer was overemphasized, and 7.8% thought gynecologic oncology was overemphasized. Ten percent thought more questions were needed on gastrointestinal malignancies. Seventy-one percent thought that the examination was moderately difficult. Most examinees took this test fresh, with 53.5% responding that they did not study for the examination, and 17.3% responding that they studied for fewer than 10 hours. Overall, 70.7% had a very positive or positive experience with the examination, and only 1.6% had a negative experience. A variety of comments was also obtained. In a 2008 ASCO survey of training program directors conducted before the examination results were disseminated, 78% of the program directors who responded (N = 102) placed a high value on the ITE. In an informal survey posted on ASCO's program director Listserv, several program directors said they appreciated the examination, and they were using the results to redirect program activities. For example, a number of programs saw poor examination results in gynecologic oncology. These programs plan to improve exposure and didactic teaching in this area. Another program plans to improve its teaching in ethics. All of the program directors who responded on the Listserv stated that they were giving the results to trainees individually at their routine feedback meetings. On the basis of feedback from examinees and programs, in addition to improving the variety of questions and adding more images, an optional break time will be added between sessions. For Canadian fellows, lab values and norms will be presented according to the International System of Units. The majority of test development committee members should commit to at least a 2-year term to improve continuity. Twenty-eight questions were categorized as legal/ethical/other issues. Within this category were questions that addressed competencies other than medical knowledge. We plan to further analyze these questions, and report the results in a separate article.
ASCO developed the MedOnc ITE in response to a perceived need by program directors for an ITE similar to that used for internal medicine residents to have an objective tool to measure learning, assist with interprogram and intraprogram development, and provide feedback to trainees regarding their mastery of the knowledge base essential to clinical oncology. The examination may be a useful tool for assessing competencies other than medical knowledge. The process—from the inception of the idea for the MedOnc ITE to delivery of the actual results—was complex and took approximately 4 years. It required ASCO to coordinate with program directors, the NBME, physician question writers, and the actual programs at test time. The test used case-based clinical vignette questions, which are more difficult to write but more compatible with real-life work experiences than are other formats. The examination needed to be administered in a secure fashion to protect the accuracy of the results and enable some of the questions to be used in the future. February was chosen as the month to hold the examination, because it was early enough to return results for a meaningful impact for third-year fellows, and late enough to gauge what first-year fellows had learned. Feedback after the test was favorable from trainees and program directors. Most trainees thought it was a useful exercise. The program directors are using the results to improve their programs. ASCO is using the results and feedback to improve future examinations. Assessing physical examination skills or communication skills by extending this examination to include simulated exercises or modules will probably not occur until year 4 to 5 of development, when the current question content will have been optimally developed. Members of the ITE committee will look at the examination as a way to identify broader learning needs across all programs and for all trainees.
The author(s) indicated no potential conflicts of interest.
Conception and design: Frances A. Collichio, Kenneth R. Hande, James A. Stewart Administrative support: Frances A. Collichio Provision of study materials or patients: Kenneth R. Hande, David J. Adelstein Collection and assembly of data: Frances A. Collichio, Karen M. Kayoumi, David J. Adelstein, James A. Stewart Data analysis and interpretation: Frances A. Collichio, Kenneth R. Hande, James A. Stewart Manuscript writing: Frances A. Collichio, Karen M. Kayoumi, Richard E. Hawkins, Janine L. Hawley, David J. Adelstein, Jean M. D'Angelo, James A. Stewart Final approval of manuscript: Frances A. Collichio, Kenneth R. Hande, Richard E. Hawkins, Janine L. Hawley, Jean M. D'Angelo, James A. Stewart
Consolidated Instructions for Single Best Answer Questions The single best answer format is composed of three parts: a stem, a lead-in question, and answer options. The stem, which may be one or more paragraphs long, contains a clinical vignette describing a patient's age and sex, site of care, brief past history (family history and social history, when appropriate), brief present history (including symptoms), and important physical findings. It should not be unnecessarily complicated or tricky. If laboratory data need to be included, they should be presented after the description. There should be enough information in the stem to enable the examinee to answer the question; no additional data should be provided in the answer options. An incorrect stem will force the examinee to look at the answers to learn what should have been covered in the stem. The lead-in question should be clear and simple. Lead-in questions with words such as "is associated with" or "is useful for" are not clear. The question should be relevant to the topic. Questions that purposely try to lead the examinee astray do not test medical knowledge. Lead-in questions that provide the examinee with a hint to the answer, a process called cueing, should be avoided. Terms such as "could be," "usually," or "frequently" should be avoided. Vague terms indicating frequency, such as "usually" and "often," should not be used, because they may have different meanings to different people. Good examples of lead-in questions are:
The answers should be absolutely true or false. The test-wise examinee will use several different strategies to try to determine answers, even if he or she does not know them. Grammatical errors lead the examinee to eliminate the grammatically incorrect answer. The use of absolute terms, such as "always" and "never," leads the examinee to eliminate the answers that include these terms. Often, the correct answer is the longest answer, and the examinee chooses it because of its length. A word repeat occurs when the question contains a word that is also in the answer. This cues the examinee to make the correct choice. Finally, there is the convergence strategy, used when the correct answer includes the most elements in common with the other options. Good answers will sort numeric data consistently from lowest to highest or highest to lowest. When this is done inconsistently, the examinee spends more time ordering the numbers, rather than demonstrating his or her knowledge base. In this example, the answer options are flawed, because there are problems with the range, and the order is confusing. In the case above, the chance of recurrence is?
Good answers will not include "none of the above" as an option, because this may require that the examinee use judgment. Which pain medication will be most effective?
Properly written questions can stand alone; they do not require other questions to precede them. When a question requires another question to precede it, the question becomes hinged. The test-wise examinee will find patterns in both questions that help him or her to find the answer, irrespective of his or her knowledge base. In addition, the examiner decreases the library of questions that can be used in the future, because one question relies on the other.
Examples of Questions Start with a clinical idea: There is a patient with a mediastinal mass and low sodium. Build the question: A man 66 years of age comes for evaluation because of a 2-week history of a nonproductive cough and 10-lb weight loss. He has smoked one pack of cigarettes per day for the past 50 years and has a history of mild hypertension. Physical examination shows a thin patient in no acute distress. Auscultation of the chest discloses no abnormalities. A radiograph of the chest shows a mediastinal mass. A positron emission tomography–computed tomography (PET-CT) scan shows positive findings in the mediastinum and both axillae. Magnetic resonance imaging (MRI) of the brain shows no abnormalities. Laboratory studies show the following findings: Serum: Sodium, 118 mmol/L Potassium, 4.1 mmol/L Chloride, 106 mmol/L CO2, 24 mmol/L Creatinine, 0.9 mg/dL A biopsy of the lung mass is most likely to show findings consistent with which of the following conditions?
The answer is B. In the example, the stem provides just the right amount of clinical information, and the question is clear. The answers are all of the same length. Increase the level of difficulty: An oncology fellow is expected to know that syndrome of inappropriate antidiuretic hormone is associated with small-cell lung cancer, so it may be more appropriate to increase the complexity of the question and test treatment. A man 66 years of age comes for evaluation because of a 2-week history of a nonproductive cough and 10-lb weight loss. He has smoked one pack of cigarettes per day for the past 50 years and has a history of mild hypertension. Physical examination shows a thin patient in no acute distress. Auscultation of the chest discloses no abnormalities. A radiograph of the chest shows a mediastinal mass. A PET-CT scan shows positive findings in the mediastinum and both axillae. An MRI of the brain shows no abnormalities. Laboratory studies show the following findings: Serum: Sodium, 118 mmol/L Potassium, 4.1 mmol/L Chloride, 106 mmol/L CO2, 24 mmol/L Creatinine, 0.9 mg/dL Etoposide and cisplatin chemotherapy is administered for six cycles. A follow-up PET scan and MRI of the brain obtained 4 weeks after the last chemotherapy dose show no abnormalities. The patient says that he feels well and has returned to work. Which of the following is the most appropriate next step in management?
The answer is E. The examinee is expected to know that one of the advances in 2007 was prophylactic whole-brain radiation therapy for patients with extensive-stage small-cell lung cancer who are in complete remission after chemotherapy. Develop a systems-based practice question with the same case: This question would not be hinged to the preceding examples (and likely would not show up in the same examination year, because the diagnosis is given in the stem). A man 66 years of age comes for evaluation because of a 2-week history of a nonproductive cough and 10-lb weight loss. He has smoked one pack of cigarettes per day for the past 50 years and has a history of mild hypertension. Physical examination shows a thin patient in no acute distress. Auscultation of the chest discloses no abnormalities. A radiograph of the chest shows a mediastinal mass. A PET-CT scan shows positive findings in the mediastinum and both axillae. An MRI of the brain shows no abnormalities. The patient is referred to a cancer center 2 hours from his home. A biopsy of the mediastinal mass obtained at the center shows findings consistent with small-cell lung cancer. Chemotherapy with intravenous cisplatin and etoposide on day 1 and intravenous etoposide on days 2 and 3 is initiated. The patient misses his therapy appointments on days 2 and 3, but returns to the clinic for his second cycle. He says that he was unable to make the appointments because he did not have money for gasoline, and he is currently feeling well. The patient's hospitalization and medications are covered by his insurance plan. Which of the following is the most appropriate next step?
The answer is D. The examinee must recognize that of the choices, the cisplatin and etoposide regimen is the best treatment for this type of cancer. Administration of etoposide orally may be less effective than intravenously, but this would at least get around the patient's problem of transportation. It is inappropriate to admit a patient to the hospital when the chemotherapy can be given in outpatient setting. Create another systems-based practice question: Use the same case, but again, do not let it hinge on the preceding examples. Have it be a standalone question that could be used in another examination year. A man 66 years of age comes for evaluation because of a 2-week history of a nonproductive cough and 10-lb weight loss. He has smoked one pack of cigarettes per day for the past 50 years and has a history of mild hypertension. Physical examination shows a thin patient in no acute distress. Auscultation of the chest discloses no abnormalities. A radiograph of the chest shows a mediastinal mass. A PET-CT scan shows positive findings in the mediastinum and both axillae. An MRI of the brain shows no abnormalities. The patient is referred to the closest cancer center, which is 2 hours from his home. A biopsy of the mediastinal mass obtained at the center shows findings consistent with small-cell lung cancer. Treatment will involve chemotherapy administered every 3 weeks. As the treating oncologist, you are concerned about handling side effects for a patient who lives so far away. Which of the following is the most appropriate recommendation for this patient?
The answer is C. This question tests the examinee's ability to work with the referring physician and recognize that practices are different in different parts of the country.
We acknowledge ASCO for its support of this project.
Disclaimer: R.E.H., J.L.H., and J.M.D. are employed by the National Board of Medical Examiners and received remuneration for this project. Authors' disclosures of potential conflicts of interest and author contributions are found at the end of this article.
1. Accreditation Council for Graduate Medical Education. Outcome Project. http://www.acgme.org/outcome/comp/refList.asp. 2. Hawkins RE, Swanson DB. Using written examinations to assess medical knowledge and its application. ES Holmboe, and RE Hawkins. Practical Guide to the Evaluation of Clinical Competence. Philadelphia, PA: Elsevier Health Sciences, 2008. p.42–59. 3. American College of Physicians. Internal Medicine In-Training Exam. http://www.acponline.org/education_recertification/education/in_training/. 4. Garibaldi RA, Trontell MC, Waxman H, et al: The in-training examination in internal medicine. Ann Intern Med 121:117–123, 1994. 5. Garibaldi RA, Subhiyah R, Moore ME, et al: The in-training examination in internal medicine: An analysis of resident performance over time. Ann Intern Med 137:505–510, 2002. 6. Batalden P, Leach D, Swing S, et al: General competencies and accreditation in graduate medical education. Health Aff (Millwood) 21:103–111, 2002. 7. Schuwirth LW, Verheggen MM, van der Vleuten CP, et al: Do short cases elicit different thinking processes than factual knowledge questions do? Med Educ 35:348–356, 2001.[CrossRef][Medline] 8. Schuwirth LW, van der Vleuten CP, Donkers HH: A closer look at cueing effects in multiple-choice questions. Med Educ 30:44–49, 1996.[Medline] 9. Case SM, Swanson DB. Constructing Written Test Questions for the Basic and Clinical Sciences, ed 3 Philadelphia, PA: National Board of Medical Examiners, 2002. 10. Quillen DM: Challenges and pitfalls of developing and applying a competency-based curriculum. Fam Med 33:652–654, 2001.[Medline] 11. Epstein RM: Assessment in medical education. N Engl J Med 356:387–396, 2007. 12. Musick DW, McDowell SM, Clark N, et al: Pilot study of a 360-degree assessment instrument for physical medicine and rehabilitation residency programs. Am J Phys Med Rehabil 82:394–402, 2003.[CrossRef][Medline] 13. Frederiksen N: The real test bias: Influences of testing on teaching and learning. Am Psychol 39:193–202, 1984.[CrossRef] 14. Cruess RL, Cruess SR: Teaching professionalism: General principles. Med Teach 28:205–208, 2006.[CrossRef][Medline] 15. National Board of Medical Examiners. Customized Assessment Services: Information Guide—Creating and Administering a Customized Web-Based Examination. http://www.nbme.org/PDF/CustomizedAssessmentServices.pdf. Submitted October 8, 2008; accepted December 16, 2008.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|||||||||||
|
|||||||||||
|
|
Copyright © 2009 by the American Society of Clinical Oncology, Online ISSN: 1527-7755. Print ISSN: 0732-183X
|
TOP
ABSTRACT
INTRODUCTION
