Randall Sargent, MSc, MD1,2,3, Cynthia Brocklebank, ACPR, PharmD4, Helen Tam-Tham, MSc5, Tyler Williamson, PhD5, Patrick Quail, MB, BCh2,3, Diana Turner, MSc, MD2,6, Neil Drummond, PhD7
1Family Medicine and Primary Care Research Office, Cumming School of Medicine, University of Calgary, Calgary;
2Integrated Supportive and Facility Living, Alberta Health Services, Calgary;
3Southern Alberta Primary Care Research Network, University of Calgary, Calgary;
4Pharmacy Services, Alberta Health Services, Edmonton;
5Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary;
6Carewest, Calgary;
7Department of Family Medicine and Dentistry, University of Alberta, Edmonton, AlbertaDOI: http://dx.doi.org/10.5770/cgj.19.205
ABSTRACT
Background
Warfarin is an anticoagulant prescribed to 12% of long-term care residents to reduce the risk of thrombo-embolism. This study used indicators to compare warfarin management by pharmacists to usual care.
Methods
This was a retrospective cohort study comparing a pharmacist-managed warfarin protocol with usual care of qualified warfarin recipients at long-term care facilities (two protocol, one control) in Calgary, Alberta. We compared the proportion of international normalized ratio (INR) tests in the range 2.0 to 3.0, time in range, number of tests, and frequency of bleeding at protocol and control sites. Our primary outcome, time in INR therapeutic range, is an indicator for assuring care quality. A cross-sectional survey at these sites compared health professionals’ perceptions of workload and effectiveness of warfarin management.
Results
Of the 197 residents’ charts reviewed in the study period, those on protocol had 45.0 INR tests while those on usual care had 52.7 tests (p = .034, 95% CI for the difference: 0.6 to 14.6 INR tests). No significant difference was found for time in therapeutic range, number of tests in range, or major bleeding events. Of 178 health professionals surveyed, those from protocol facilities were more satisfied with warfarin management (p = .013). Workload and safety were perceived similarly at all sites.
Interpretation
Our results suggest that a pharmacist-managed warfarin protocol is as effective as usual care and has advantages pertaining to work satisfaction, knowledge of drug interactions, consistent documentation, and fewer INR tests. Further research on teamwork and coagulation management in long-term care facilities is recommended.
Key words: continuing care, pharmacy, warfarin, anticoagulation, international normalized ratio (INR), protocol
In long-term care, the scope of practice for pharmacists may involve warfarin dose management. A warfarin dosing protocol can be a useful tool for consistent antithrombotic care that allows the physician and pharmacist to work together with confidence. The American College of Chest Physicians Clinical Practice Guideline recommends antithrombotic therapy for atrial fibrillation, among other indications such as pulmonary embolus, and supports the use of a protocol.(1) Warfarin is one management choice for atrial fibrillation, and atrial fibrillation is the most common antithrombotic indication at long-term care sites in this study.
Motycka et al.(2) reported on benefits of pharmacist management, which has been shown in hospital settings to improve warfarin therapy,(3,4) reduce health care cost,(5) and improve quality of patient care.(3,6) Use of a warfarin dosing protocol by pharmacists and the effect on quality of long-term care, however, has not been studied. In Alberta, Canada, pharmacists have been able to prescribe warfarin since 2007. This study compared the effectiveness and safety (days within INR range 2.0–3.0) of pharmacist-managed warfarin dosing using a printed nomogram to that of conventional physician-managed dosing in long-term care. We also compared long-term care health professionals’ perceptions of warfarin management. From this work we have drawn conclusions regarding quality indicators, effectiveness of a protocol, and choices in thrombus risk management in long-term care. We undertook this study to contribute evidence in support of increasing scopes of practice of non-physician professionals in long-term care settings.
Alberta is a good choice of study location since it has 14,500 long-term care beds with two administrative models: public and private-for-profit, but a single-entry model of patient access to the 160 care sites.(7) The Alberta population includes 11.1% over 65 years (Canada is 14.8% overall).(8) Provincial funding for each long-term care bed is based on professional care hours per week.
This study was conducted at three comparable long-term care sites in Calgary, Alberta (Table 1). Two of the sites (comprising 1,059 beds) had implemented the Calgary Warfarin Protocol (“the Protocol”) in 2008 and the control site (629 beds) had not. The three long-term care sites operated under supervision by Alberta Health Services, had similar management regimes, and were comparable in terms of the composition of their nurse, physician, and pharmacist workforces (Table 1). All sites provide standard care for approximately 1,700 Alberta long-term care patients of 65 years and older, excluding patients in specialty care units, such as brain injury, who were not considered for participation in this study.
TABLE 1. Study characteristics, by long-term care site
At all sites, the pharmacist’s role included reviewing prescribed medications for risk of drug interaction and risk of bleeding (see Table 1). Registered nurses and licensed practical nurses shared responsibility for patient assessment and warfarin administration. Physicians were responsible for warfarin management as part of their patient care plan. Most of the physicians at each site cared for at least 10 patients, since community physicians tend not to follow the patient when admitted to long term-care. Physicians visited at least monthly and medications were reviewed quarterly by physician and pharmacist. Team roles and administration at each site remained stable throughout the study period of mid-2010 to late 2011.
The Calgary Warfarin Protocol was adapted from the validated protocol of Hirsch et al.(9) and from the Calgary Anticoagulation Management Service warfarin nomogram. Designed to support warfarin dosing decisions, it was introduced in 2008, shortly after Alberta pharmacists were granted prescribing privileges. Where the Protocol was adopted, pharmacists were trained by a managing pharmacist from Alberta Health Services Seniors Health Division. At protocol sites, once the INR is stable in the range 2.0 to 3.0 for at least two weeks, the patient’s INR report is sent to the delegated pharmacist, who consults the dosing algorithm (plus calculator) and determines the dose to be dispensed. Testing INR is at the discretion of the physician or, if ordered by a pharmacist, as indicated by the Protocol. When the INR is reported above 5.0, warfarin dosing is transferred to the physician, since such management is difficult to incorporate into a protocol; below an INR of 2.0 the physician may be consulted, but the protocol does address the scenario. Since our study was conducted in 2010 and 2011 and the Protocol was introduced in 2008, there was adequate time for familiarization of pharmacists with the Protocol.
For the purposes of this study, “safe warfarin management” means the INR is in the range 2.0 to 3.0 more than half the times tested.(10) A “major” drug interaction is characterized by a change in INR following introduction of a drug that requires alteration of the warfarin dose to avoid adverse effects (e.g., amiodarone).(11) “Major bleeding” is defined by the International Society of Thrombosis and Hemostasis (ISTH) as including fatal bleeding in non-surgical patients, symptomatic bleeding, bleeding causing a fall in hemoglobin level of 20 g/L, or need for transfusion.(12)
This two-part study was approved by the University of Calgary’s Conjoint Health Research Ethics Board. The first part (2010–2011) employed a retrospective cohort design comparing warfarin management at two study sites that had implemented the Calgary Warfarin Protocol for at least the past year, to a control site with no protocol in place. The second part of the study (2010–2011) was a cross-sectional survey comparing the long-term care professionals’ perceptions of ongoing warfarin management safety, satisfaction, and workload at their respective sites. In order to avoid confounding results, we ensured that none of the three sites had employed any type of warfarin protocol before the 2008 introduction date.
The charts of all patients prescribed warfarin were analyzed for study eligibility. When reviewing each chart, we collected data on medication management and risk of harm. Charts included in the study documented warfarin therapy of at least four weeks’ duration (suggesting stability) and INR 2.0 to 3.0 in a two-week or greater period from September 2008 to September 2010. Across the three sites, 197 patients were eligible (45 and 71 from population 446 and 620, respectively, at protocol sites and 81 from a population of 629 at the control site). A preliminary chart audit of the three long-term care facilities (1,900 patients) confirmed that 10% of patients in our study sites were receiving warfarin (n = 190), in line with the existing literature.(10) Verhovsek et al.(10) also found that patients are on INR target 54% of the time. Since a patient’s INR was tested 20 times, on average, over a two-year study period, a sample of 190 patient chart reviews provides in excess of 90% power to detect a difference of 10% in the on-target time, at the 5% level of significance.
Data abstracted from patients’ charts (INR values, lab testing, and bleeding risk) were recorded on standardized sheets by two trained assistants and reviewed for consistency by a third assistant and the lead investigator (RS). A single blinded reviewer (RS) audited participants’ chart data to identify incidents of bleeding and categorized them according to the ISTH definition. A separate category of “clinically relevant non-major bleeding”, defined by ISTH as requiring some form of medical intervention, was identified for future study.(13,14)
The cross-sectional survey of long-term care professionals’ perceptions of warfarin management safety, satisfaction, and workload at their respective sites was based on the NASA (National Aeronautics and Space Administration) Task Load Index Instrument,(15) which has been validated in estimating differences in workload.(16) It consisted of the NASA workload questions, the demographic questions, and the six survey Likert-type scale questions (see Appendix A). The expected time for completion was 5 minutes. For the survey, 147 staff at the two protocol sites and 107 staff at the control site were eligible. In keeping with our inclusion criteria, all were proficient in English and had worked at the site throughout the previous month and were actively involved in warfarin management.
Our primary outcome measure was the proportion of time (days in range/total observation days) within INR range 2.0 to 3.0. Secondary outcomes included incidence of major bleeding (including referral to emergency department), proportion of INR measures above 5.0, mean number of INR tests per patient at each site, proportion of INR measures in the 2.0 to 3.0 range, and the number of major drug interactions. The outcome of the health professionals’ survey was the distribution (Likert scale) of each profession’s perceptions of warfarin management safety, satisfaction, and workload at control and protocol sites.
In accordance with the method of Rosendaal et al.,(17) INR measurements were analyzed by the proportion of time in INR range 2.0 to 3.0 for each patient chart, and categorized by their status as protocol or control. The Rosendaal calculation is as follows:
Interpolate between adjacent INR values, daily INR increments result.
Calculate the number of days from first to last INR measured.
Estimate the number of days in INR range 2.0 to 3.0 for that individual.
Group the individual days in range for all site participants and calculate the mean.
Determine if the mean scores were different between Protocol and control groups.
The cross-sectional survey workload data were collected on continuous scales (NASA Task Load Index) and compared by two sample t-tests; skewed data was assessed with a Wilcoxon ranked-sum (Mann-Whitney) test. Worker perception data (Table 2) lists the five-point Likert-type scale survey questions, which were interpreted using a two sample test of proportion. Responses were analyzed for sensitivity by combining missing (no answered offered) and “neither” responses in ordinal data, then compared using chi-square tests. Five-point Likert-type scale results were combined (0=”very satisfied,” 1=”neither,” and 2=”dissatisfied” and “very dissatisfied”) at the two ends because of low end numbers. Stratified analysis was conducted based on profession type. Stata software (version 11.2) (StataCorp LP, College Station, TX) was utilized for all statistical analysis.
TABLE 2. Survey outcomes from staff working at Protocol and control long-term care sites
The demographic character of the three sites was comparable (see Table 1). Of the 197 (protocol 116 and control 81) patients, 135 (68.5% overall; 63.8% protocol vs. 75.3% control) were female, and 170 (86.5% overall; 87.1% protocol vs. 85.2% control) were alive at the time of chart review. The age of 188 patients (overall 95.4%; 98.3% protocol vs. 91.4% control) was 65 years or older. All patients’ charts documented anti-coagulation therapy for at least 28 days, with 135 (68.5% overall; 69.0% protocol vs. 67.9% control) having received it for more than one year. Atrial fibrillation was a diagnosis in 142 charts (72.0% overall; 73.3% protocol vs. 70.4% control), deep vein thrombosis in 28 charts (14.2% overall; 14.7% protocol vs. 13.6% control), and pulmonary embolism in 11 charts (5.6% overall; 6.0% protocol vs. 4.9% control).
The overall proportion of time INR was between 2.0 and 3.0 was not significantly different between protocol (71.3%) and control (66.2%) groups (p = .447, 95% CI of the difference: −18.3% to 8.1%). Overall, INR was within target range 69.2% of the time, regardless of protocol status. There was no significant difference in the number of INR test results within therapeutic range (protocol 63.9%; control 61.3%, p = .703, 95% CI of the difference: −16.4% to 11.1%). The proportion of INR values greater than 5.0 was not significantly different for the groups (0.7% in protocol; 1.8% in control, p = .486, 95% CI of the difference 2.2% to 4.3%). In total, 9476 INR tests were reported for 197 patients. On average, patients at protocol sites had 45.0 INR tests compared to 52.7 INR tests for those at the control site (p = .034, 95% CI for the difference: 0.6 to 14.6 INR tests) (See Table 3). This represents a significant difference in the number of INR tests.
TABLE 3. INR outcomes among protocol and control groups living in long-term care facilities (95% CI)
The majority (168, 85.3%) of the 197 charts documented at least one prescription known to interact with warfarin.(11) Of these, 100 (50.8%, 63 protocol and 37 control) were known to have “major” drug interaction with warfarin. (See Appendix B)
Five major bleeds (four protocol, one control) and seven clinically relevant non-major bleeds (five from the protocol group) were identified. Three bleeding events occurred when the INR was above target range, one event when it was below target range, and eight events when it was within range. One bleeding event (protocol) correlated with a known major interacting medication (sulfamethoxazole/trimethoprim).(11) There was no significant difference in the number of bleeding events at protocol and control sites (p = .112). All major bleeds were referred to the emergency department of acute care.
Of the 178 professionals surveyed, responses were received from 132 nurses (78% of registered and licensed practical nurses), 19 pharmacists (7 % response rate), and 27 physicians (43% response rate). This represented a 72% response rate from protocol sites (106 respondents) and 67% from control (72 respondents).
Dissatisfaction with warfarin management at the control site was significantly greater (0.0% at protocol vs. 5.9% at control dissatisfied or very dissatisfied, p < .001, see Table 2). No difference was found in the protocol and control groups’ perceptions of workload or warfarin management effectiveness.
We found no difference in the time within INR therapeutic range between protocol and control groups, and no statistical difference in frequency of bleeding (Table 3). These results indicate that pharmacist-managed warfarin dosing using the Protocol is as effective and safe as physician management, since amount of time in INR range 2.0 to 3.0 is equivalent and more than half the INR measures fall in that range.
Pharmacist-managed dosing with the Protocol required fewer INR tests, likely because the Protocol directs INR ordering, suggesting potential savings in staff time and laboratory utilization. Other advantages include the availability of INR records, dose records, and the focus on pharmacist decision-making instead of the other care team professionals. Health-care professionals at protocol sites were more satisfied with warfarin management than those at the control site, though there were no differences in perceptions of safety and workload. We suspect that nurses and physicians experienced reduced workload with management by protocol but did not report it as a significant factor. Further study would require validation of the new survey questions. Overall, these findings are consistent with those of other published accounts,(2,4) and indicate long-term care teams’ acceptance of the warfarin management by pharmacists.
We have identified no other study in which pharmacists were trained to use a warfarin protocol in long-term care settings. Motycka et al.(2) reported that pharmacists’ warfarin management is more often in the therapeutic range than usual practice (we found equal time in range). Other studies reported that pharmacists improve warfarin therapy,(3,4) reduce health-care costs,(5) and improve the quality of patient care.(6) Our finding of fewer lab tests associated with use of the Protocol demonstrates it is effective warfarin management with systematic laboratory testing. While Gurwitz et al.(18) stated that pharmacists prevent adverse drug events in nursing homes, major interacting drugs (Appendix B) were identified in both study groups, so it is difficult to know the effect on adverse events. Major bleeding numbers were low in both groups and thus not statistically significantly different. Further study of this issue specifically will require a larger study population.
Neidecker et al.,(19) acknowledging risk of adverse events with warfarin therapy, stated that the drug’s low cost, long-standing use, and reversibility of effect help mitigate risk to the patient. We suggest that site-dedicated pharmacist management with the Protocol also optimizes warfarin safety through services provided to the team. In Alberta, where pharmacists can now prescribe warfarin without requiring a dosing protocol, further study of protocol value is indicated. That a physician can manage warfarin dosing without a protocol attests to the ongoing value of individual patient assessment.
This study’s conclusions are applicable to long-term care populations. Combining Rosendaal calculations for individuals within each site and within patient groups, such as those diagnosed with atrial fibrillation, reduced the power of our conclusions since the Rosendaal calculation estimates the INR value between individual INR tests, and variation is individualized, not group-dependent.(17) A prospective study of individual coagulation management could further clarify the role of the Protocol. Variation in chart information and access to acute care records limited the information available for categorizing bleeding events. Resource constraints confined bleeding categorization to a single blinded reviewer. The overall number of bleeding events was small, which reflects the size of the study population and limits the generalizability of our conclusions. Since the impact of new antithrombotic agents was not within the scope of this study, we await research comparing their characteristics to warfarin.(20) There is also need for research into bleeding issues affected by renal insufficiency and haematologic function that may be particular to people over 65 years.
This study concludes that delegating warfarin management to pharmacists using the Calgary Warfarin Protocol and a printed nomogram is as effective and safe as management by physicians in long-term care. It frees physicians and nurses to focus on other aspects of care, and makes use of a pharmacist’s knowledge of drug interactions and dose monitoring while offering opportunities for quality assurance activities. Given the pace of change in health care, it is advantageous to have measurement tools and guidelines such as protocols. Studies such as this suggest indicators of quality and measures of team function that can assure safe care for patients. Prospective research focused on individual bleeding risk is a natural follow-up to this study, and direct comparison of antithrombotic management where other health professionals have access to the Protocol would be welcome.
We gratefully acknowledge funding for this study from the Southern Alberta Primary Care Research Network (SAPCReN), Department of Family Medicine, University of Calgary. We acknowledge Meghan Doherty’s assistance in formulating the proposal for this study, and Jill Watson, PhD, for editorial contributions to the completion of this manuscript.
The authors declare that no conflicts of interest exist.
1. You JJ, Singer DE, Howard PA, et al. Antithrombotic therapy for atrial fibrillation: antithrombotic therapy and prevention of thrombosis, 9th Ed: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines. CHEST J. 2012;141(2 Suppl):e531S –e575S.
2. Motycka C, Kesgen C, Smith SM, et al. Potential benefits of warfarin monitoring by a clinical pharmacist in a long term care facility. J Thromb Thrombolysis. 2012;33(2):173–77.
3. Dawson NL, Porter IE 2nd, Klipa D, et al. Inpatient warfarin management: pharmacist management using a detailed dosing protocol. J Thromb Thrombolysis. 2012;33(2):178–84.
4. Donovan JL, Drake JA, Whittaker P, et al. Pharmacy-managed anticoagulation: assessment of in-hospital efficacy and evaluation of financial impact and community acceptance. J Thromb Thrombolysis. 2006;22(1):23–30.
5. Hall D, Buchanan J, Helms B, et al. Health care expenditures and therapeutic outcomes of a pharmacist-managed anticoagulation service versus usual medical care. Pharmacotherapy. 2011;31(7):686–94.
6. Verret L, Couturier J, Rozon A, et al. Impact of a pharmacist-led warfarin self-management program on quality of life and anticoagulation control: a randomized trial. Pharmacotherapy. 2012;32(10):871–79.
7. Health Quality Council of Alberta. Long term care family experience survey report. provincial results [Internet]. Calgary, AB: HQC of Alberta; 2015. Accessed 1 Nov 2015. Available from: https://d10k7k7mywg42z.cloudfront.net/assets/562fabf3d4c9610610181c2c/HQCA_LTC_Provincial_Report_2015.pdf
8. Statistics Canada. Alberta (Code 48) and Canada (Code 01) (table). Census Profile. 2011 Census. Statistics Canada Catalogue no. 98-316-XWE [Internet]. 2012. Accessed 1 Nov 2015. Available from: http://www12.statcan.gc.ca/census-recensement/2011/dp-pd/prof/details/Page.cfm?Lang=E&Geo1=PR&Code1=48&Geo2=PR&Code2=01&Data=Count&SearchText=Alberta&SearchType=Begins&SearchPR=01&B1=All&GeoLevel=PR&GeoCode=48
9. Hirsh J, Fuster V, Ansell J, Halperin JL. American Heart Association/American College of Cardiology Foundation guide to warfarin therapy. J Am Coll Cardiol. 2003;41(9):1633–52.
10. Verhovsek M, Motlagh B, Crowther MA, et al. Quality of anticoagulation and use of warfarin-interacting medications in long-term care: a chart review. BMC Geriatr. [Internet]. 2008;8(13). Accessed 17 June 2010. Available from: http://www.ncbi.nlm.nih.gov/pubmed/18598364
11. Bungard T, Yakiwchuk E, Foisy M, et al. Drug interactions involving warfarin: practice tool and practical management tips. Can Pharm J. 2011;144(1):21–25.
12. Schulman S, Kearon C. Definition of major bleeding in clinical investigations of antihemostatic medicinal products in non-surgical patients. J Thromb Haemost. 2005;3(4):692–94.
13. van Gogh Investigators, Buller HR, Cohen AT, et al. Idraparinux versus standard therapy for venous thromboembolic disease. N Engl J Med. 2007;357(11):1094–104.
14. Granger CB, Alexander JH, McMurray JJV, et al. Apixaban versus warfarin in patients with atrial fibrillation. N Engl J Med. 2011;365(11):981–92.
15. Hart S, Stavenland L. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research, Chapter 7. In: Meshkati N, Hancock PA, editors. Human Mental Workload [Internet]. Elsevier; 1988. Accessed June 17 2010. Available from: http://humansystems.arc.nasa.gov/groups/tlx/downloads/NASA-TLXChapter.pdf
16. France DJ, Levin S, Hemphill R, et al. Emergency physicians’ behaviors and workload in the presence of an electronic white-board. Int J Med Inf. 2005;74(10):827–37.
17. Rosendaal FR, Cannegieter SC, van der Meer FJ, et al. A method to determine the optimal intensity of oral anticoagulant therapy. Thromb Haemost. 1993;69(3):236–39.
18. Gurwitz JH, Field TS, Avorn J, et al. Incidence and preventability of adverse drug events in nursing homes. Am J Med. 2000;109(2):87–94.
19. Neidecker M, Patel AA, Nelson WW, et al. Use of warfarin in long-term care: a systematic review [Internet]. BMC Geriatr. 2012;12(14). Available from: http://bmcgeriatr.biomedcentral.com/articles/10.1186/1471-2318-12-14
20. Gross PL, Weitz JI. Venous thromboembolism: mechanisms, treatment and public awareness. New anticoagulants for treatment of venous thromboembolism. Arterioscler Thromb Vasc Biol. 2008;28(3):380–86.
Please complete the following questions in relation to warfarin therapy to the best of your knowledge. Your responses will be kept anonymous. Your participation in this survey is greatly appreciated.
Have you worked at this institution for more than 1 month? (choose one)
□ Yes
□ No
What is your profession? (choose one)
□ Pharmacist
□ Registered Nurse
□ Licensed Practical Nurse
□ Physician
How safe are the patients on your unit from overtreatment (bleeding events e.g. gastrointestinal or intracranial) or undertreatment (e.g. embolic or thromobotic stroke) with warfarin? (choose one)
□ Very safe
□ Safe
□ Neither safe nor unsafe
□ Unsafe
□ Very unsafe
Defining workload as: “Professional time spent preparing for and discussing results and orders with other colleagues,” overall, how would you rate your workload? (choose one)
□ Very high
□ High
□ Neither high nor low
□ Low
□ Very low
How satisfied are you with the management of warfarin dosing (INR control) in your facility? (choose one)
□ Very satisfied
□ Satisfied
□ Neither satisfied nor dissatisfied
□ Dissatisfied
□ Very dissatisfied
Have you ever used a pharmacist-managed warfarin protocol in INR control? (choose one)
□ No
□ Yes (Please complete questions 7 and 8)
Consider this statement: “A pharmacist-managed warfarin protocol is an asset in INR control.” (choose one)
□ Strongly Agree
□ Agree
□ Neither Agree nor Disagree
□ Disagree
□ Strongly Disagree
Consider this statement: “A pharmacist-managed warfarin protocol is the best method for INR control in most patients.” (choose one)
□ Strongly Agree
□ Agree
□ Neither Agree nor Disagree
□ Disagree
□ Strongly Disagree
The scales below describe six key aspects of workload. Please indicate with an “X” on each scale the point that corresponds to your perception of your workload in relation to warfarin therapy and maintaining target INR (2–3) values (referred to as the “task” on each scale).
For definitions of any the following categories please refer to the table on page 3.
How mentally demanding was the task?
How physically demanding was the task?
How hurried or rushed was the pace of the task?
How successful were you in accomplishing what you were asked to do?
How hard did you have to work to accomplish your level of performance?
How insecure, discouraged, irritated, stressed, and annoyed were you?
NASA-TLX Rating Scale Definitions
Please use the space below to write additional comments:
____________________________________________________________________________________________________
____________________________________________________________________________________________________
____________________________________________________________________________________________________
____________________________________________________________________________________________________
____________________________________________________________________________________________________
____________________________________________________________________________________________________
____________________________________________________________________________________________________
Thank you for taking part in this study
Canadian Geriatrics Journal, Vol. 19, No. 2, June 2016