| Title |
Gestalt assessment of online educational resources may not be sufficiently reliable and consistent
|
|---|---|
| Published in |
Tijdschrift voor Medisch Onderwijs, February 2017
|
| DOI | 10.1007/s40037-017-0343-3 |
| Pubmed ID | |
| Authors |
Keeth Krishnan, Brent Thoma, N. Seth Trueger, Michelle Lin, Teresa M. Chan |
| Abstract |
Online open educational resources are increasingly used in medical education, particularly blogs and podcasts. However, it is unclear whether these resources can be adequately appraised by end-users. Our goal was to determine whether gestalt-based recommendations are sufficient for emergency medicine trainees and attending physicians to reliably recommend online educational resources to others. Raters (33 trainees and 21 attendings in emergency medicine from North America) were asked to rate 40 blog posts according to whether, based on their gestalt, they would recommend the resource to (1) a trainee or (2) an attending physician. The ratings' reliability was assessed using intraclass correlation coefficients (ICC). Associations between groups' mean scores were assessed using Pearson's r. A repeated measures analysis of variance (RM-ANOVA) was completed to determine the effect of the level of training on gestalt recommendation scale (i. e. trainee vs. attending). Trainees demonstrated poor reliability when recommending resources for other trainees (ICC = 0.21, 95% CI 0.13-0.39) and attendings (ICC = 0.16, 95% CI = 0.09-0.30). Similarly, attendings had poor reliability when recommending resources for trainees (ICC = 0.27, 95% CI 0.18-0.41) and other attendings (ICC = 0.22, 95% CI 0.14-0.35). There were moderate correlations between the mean scores for each blog post when either trainees or attendings considered the same target audience. The RM-ANOVA also corroborated that there is a main effect of the proposed target audience on the ratings by both trainees and attendings. A gestalt-based rating system is not sufficiently reliable when recommending online educational resources to trainees and attendings. Trainees' gestalt ratings for recommending resources for both groups were especially unreliable. Our findings suggest the need for structured rating systems to rate online educational resources. |
X Demographics
The data shown below were collected from the profiles of 38 X users who shared this research output. Click here to find out more about how the information was compiled.
Geographical breakdown
| Country | Count | As % |
|---|---|---|
| United States | 14 | 37% |
| Canada | 8 | 21% |
| Australia | 2 | 5% |
| Turkey | 1 | 3% |
| Kenya | 1 | 3% |
| Unknown | 12 | 32% |
Demographic breakdown
| Type | Count | As % |
|---|---|---|
| Members of the public | 19 | 50% |
| Scientists | 10 | 26% |
| Practitioners (doctors, other healthcare professionals) | 8 | 21% |
| Science communicators (journalists, bloggers, editors) | 1 | 3% |
Mendeley readers
The data shown below were compiled from readership statistics for 42 Mendeley readers of this research output. Click here to see the associated Mendeley record.
Geographical breakdown
| Country | Count | As % |
|---|---|---|
| Unknown | 42 | 100% |
Demographic breakdown
| Readers by professional status | Count | As % |
|---|---|---|
| Student > Bachelor | 7 | 17% |
| Professor | 5 | 12% |
| Student > Doctoral Student | 4 | 10% |
| Other | 3 | 7% |
| Lecturer | 2 | 5% |
| Other | 10 | 24% |
| Unknown | 11 | 26% |
| Readers by discipline | Count | As % |
|---|---|---|
| Medicine and Dentistry | 13 | 31% |
| Social Sciences | 5 | 12% |
| Computer Science | 4 | 10% |
| Agricultural and Biological Sciences | 2 | 5% |
| Nursing and Health Professions | 1 | 2% |
| Other | 4 | 10% |
| Unknown | 13 | 31% |
Attention Score in Context
This research output has an Altmetric Attention Score of 29. This is our high-level measure of the quality and quantity of online attention that it has received. This Attention Score, as well as the ranking and number of research outputs shown below, was calculated when the research output was last mentioned on 29 April 2020.
All research outputs
#1,355,997
of 25,382,440 outputs
Outputs from Tijdschrift voor Medisch Onderwijs
#46
of 574 outputs
Outputs of similar age
#26,976
of 325,414 outputs
Outputs of similar age from Tijdschrift voor Medisch Onderwijs
#4
of 27 outputs
Altmetric has tracked 25,382,440 research outputs across all sources so far. Compared to these this one has done particularly well and is in the 94th percentile: it's in the top 10% of all research outputs ever tracked by Altmetric.
So far Altmetric has tracked 574 research outputs from this source. They typically receive a lot more attention than average, with a mean Attention Score of 11.7. This one has done particularly well, scoring higher than 91% of its peers.
Older research outputs will score higher simply because they've had more time to accumulate mentions. To account for age we can compare this Altmetric Attention Score to the 325,414 tracked outputs that were published within six weeks on either side of this one in any source. This one has done particularly well, scoring higher than 91% of its contemporaries.
We're also able to compare this research output to 27 others from the same source and published within six weeks on either side of this one. This one has done well, scoring higher than 85% of its contemporaries.