-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Request for Letter-Based Representation of All-Pairwise Comparisons in post-hoc testing #342
Comments
@JohnnyDoorn: I think this is meant for ANOVA's? |
Yes, it's about the post-hoc comparison after performing an ANOVA. It would greatly improve the convenience for me (and I can only assume a lot of others) if there could be a letter-based grouping in addition to just the P-value for each comparison. Especially with a lot of comparisons (i.e. with >5 treatments) it is quite tough to manually calculate the letter-based grouping (necessary for graphs and tables), while it would/could be quite simple automatically. Thanks for responding/looking at the request! |
Any chance this could be implemented? @JohnnyDoorn Puzzling out those significance levels manually is quite a pain... |
Hi @CJAB93, Thanks for your suggestion. Just to clarify, would you like to just be able to sort the table by significance? And how would this relate to letters? Cheers, |
Hi @JohnnyDoorn , Thanks for your response. What I would like is the following. If I analyse my data with JASP (ANOVA followed by Tukey post-hoc test) I get the following: If I perform the same analysis using R, I get two things that I like: asterisks showing significance level, and letters showing which treatments differ from each other: This way, I can see within 5 seconds that treatment 5 and 7 differ from 1 (the control), but that 2, 3, 4 and 6 do not. Additionally, I can see at once that 5 and 7 do not differ from each other, but 7 differs from 1, 2, 3, 4 and 6, while 5 does not differ from 4 and 6. These letters are very useful for reporting significance data in a graph: I would like it if JASP could report those letters, so that I can report those letters in Excel-generated graphs and tables for example. If you have many treatments, like in this example, it is virtually impossible (at least quite confusing and time-consuming) to 'calculate' these letters by hand. As a result, me and people I recommended JASP to, switch to other software to perform the ANOVA and Tukey since we use those letters quite often. For me R is an option, but we all really like the clean and straightforward interface of JASP so we would like to use it for all our analyses. I hope these examples clarify what I would like. |
Thanks for explaining! |
@JohnnyDoorn Sorry forgot to respond! The R-package that creates those letters is multcompView. (multcompView::multcompLetters). The grouping is based on the (non-significant) p-values...I do not know how to explain it better than in the above post(s). I suppose researching the package would help you better than my explanations (https://cran.r-project.org/web/packages/multcompView/multcompView.pdf). Example 1: If treatment 1 differs significantly with tr3, and tr2 differs with tr3, but tr1 and tr2 don't differ from eachother, tr1 and tr2 get an "A" while tr3 gets a "B". If there is a tr4 which differs from tr1 AND with tr3, but NOT tr2, it gets a "C" and tr2 gets a "C" as well, so it will have "AC"... Example 2: If you look at the above post at the $comparison table, you'll see that tr 1, 2, 3, 4 and 6 do not differ significantly in all their possible multiple comparisons. So they get the same letter ("a"). However, 1, 2 and 3 DO differ with 5, while 4 and 6 do not differ with 5. So 5, 4 and 6 get the same letter as well, ("b"). Etc. So the grouping is based on the p-values, each group consisting of treatments that all have no significant differences between themselves. This group gets the same letter. But members can be in different groups at the same time, gaining multiple letters. |
@JohnnyDoorn Any progress/insight? :) |
Hi @CJAB93, Thanks for the reminder! I just finished rewriting the R-code of the ANOVA analyses, and your suggestion was on my list for things to add. However, when implementing it, I noticed that the whole grouping method is based solely on the p-value. After discussing it with some team members, we are not sure if we want to implement this method, as it can very quickly get misleading. Kind regards, |
@JohnnyDoorn
These two presentions tell which pairs are significantly different. But, #2 is very easy to understand (although, #1 has some good information). In some cases, however, only #2 is required. I use statistix. It is very easy there. Do you need output of statistix? Include this feature. If you add this, hundreds of students at my university may switch from statistix to JASP! |
I am not sure #2 is easy to understand. I had to reread the explanation above. I do agree with you that some more structure would be nice -- for instance, I like Johnny's checkbox, but maybe more could be done. I particularly like the idea of creating some sort of figure to show this information. For instance, a matrix-style plot would be nice, with p-values on the panels above the diagonal and the jittered boxplot of the two relevant conditions on the corresponding lower panels. |
I have two points to discuss.
Yes, # 1 can also answer the fertilizer ranking question mentioned above (but it will take more time).
Maybe, I am unable to write what I mean. Please ask me if the above text is hard to understand or you have any other question. |
"Coincidentally" I am also active in the field of plant science. In plant science this way of summarizing the results of statistic tests is extremely prevalent (by which I mean almost omnipresent). So prevalent that I took it for granted that every scientist is familiar with this way of reporting statistic differences. A few months back I was already quite surprised that I had to explain it to Johnny but now that EJWagenmakers is also suggesting that she needed explanation, I am convinced it is simply a 'plant science vs mathematical/statistical (?) science' thing where each field has its own conventions. That is why the lack of it makes JASP a lot less interesting for scientists active in the plant science field since it takes quite some time to generate the significance 'levels' (with a, b, etc) by hand. Figure 3 and 4 in the paper mentioned by idhussain are nice examples of how these letters are used to report significant differences of a multiple comparison in a single glance (if you are aware of the conventions regarding these letters, of course). |
Hi @CJAB93 and @idhussain , Back at the time, I already wrote some code to do this, so I will take a look at it again and see if this can be in the next major release! Cheers, |
Hi @JohnnyDoorn |
@EJWagenmakers @JohnnyDoorn @TimKDJ Any luck with the addition of this feature? Regards, Shahid |
@EJWagenmakers @JohnnyDoorn @TimKDJ I request you to please add this feature. Following a procedure on how it is done with R (In post hoc test, we need letter-based raking of means).
$parameters $means $comparison $groups |
Hi @JohnnyDoorn I must agree with the others that some additional representations of (especially very many) multiple comparisons could be very useful! Although whether CLDs as I think they're called, are the optimal solution, remains open to debate, as you suggest above. Several, seemingly more flexible implementations are available in emmeans and the maintainers suggest some improvements over the default at the bottom of the linked page. I would be glad to hear what you think! See below: |
@idhussain @CJAB93 @TarandeepKang |
Is this function now available in the latest version available for download?
…On Wed, Apr 24, 2024 at 6:06 PM Johnny van Doorn ***@***.***> wrote:
Closed #342 <#342> as
completed via jasp-stats/jaspAnova#321
<jasp-stats/jaspAnova#321>.
—
Reply to this email directly, view it on GitHub
<#342 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APQ4TEWAKNK6UOSAH2MDGA3Y66U6HAVCNFSM4HFDFB32U5DIOJSWCZC7NNSXTWQAEJEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW4OZRGI2TSMJWGU4TIOBW>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
This is in 0.19beta available in a few weeks as 0.19final |
@idhussain if you want to check out the beta version you can download the latest version here (depending on your OS) |
Got it. I will try to use. Thank you.
…On Tue, May 21, 2024 at 2:55 PM Johnny van Doorn ***@***.***> wrote:
@idhussain <https://github.com/idhussain> if you want to check out the
beta version you can download the latest version here
<https://static.jasp-stats.org/Nightlies/> (depending on your OS)
—
Reply to this email directly, view it on GitHub
<#342 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APQ4TET6MBMUFCL7J4JON3TZDMKZDAVCNFSM4HFDFB32U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJSGIZDGNZUGM3A>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Thank you for this. I am using it now. Once the regular version is
released, I will share that with my students.
…On Tue, May 21, 2024 at 2:55 PM Johnny van Doorn ***@***.***> wrote:
@idhussain <https://github.com/idhussain> if you want to check out the
beta version you can download the latest version here
<https://static.jasp-stats.org/Nightlies/> (depending on your OS)
—
Reply to this email directly, view it on GitHub
<#342 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APQ4TET6MBMUFCL7J4JON3TZDMKZDAVCNFSM4HFDFB32U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJSGIZDGNZUGM3A>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I was wondering if you could implement that a post-hoc tests will not only be reported using the pairwise comparison p-values, but also sorted to significance level (like multcompView::multcompLetters in R does).
So instead of this:
F P
2-1 0.0001
3-1 0.7998
3-2 0.0050
Also this:
F Level
1 a
2 b
3 a
The text was updated successfully, but these errors were encountered: