“The [3] video pretty much sums it up and by all means is worth a watch.”
Visual demonstration of how unstable p-values are across samples
Full comment
The p-value [1] criterion is often used to test a proposed hypothesis in medical and social science research papers. When you stumble upon something along the lines of "studies that shown that eating broccoli makes people happy" in your everyday life, it comes down to the p-value calculation being small enough <0.05 for the before and after gathered datasets. The p-value method is practically a standard for scientific reporting in some fields. It also has some drastic shortcomings, including, e,g., dramatic instability for tests with only little data [2]. Naturally, people realize that and try to use additional tools and criteria when available. However, scientists are pretty brutally incentivized to publish positive results and, as a result, more often than it should be, too much weight is put on the single p-value criterion. With issues like this in mind, in my opinion, it makes sense to be somewhat skeptical when seeing reports in the news that "A effects B" and definitely not to rush with the conclusions. Trivial, I know. The [3] video pretty much sums it up and by all means is worth a watch. ------------------- [1] link [2] link [3] this video