Effect Size – Explanation, Significance & Examples

Time to read: 7 Minutes
Effect-size-Definition

Any quantitative academic study or experiment, which involves a comparison between two or more variables, requires a means of determining how substantially different these variables are from one another.

The way in which disparities between variables either in terms of their differences or similarities can be quantified is referred to as the Effect Size.

By determining the effect size associated with any data included in their work, academics enable readers to understand the magnitude of differences and similarities found.1

Effect Size – In a Nutshell

In sum, the effect size is most commonly calculated in two different ways, depending on what the research revolves around. The degree to which one may be able to depict when an effect starts to impact and how large the effect size may be is very broad. The most important aspects of the effect size:

  • Has a numeric value.
  • It is distinct from statistical significance.
  • It is enhanced with the introduction of a measure of practical significance in order to allow effective “real-world” comparisons.
  • It can be analyzed in more detail by calculating Cohen’s d or Pearson’s r value.

Definition: Effect size

To give a concise definition, the effect size, quite simply, is the size (or magnitude) of an effect.2
A large effect size indicates that a considerable degree of significance can be attributed to the data in question, while a small effect size suggests that the veracity of research results will be negligible.

Note:

Report results can be presented in one of several styles, however, this article follows the APA guidelines.

Effect size: Significance

While an equivalence is often assumed with the more general concept of significance, in reality, the effect size is distinct for a number of important reasons. In order to explore this further, it is useful to look at what role significance, and ‘statistical’ significance, in particular, plays in the effect size.

Statistical significance

Statistical significance shows that an effect exists in a study. Statistical significance is represented by calculating and assigning a p-value, or probability value, to data. Using the concept of the null hypothesis whereby an inconclusive, random result is considered to be devoid of statistical relevance (ergo “null”) as a starting point, a low p-value indicates that the reverse is true, i.e. an acceptable degree of statistical relevance is in evidence.

However, statistical significance can be misleading as it does not take into account the sample size. Increased sample size will achieve a closer resemblance to “real-world” conditions, moving researchers closer to establishing a causal relationship between two factors.

Practical significance

Research samples, which are large enough to achieve an approximation of “real-world” conditions, as mentioned above, are considered to have practical significance.

While not dependent on sample size, research indicates that the variability of effect sizes is diminished with increasing sample size.3

It is important to report the effect size in research papers in order to indicate the practical significance of any data, which results from a given research project. APA guidelines require published research to include effect sizes and confidence intervals (a method of describing the uncertainty inherent in an estimate) whenever possible.4

Statistical significance vs. practical significance: Example

A widely acknowledged example of an exemplary experiment is “Visual Adaptation of the Perception of Causality” (Rolf et al, 2013).5

In this study, research subjects were shown a digital image of one grey disc moving toward another at the center of a screen. The study concluded that the degree to which the two discs overlapped at the point of contact dictated whether research participants perceived the first disc to have impacted or passed under the second.

The results are statistically significant because they are derived from a relatively large sample size (10,000 repetitions of the experiment), and comprehensive steps were taken to eliminate outside variables, e.g. by ensuring that no subjects were affected by poor eyesight and by using sound shielding and dim room lighting in the laboratory in which the experiments took place. The study has a p-value of 0.5.

The addition of a measure of practical significance would give an indication of the extent to which these results would be replicated under “real-world” conditions.

Effect size: Calculation

While there are many different measures to calculate the effect size, the two which are used mostly are Cohen’s d and Pearson’s r.

In simple terms, Cohen’s d measures the difference in size between two groups, while Pearson’s r measures the strength of the relationship between two variables.

Calculating the effect size with Cohen’s d

Designed in order to provide researchers with a clear method by which to compare two groups, Cohen’s d measures the effect size as a number of standard deviations.6

This is accomplished by subtracting the mean value of group two from the mean value of group one (M1 – M2) and dividing the result by the pooled standard deviation (SD).7 This may be expressed as an equation as follows:

d= (M1 – M2) / SD pooled

A result is a single number that summarizes the variability in a dataset, whereby a higher number indicates that more data points are further away from the mean.8 In other words, a dataset that has been assigned a higher number according to this method, will exhibit more frequent occurrences of extreme discrepancies between individual values within the data.

The key elements of Cohen’s d method may be summarized as follows:

Pooled standard deviation The average degree to which individual values differ within a given dataset.
Standard deviation from a critical group The average degree to which individual values differ within a subset of a given dataset deemed to be of particular significance.
Standard deviation from the pretest data The average degree to which individual values differ within a dataset collected by researchers as a preparatory measure prior to commencing collection of their main dataset.

Example:

Cohen’s d may be used in the calculation of the reliability with which the size of apples can be estimated according to the availability of direct sunlight, i.e. one apple tree is positioned in an open area, while another is positioned nearby beneath the shade of a large agricultural building, affording it little to no direct sunlight.

Upon harvesting the apples from both trees, researchers would be able to take individual weight measurements of each individual apple, with their hypothesis being that the apples grown in the shaded area will be smaller.

In the event, results show that while both trees produce apples with an average weight of 200g, the apples harvested from the tree in the shade weighed between 50g and 500g (admittedly an unlikely scenario), while the apples produced by the tree which has been planted in an open area vary in weight between 180g and 220g.

In this instance, the researchers would be able to assign a higher Cohen’s d number to their dataset, and as such would be able to confirm their hypothesis, despite the fact that the average weight of the apples harvested from each tree was the same.

Calculating the effect size with Pearson’s r

Pearson Correlation or Pearson’s r measures the effect size as an extent of a linear relationship between two variables. This measurement will indicate whether two factors move in the same direction (a negative correlation), or in the opposite direction (a positive correlation).

This spectrum of positive and negative correlations establishes a range between +1 and -1 within which Pearson’s r measures are quantified. A Pearson’s r measurement of 0 (a neutral rating) between two factors indicates that the factors in question do not have an effect on one another.

Calculating a Pearson’s r value requires the use of statistical software in order to generate visual graphs for use in the interpretation of the dataset presented. Pearson’s r may be presented as a formula as follows:

The various elements included in the formula are as follows:

  • Correlation Coefficient
  • Value of the x-variable included in a sample
  • Mean of the values of the x-variable
  • Value of the y-variable in a sample
  • Mean of the y-variable values

FAQs

As discussed above, effect size is the size or magnitude of the relationship between two variables. This relationship is presented as a numeric value.

In order to calculate and assign a numeric effect size value to data, researchers take the differences between a pair of groups and divide it by the standard deviation of one group in the pair.

Effect size is significant because it equips readers with a quantifiable numeric value by which the relevance of data can be reported and assessed.

Sources:

1 Sullivan, Gail M. Feinn, Richard. “Using Effect Size – or Why the P Value Is Not Enough.” Journal of Graduate Medical Education. September, 2012.  https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3444174.

2 Research Rundowns. “Effect Size.” Accessed November 14, 2022. https://researchrundowns.com/quantitative-methods/effect-size/.

3 The Wing Institute. “Does sample size correlate to larger or smaller effect*sizes obtained from reviews of research studies?” Accessed November 14, 2022. https://www.winginstitute.org/does-sample-size-correlate.

4 Cochrane Training. “Interpreting results and drawing conclusions.” Accessed November 14, 2022. https://training.cochrane.org/handbook/archive/v6.2/chapter-15.

5 Rolfs, Martin. Dambacher, Michael. Patrick, Cavanagh. “Visual Adaptation of the Perception of Causality.” Science Direct. February 4, 2013. https://www.sciencedirect.com/science/article/pii/S096098221201490X.

6 Héroux, Martin. “Cohen’s d: A Standardised Measure of Effect Size.” Scientifically Sound. June 29, 2017. https://scientificallysound.org/2017/06/29/cohens-d-and-effect-size/.

7 McLeod, Saul. “What does effect size tell you?” Simply Psychology. 2019. https://online210.psych.wisc.edu/wp-content/uploads/PSY-210_Unit_Materials/PSY-210_Unit09_Materials/McLeod_EffectSize_2019.pdf.

8 Frost, Jim. “Standard Deviation: Interpretations and Calculations.” Accessed November 14, 2022. https://statisticsbyjim.com/basics/standard-deviation/.