Academic disciplines that rely significantly on analyzing data and research, like mathematical and scientific studies, use statistical significance.1 A research result is considered statistically significant if it is unlikely to be explained independently by random factors or chance.2 This blog post will explain the idea of statistical significance and what it entails in detail.
Inhaltsverzeichnis
Statistical Significance – In a Nutshell
- Statistical significance is the claim that a set of observed information or data is not the result of coincidence.
- Statistical significance is a measure of the likelihood of a study’s null hypothesis being correct.1
- A high statistical significance shows that an observed correlation between the observed data is unlikely to be coincidental.
- Researchers can use different types of significance tests.
Definition: Statistical significanc
Statistical significance is a claim or determination made by a researcher that a group of observed data results from a particular cause instead of the product of chance or coincidence.2 The statistical significance can be described as strong or weak.3
Testing for statistical significance
Testing for statistical significance is vital in quantitative research. Researchers conducting quantitative studies analyze their observed data through hypothesis testing.1 Therefore, statistical significance testing is the formal way of assessing the correlation between variables or sets of data.3 Here are guidelines for testing statistical significance.
Null and alternative hypotheses
The first step is categorizing the research predictions into null or alternative hypotheses. Hypothesis testing always begins by assuming that the null hypothesis is correct or justified.
After assuming the null hypothesis is accurate, you can use hypothesis testing to assess the probability of obtaining your research results under this assumption. The outcome of your test will help you determine whether to reject or accept your null hypothesis.1
Example: How to formulate the null and alternative hypothesis
You plan an experimental study to test if socializing can make you less productive. Start your experiment by stating your prediction into null or alternative hypothesis:
- Null hypothesis: There isn’t a difference in productivity between socializing and not socializing
- Alternative hypothesis: Socializing less leads to more productivity than socializing more
Test statistics and proportional values (p-values)
All statistical tests produce a test statistic and p-value.
- A test statistic indicates how closely your observed data matches the formulated null hypothesis.3
- The p-value indicates the probability of getting the study outcome if the null premise is correct.
A low p-value means that a result is not easily explainable by chance alone; hence the null hypothesis can be rejected. In contrast, a large p-value means that the result is explainable by chance alone, so you can retain the null hypothesis. Therefore, the p-value determines the statistical significance.
Example: Testing the hypothesis
- You must begin by collecting data from the experiment and control group. The experiment group includes subjects that socialize a lot, while the control group does both (socializing and not socializing).
- Next, record the productivity ratings for both groups on a scale from 1-5.
- The next step is performing a t test to determine whether actively socializing leads to less productivity. Use the difference in the average productivity rates between the two groups to calculate the following:
- The test statistic (t value) to help you determine how much the sample differs from your formulated null hypothesis.
- The p value to show the likelihood of the results showing if the null premise is right
- Compare the p value to your predetermined significance level to interpret your hypothesis test results.
Statistical significance and significance level
The significance level is a value set by a researcher in before the experiment as the brink for statistical significance. The significance level is the extreme jeopardy of making a false optimistic inference that you are prepared to take. The significance level measures the strength of the evidence that must be present in your sample before you decide to reject or accept the null hypothesis.5
A hypothesis test always ends by comparing the p value to the significance level. This helps you determine whether to retain or castoff the null hypothesis.
- If the p value is greater than the significance level, it means that the null supposition is not disproved and the results of the study are not statistically noteworthy
- If the proportional value is smaller than the significance level, the outcomes are statistically significant and are construed as rebutting the null hypothesis.
Most researchers set the significance level at 5%. A 0.05 significance level indicates a 5% risk of concluding that a difference exists when no difference exists.5 It is worth mentioning that hypothesis testing only shows you whether to castoff or maintain the null hypothesis in favour of the alternate hypothesis.
Example: determining the statistical significance
Your hypothesis test gives you a proportional value of 0.00029. This value is less than the predetermined significance level of 0.04. So, you can deliberate your outcomes as statistically significant. Therefore, you can castoff the null hypothesis. This means that the difference in productivity level can be accredited to the tentative influences.
The problem with statistical significance
The primary problem with statistical significance is that researchers categorize outcomes as statistically significant or not using conformist thresholds that lack theoretical bases. This implies that a slight decrease in the p value can alter the findings from nonsignificant to significant, even if there is no noteworthy change in the effect.3
Furthermore, statistical significance can be misleading when used independently. This is because the sample size affects it. For instance, in large samples, the probability of obtaining statistically significant results is high even if the effects are minimal or not noteworthy in real word contexts.
Types of significance in research
Apart from statistical significance, you can use the following to predict research outcomes:
- Practical significance determines if the research results are vital enough to be valuable in the real world. Therefore, it is indicated by the study’s effect size.
Example:
Calculate the impact size of your study’s statistically significant findings in the experiment group.
The Cohen’s d of this result is 0.266, which indicates a minimal impact size
Clinical significance is preferred for intervention and treatment research. Treatments are marked clinically significant when they tangibly improve patients’ lives.3
FAQs
Statistical significance is the claim that a set of observed information or data is not the result of coincidence but can be credited to a particular cause.
Statistical significance measures the likelihood of a study’s null hypothesis being correct, likened to the acceptable level of ambiguity concerning the correct answer.
A hypothesis test always ends by comparing the p value to the significance level.
The p value measures the likelihood that an observed difference could have occurred by coincidence. It determines a result’s statistical significance.
1 Tenny, Steven and Ibrahim Abdelgawad. “Statistical Significance.” National Library of Medicine. November 21, 2022. https://pubmed.ncbi.nlm.nih.gov/29083828/https://www.investopedia.com/terms/s/statistical-significance.asp.
2 CFI Team. “Statistical Significance.” CFI. November 4, 2022. https://corporatefinanceinstitute.com/resources/data-science/statistical-significance/.
3 Kenton, Will. “Statistical Significance: What It Is, How It Works, With Examples. Investopedia. March 28, 2022. https://www.investopedia.com/terms/s/statistically_significant.asp.