A prerequisite for a Pearson correlation is normal distribution and metrical data.
If your data is not normally distributed or you have variables with ordinal data (like grades, or a Likert scale or a ranked variable from “low” to “high”) you can still calculate a correlation with the Spearman rank correlation.
This can be done quickly with SciPy
A correlation shows the relationship between two variables. The correlation coefficient can take on values between -1 (negative correlation), 0 (no correlation) and +1 (perfect correlation).
Prerequisites for calculating the Pearson correlation are normal distribution and metrical data. If our data is not normally distributed or not metrical, meaning the data is not a measurement like height or weight, but for example a Likert scale or grades, we can still calculate a rank correlation with Spearman’s rho:
The Spearman rank correlation is also designed for ranked nominal data, meaning data that can be ranked from “low” to “high” and expressed as numerical data.
💡 Note: Be aware to not mix causality and correlation. Two variables that correlate do not necessarily have a causal relationship. It could be a third variable missing that explains the correlation or it is just by chance. This is called a spurious relationship.
First, we import several packages that we will use:
import pandas as pd import numpy as np import pingouin as pg import seaborn as sns import scipy as scipy
We will load the penguin dataset with measurements of different penguin species from seaborn. The data is not normally distributed. You can test for normal distribution visually and with the Shapiro-Wilk-Test.
The question for our correlation is: is there a relationship between the length of the penguin’s flipper and their weight?
We first load and inspect the dataset on the penguin species. Seaborn has different datasets to use, as do other packages.
penguins = sns.load_dataset('penguins') penguins.head()
A seaborn scatterplot shows that the flipper length and the weight do seem to correlate positively.
sns.scatterplot(data=penguins, x="flipper_length_mm", y="body_mass_g")
So, we can test this hypothesis with several spearman correlation functions from different packages.
Spearman Rank Correlation in Pandas
Numpy does not have a correlation function for Spearman’s rho, only for a Pearson correlation. Pandas however does.
x.corr(y) will deliver the relationship between two variables with a Pearson correlation, by adding
method="spearman" we can calculate Spearman’s rho.
Pandas delivers the Spearman’s rho correlation coefficient but does not calculate a p-value. For hypothesis testing, the p-value is crucial to determine the significance of our assumption.
Spearman Rank Correlation in SciPy
Pandas and pingouin’s correlation function handle missing values automatically.
For SciPy we will have to define how missing values should be handled or the result will be “
NaN”. So, we choose the
nan_policy to be
"omit", meaning to ignore and drop missing values. The function will look like this
scipy.stats.spearmanr(x, y, nan_policy="omit").
scipy.stats.spearmanr(penguins["flipper_length_mm"], penguins["body_mass_g"], nan_policy='omit')
The function delivers the Spearman’s rho coefficient and the p-value to determine the significance of the relationship.
Spearman Rank Correlation in Pingouin
A detailed solution is being delivered by a great statistical package called “
We pass our x and y into the correlation function
pg.corr(x,y, method="") and define the
method="spearman". If we leave the method definition out, we will calculate a Pearson correlation.
pg.corr(penguins["flipper_length_mm"], penguins["body_mass_g"], method="spearman")
The output contains the number of cases n, the coefficient rho, the confidence intervals, the p-value, and the power.
The power tells us the probability of detecting a true and strong relationship between variables. If the power is high, we are likely to detect a true effect.
The most important values are the correlation coefficient rs and the p-value. Pingouin also delivers some more useful test statistics.
- If p < 0.05 we assume a significant test result.
- rs is 0,84 which is a highly positive correlation, when 1 is the maximum and a perfect correlation.
Based on rs, we can determine the effect size which tells us the strength of the relationship by interpreting rs after Cohen’s effect size. There are also other interpretations for the effect size.
After Cohen (1992) a value of rs around 0.1 shows a weak relationship, from 0.3 on would be an average effect and from 0.5 upwards will be a strong effect. With rs = 0,84 we interpret a strong relationship.
We can write our results like this:
The relationship between the flipper length and the mass of penguins is highly significant with rs = .840, p < .001, n = 342. According to Cohen (1992) this is a strong effect.
❓ Question: Why do programmers always mix up Halloween and Christmas?
❗ Answer: Because Oct 31 equals Dec 25.
(If you didn’t get this, read our articles on the
int() Python built-in functions!)