Correlation

Written and reviewed by Devendra Dubey, Mathematics & Commerce faculty at UniDrill (M.Sc. Mathematics, Osmania University · 10 years teaching). Based on NCERT Economics Class XI (Statistics for Economics), Chapter 6.

Does ice cream cause drowning?

In a typical year, the months when ice cream sells the most are also the months when the most people drown. A careless reader would ban ice cream. A careful one would notice that both go up in summer: hot weather sends people to the ice cream stall and to the river.

This is the whole chapter in a story. Correlation studies whether two variables move together, in which direction and how closely. It is a tool for describing the pattern in pairs of numbers: price and quantity demanded, height and weight, income and consumption. What it cannot do is tell you why they move together. Keep that warning in mind, because CUET sets a trap on it every year.

What correlation measures

Correlation always works on paired observations: two values recorded for the same household, the same year or the same firm. Two separate unrelated lists cannot be correlated. There are two things to read from the result: the direction (do they rise together or move opposite ways?) and the intensity (how closely do they follow each other?).

Why should two variables move together at all? There are three different reasons, and only one of them is cause and effect.

For simplicity, this chapter assumes the relationship is linear: the points follow a straight line, rising or falling. Non-linear relationships exist, but the coefficients below do not measure them.

The scatter diagram: look before you calculate

A scatter diagram plots each pair as a point, with one variable on the X-axis and the other on the Y-axis. It gives no number, but it shows two things at a glance: the tilt of the cloud tells the direction, and its tightness tells the intensity. It is the only one of the three tools in this chapter that is not restricted to straight-line relationships.

Karl Pearson's coefficient: the idea

Pearson's coefficient $r$ gives a number for the direction and intensity of a linear relationship between two cardinal (measured) variables. Here is where the formula comes from.

Mark the means $\bar X$ and $\bar Y$ on the scatter diagram. They cut the plane into four regions. For each point, form the deviations $x = X - \bar X$ and $y = Y - \bar Y$. If both are positive (right and above) or both negative (left and below), the product $xy$ is positive. If they have opposite signs, the product is negative. So $\sum xy$ will be large and positive when most points lie in the upper-right and lower-left regions, which is what an upward-sloping cloud looks like. It is large and negative for a downward cloud, and near zero for a shapeless cloud.

But $\sum xy$ has a problem: it depends on the units. Measure the same heights in millimetres instead of centimetres and the number is ten times larger, with the same relationship. The fix is to divide by something carrying the same units, which is $\sqrt{\sum x^2 \sum y^2}$. This makes the result a pure number, and it also guarantees that it never leaves the range from $-1$ to $+1$.

$$r = \dfrac{\sum xy}{\sqrt{\sum x^2 \cdot \sum y^2}} = \dfrac{\sum xy}{N \sigma_x \sigma_y}, \qquad x = X - \bar X, \quad y = Y - \bar Y$$

The two forms are the same, because $\sigma_x = \sqrt{\sum x^2 / N}$. The numerator, $\sum xy / N$, is the covariance, and it carries the sign of $r$ because the denominator is always positive.

Worked example: Pearson's r. Five students study for $X$ hours a week and score $Y$ marks: $X$ = 2, 4, 6, 8, 10 and $Y$ = 45, 48, 60, 58, 74. Find $r$.

A value of 0.94 means a strong positive linear relationship: students who study more tend to score more.

Common wrong step. Forgetting the square root in the denominator: $136 / (40 \times 524) \approx 0.0065$, a meaningless answer. A quick sanity check is to remember that $|r|$ cannot exceed 1. Another common slip is to multiply the raw values $X \times Y$ instead of the deviations. The formula needs deviations from the means.

Pause and predict: if $Y = -3X + 5$ exactly for every point, what is $r$? And if every $X$ is multiplied by 10 and then increased by 5, what happens to $r$?

Check your answer: $r = -1$, because the points lie exactly on a downward line (the steepness of $-3$ does not matter, only the direction). Multiplying by 10 and adding 5 is a change of scale and origin with a positive factor, so $r$ is unchanged.

The step deviation shortcut

When the values are large, work with $U = (X - A)/B$ and $V = (Y - C)/D$, where $A$ and $C$ are assumed means and $B$ and $D$ are common factors. Then $r_{UV} = r_{XY}$, provided $B$ and $D$ have the same sign. When the assumed means are not the actual means, use the raw sums form:

$$r = \dfrac{N\sum UV - \sum U \sum V}{\sqrt{N\sum U^2 - (\sum U)^2}\;\sqrt{N\sum V^2 - (\sum V)^2}}$$

Worked example: step deviation. Advertising spend $X$ (₹ lakh) and sales $Y$ (₹ lakh) of five shops: $X$ = 100, 120, 140, 160, 180 and $Y$ = 200, 230, 250, 290, 300. Use $U = (X - 140)/20$ and $V = (Y - 250)/10$.

Working directly with $X$ and $Y$ gives the same $0.988$, which confirms that scaling and shifting does not change $r$.

Common wrong step. Dropping the $\sum U \sum V$ and the $(\sum V)^2$ terms. That formula is valid only when the assumed means are the actual means. Here $\bar Y$ is 254, not 250, so the shortcut gives $26/\sqrt{10 \times 70} \approx 0.983$, close but wrong. A second slip is using $B$ and $D$ of opposite signs, such as $V = (Y - 250)/(-10)$. That flips every $V$ and so flips $r$ to $-0.99$.

Spearman's rank correlation

Pearson's $r$ needs numbers that can be measured precisely and a straight line relationship. Spearman's rank correlation $r_s$ works on ranks, so it suits three situations: the quality cannot be measured but can be ranked (beauty, honesty, performance), only the ranks are available, or there are extreme values that would distort Pearson's $r$ because ranking limits the influence of any one value.

Rank each variable separately (rank 1 for the highest), find the difference $D$ in the two ranks of each unit, and use

$$r_s = 1 - \dfrac{6\sum D^2}{n^3 - n}$$

The logic: if both rankings agree, every $D$ is $0$ and $r_s = 1$. The more the rankings disagree, the bigger $\sum D^2$ grows, and when one ranking is exactly the reverse of the other, the fraction equals $2$ and $r_s = -1$. We square $D$ because the plain differences cancel out, as the mean's zero deviation property warned us.

Worked example: no ties. Two judges rank six contestants as shown. Find $r_s$.

Here $\sum D^2 = 6$ and $n = 6$, so $r_s = 1 - \dfrac{6 \times 6}{216 - 6} = 1 - \dfrac{36}{210} = 0.83$. The two judges largely agree.

Common wrong step. Adding the plain differences. Here $\sum D = 0$, which would give $r_s = 1$, a perfect agreement that does not exist. Always square before adding.

Ties. When two or more observations have the same value, they share the average of the ranks they would have occupied (two values tied for 4th and 5th each get 4.5). This shrinks $\sum D^2$ artificially, so for each group of $m$ tied values, add $\dfrac{m^3 - m}{12}$ to $\sum D^2$ inside the formula:

$$r_s = 1 - \dfrac{6\left[\sum D^2 + \sum \dfrac{m^3 - m}{12}\right]}{n^3 - n}$$

Worked example: ties. Six students take two tests. Find $r_s$.

Common wrong step. Giving tied students ranks 4 and 5 instead of 4.5 each, or leaving out the correction. Skipping the correction here gives $0.59$ instead of $0.56$, which could land you on a wrong option.

Worked example: an extreme value. Years of experience $X$ = 1 to 6 and salary $Y$ (₹ thousand) = 10, 12, 15, 18, 20, 95. Compare Pearson's $r$ with Spearman's $r_s$.

The one extreme salary drags Pearson's $r$ down to $0.74$, because it sits far from the straight line the other five points follow. (Replace 95 by 22 and $r$ is $0.997$.) Spearman's $r_s$ only asks whether the order agrees, and it does perfectly.

Worked example: perfect relationship, zero correlation. Let $X$ = −2, −1, 0, 1, 2 and $Y = X^2$ = 4, 1, 0, 1, 4. Find $r$.

Here $Y$ is exactly determined by $X$, yet $r = 0$. The relationship is a U-shape: the left half falls and the right half rises, and the two cancel in $\sum xy$. This is why $r = 0$ means "no linear relationship", not "independent".

Beyond the textbook. For the exam: Spearman's $r_s$ is generally lower than Pearson's $r$, because converting values to ranks throws away information. In reality: that is a tendency, not a law. In the extreme value example above, $r_s$ is larger than $r$. Why it matters: never answer "which is bigger" from the general rule alone when the data are given.

Beyond the textbook. For the exam: $r$ is read as strong, moderate or weak by its distance from zero. In reality: statisticians usually quote $r^2$, the share of the variation in $Y$ that a straight line on $X$ accounts for. With $r = 0.8$, $r^2 = 0.64$, so 64 per cent of the variation is accounted for and 36 per cent is not. Why it matters: $r = 0.8$ sounds nearly perfect, but it leaves over a third of the variation unexplained. "Accounted for" is a statistical statement, not a causal one.

Correlation is not causation

A high $r$ shows that two variables move together. It does not show that one makes the other move. The same $r$ can arise from cause and effect, from coincidence or from a hidden third variable, as the table near the top showed. Only reasoning about the real world, not the number, can tell them apart.

Pause and predict: a study finds $r = +0.9$ between ice cream sales and drowning deaths across months of the year. Does the town need to restrict ice cream?

Check your answer: no. The temperature is a third variable that drives both. The coefficient is a fact about the data, and it says nothing about the cause.

Beyond the textbook. For the exam: a steeper line does not mean a larger $r$. In reality: the lines $Y = 2X$ and $Y = 0.1X$ both give $r = 1$ if the points lie exactly on them. The slope depends on the units of $X$ and $Y$, while $r$ does not. Why it matters: $r$ measures how tightly the points hug a line, not how steep the line is.

Exam traps

Formula sheet

60-second revision

Where this chapter connects

• Previous: Measures of Central Tendency provides the means from which every deviation $X - \bar X$ in this chapter is measured.
• Earlier foundation: Organisation of Data introduces the bivariate frequency table, and Presentation of Data the diagrams of which the scatter diagram is a cousin.
• Next: Index Numbers moves from pairs of variables to a single figure that summarises price change over time.