Post 1 Introduction to Statistical analysis of data with outliers
Post 2 Correlation when outliers in the data.
Post 3 Trend when outliers in the data.
Post 4 Correlation and trend when an outlier is added. Example
Post 5 Compare Kendall-Theil and OLS trends. Simulations.
Post 6 Detect serial correlation when outliers. Simulations.
The posts are gathered in this pdf document.
Start of post 4:
Correlation and trend when an outlier is added.
This blog post contains an example that demonstrates the shortcomings of the mostly used methods to calculate trends and correlations when an outlier is added to the data. It demonstrates that alternative methods based on medians and ranks are more robust against outliers.
This blog post is based on an example in a course in statistics at Penn State. I chose this example for two reasons. Firstly the course uses other equations for the Kendall tau-b correlation coefficient than mine equations (2.7), (2.8) and (2.9) in the second blog post in this series, and secondly it contains tied values. The course provides the answer with four decimal digits, which allows me to test that my equations are equivalent with the equations in the course and that my implementation is correct.
The x and y variables in the example are:
x: 23 23 27 27 39 41 45 49 50 53 53 54 56 57 58 58 60 61
y: 9.5 27.9 7.8 17.8 31.4 25.9 27.4 25.2 31.1 34.7 42.0 29.1 32.5 30.3 33.0 33.8 41.1 34.5
They are shown graphically in Figure 4.1.
4.1 CorrelationThe correlation coefficients calculated with mine equations are shown in the legend in Figue 4.1. They are exactly the same as calculated by Penn State.
|Figure 4.1: Correlation between the Penn State x and y values calculated with Pearson (r), Spearman (rho) and Kendall (tau-b). The coefficients are exactly the same as calculated by Penn State.|
The Kendall tau-b correlation coefficient is smaller in absolute value than the Pearson and the Spearman correlation coefficients. According to Penn State this is usual. I have experienced the same with other data sets too. The calculated p-value is 0.0001 with Pearson, 0.0003 with Spearman and 0.0011 with Kendall.
4.2 TrendIn climate analysis, x is normally a monotonic increasing time and consequently it does not contain tied values. The equations in the third blog post in this series therefore assume no tied x values when calculating trends. The Penn State example has four sets with tied x values. I therefore adjusted the second of the x values in each set by adding 0.1 to it before calculating the trends with both OLS and Kendall-Theil. The trends are shown in Figure 4.2.
The OLS and the Kendall-Theil trends are close to each other. Both are statistically significant, and both are well within each other's 95% confidence intervals.
The Pearson, Spearman and Kendall tau-b correlation coefficients are almost not influenced by this small adjustment of the x values. The p-values of the Spearmann and the Kendall tau-b correlation coefficients are improved (not shown) because the adjustment increases the number of xy pairs that contributes in the calculations.
4.3 Add an outlierI changed the fifth last y value from 30.3 to -17.8, i.e. I changed it to be an outlier. Thereafter I calculated the trends anew, as shown in Figure 4.3. The OLS trend is largely influenced by the outlier. It is no longer statistically significant, and its 95% confidence interval is much wider than it was before the outlier was added. The Kendall-Theil trend is almost not influenced by the outlier, neither the slope nor its uncertainty. This demonstrates that the Kendall-Theil trend line is much more robust against outliers than the OLS trend line is.
The outlier reduces the correlation coefficient between x and y. With Pearson r the reduction is large, from 0.79 to 0.35. The reduction is much smaller with the two rank correlation coefficients; Spearman rho is reduced from 0.76 to 0.62 and Kendall tau-b from 0.59 to 0.49.
Without the outlier the correlation coefficients are statistically significant with all the three methods. With the outlier added, the Pearson correlation coefficient is far from being so (p-vale 0.16), while the rank correlation coefficients are statistically significant with good margins (p-value 0.01 for both Spearman and Kendall).
This demonstrates that the rank correlation coefficients are more robust against outliers than the Pearson coefficient.
Often we don't know if an outlier is an error measurement or if it is real. But anyway it is wrong to let a single or a few outliers have much more influence on the result than the majority of the measurements. Therefore, when the data contains outliers, it may be best to apply trend calculations based on medians, as the Kendall-Theil calculation is, and correlation calculations based on ranks, as the Kendall tau-b calculation is.