tirsdag 9. mai 2017

The Great Transition

This blog post comments on Lester Brown's book The Great Transition, Shifting from Fossil Fuels to Solar and Wind Energy.

I start with a micro-summary of Lester Brown's view on the alternatives to fossil fuels.

Lester Brown is skeptical about nuclear power, mainly due to cost, long planning and construction time to build new nuclear power plants, problems with nuclear waste, and the probability for accidents like those in Chernobyl in 1986 and Fukoshima in 2011. The share of nuclear power in the global electricity production has fallen from 18 percent in 1996 to 11 percent in 2013.

Lester Brown is also skeptical to increase the use of biomass for energy, mainly because cropland should be used for food production and because intact woods are important for biodiversity.

Lester Brown argues that wind and solar energy is the best alternative to fossil fuel energy. He presents many cases where solar or wind accounts for a significant share of the electricity production. Examples of such cases are wind in Denmark and wind and solar in Germany.

Based on what I have read in other sources I fully agree with Lester Brown's viewpoints on the alternatives to fossil fuels.

mandag 13. mars 2017

Detect serial correlation in data with outliers

This is the sixth post in a series of six that describes mathematics for calculation of correlation and trend in data with outliers. The posts are numbered 1 to 6. They should be read consecutively.

Post 1  Introduction to Statistical analysis of data with outliers
Post 2  Correlation when outliers in the data.
Post 3  Trend when outliers in the data.
Post 4  Correlation and trend when an outlier is added. Example.
Post 5  Compare Kendall-Theil and OLS trends.            Simulations.
Post 6  Detect serial correlation when outliers.          Simulations.

The posts are gathered in this pdf document.

Start of post 6:    Detect serial correlation in data with outliers


This chapter deals with Monte Carlo simulations that calculate the serial correlation coefficients in noisy data. They are calculated with two different approaches. One uses the noise values, and the other uses the ranks of the noise values. Both approaches work well when the noise is white and when there is serial correlation in the noise. The approach that uses the ranks works much better than the other when there are outliers in the noise. The results are presented as probability density plots.

torsdag 9. mars 2017

Compare Kendall-Theil and OLS trends

This is the fifth post in a series of six that deals with mathematics for calculation of correlation and trend in data with outliers. The posts are numbered 1 to 6. They should be read consecutively.

Post 1  Introduction to Statistical analysis of data with outliers
Post 2  Correlation when outliers in the data.
Post 3  Trend when outliers in the data.
Post 4  Correlation and trend when an outlier is added. Example.
Post 5  Compare Kendall-Theil and OLS trends.        Simulations.
Post 6  Detect serial correlation when outliers.               Simulations.

The posts are gathered in this pdf document.

Start of post 5:  Compare Kendall-Theil and OLS trends


This blog post deals with Monte Carlo simulations that calculate trends. The calculations use both the Kendall-Theil (K-T) methodology and the Ordinary Least Squares (OLS) methodology. The post compares the results. Both methodologies work well when the noise in the dependent variable is white. They are about equal when there is serial correlation in the dependent variable. K-T is much more robust against outliers than OLS is. The results are presented in probability density plots.

tirsdag 28. februar 2017

Correlation and trend when an outlier is added

This is the fourth blog post in a series of six that deals with mathematics for calculation of correlation and trend in data with outliers. The posts are numbered 1 to 6. They should be read consecutively.

Post 1  Introduction to Statistical analysis of data with outliers
Post 2  Correlation when outliers in the data.
Post 3  Trend when outliers in the data.
Post 4  Correlation and trend when an outlier is added. Example
Post 5  Compare Kendall-Theil and OLS trends.                  Simulations.
Post 6  Detect serial correlation when outliers.                     Simulations.

The posts are gathered in this pdf document.

Start of post 4:
Correlation and trend when an outlier is added.


This blog post contains an example that demonstrates the shortcomings of the mostly used methods to calculate trends and correlations when an outlier is added to the data. It demonstrates that alternative methods based on medians and ranks are more robust against outliers.

lørdag 25. februar 2017

Trend when outliers in the data

This is the third post in a series of six that deals with mathematics for calculation of correlation and trend in data with outliers. The posts are numbered 1 to 6. They should be read consecutively.

Post 1  Introduction to Statistical analysis of data with outliers
Post 2  Correlation when outliers in the data.
Post 3  Trend when outliers in the data.
Post 4  Correlation and trend when an outlier is added.    Example.
Post 5  Compare Kendall-Theil and OLS trends.              Simulations.
Post 6  Detect serial correlation when outliers.                 Simulations.

The posts are gathered in this pdf document.

Start of post 3: Trend when outliers in the data


The method most commonly used in linear regression analysis is to calculate the trend based on the data values. But it is more robust against outliers to calculate it based on the ranks of the data. This blog post discusses the mathematics behind both methods.

onsdag 22. februar 2017

Correlation when outliers in the data

This is the second post in a series of six that deals with mathematics for calculation of correlation and trend in data with outliers. The posts are numbered 1 to 6. They should be read consecutively.

Post 1  Introduction to Statistical analysis of data with outliers
Post 2  Correlation when outliers in the data.
Post 3  Trend when outliers in the data.
Post 4  Correlation and trend when an outlier is added.   Example.
Post 5  Compare Kendall-Theil and OLS trends.             Simulations.
Post 6  Detect serial correlation when outliers.                Simulations.

The posts are gathered in this pdf document.

Start of post 2: Correlation when outliers in the data


The method most commonly used to estimate the correlation between two datasets is to calculate the correlation coefficient based on the values in the two data sets.. But it is more robust against outliers to calculate it based on the ranks of the data. This blog post discusses the mathematics behind both methods.

fredag 17. februar 2017

Introduction to Statistical analysis of data with outliers

This is the first blog post in a series of six that deals with mathematics for calculation of correlation and trend in data with outliers. The posts are numbered 1 to 6. They should be read consecutively. This first post is just an introduction.

Post 1  Introduction to Statistical analysis of data with outliers
Post 2  Calculate correlation when outliers in the data.
Post 3  Calculate trend when outliers in the data.
Post 4  Correlation and trend when an outlier is added.   Example.
Post 5  Compare Kendall-Theil and OLS trends.             Simulations.
Post 6  Detect serial correlation when outliers.                Simulations.

The posts are gathered in this pdf document.

Start of post 1 Introduction to Statistical analysis of data with outliers


Five blog posts in June 2014 deal with the mathematics that is most commonly used when analysing global temperature series. That mathematics is not well suited when there are large outliers in the data. The first blog post in that series gives an overview of those five posts.

Ordinary least square (OLS) error mathematics is the most commonly used method to calculate trends. It is based on data values, and it therefore performs poorly when there are large outliers in the data. Global temperatures do not have large outliers due to both the inertia in the global climate system and due to the thorough processing before the temperature data is released. Other climate data, such as precipitation, snow depth and skiing conditions at specific locations, have large outliers, and the OLS mathematics is not suitable for those data.

The calculation of the Pearson correlation coefficient is also based on data values. This is the most commonly used method to calculate correlation between variables. It too performs poorly when there are large outliers in the data.

Mathematics based on data ranks performs better than mathematics based on data values when analysing data with large outliers. In this series of blog posts I will describe the rank mathematics which I use to calculate the Kendall tau-b correlation coefficient and the Kendall-Theil robust trend line. For comparison I also shortly describe the Pearson and the OLS mathematics.

As will be seen, the mathematics that is used to calculate the Kendall tau-b correlation coefficient and the Kendall-Theil robust trend line is rather simple and easy to explain. But the mathematics that is used to quantify their uncertainties, which are p-values and confidence intervals, is more complicated.

Next post in the series

fredag 13. januar 2017

Rekalkulere trender og korrelasjoner rundt skiforholdene i Nordmarka med metoder som er robuste mot slengere.

Det forrige innlegget, Fremskrivninger for skiforholdene i Nordmarka er for optimistiske, baserte seg på de vanligste metodene for å beregne trendlinjer og korrelasjonskoeffisienter. Innlegget som du leser nå, gjentar beregningene med alternative metoder som kan være mer velegnet for de aktuelle dataene. Resultatene varierer noe, men hovedbildet er det samme. De alternative beregningsmetodene støtter konklusjonene i det forrige innlegget.