The Method of Least Squares and Some Alternatives - Part 1

A very important problem in mathematical statistics is that of finding the best linear or non-linear regression equation to express the relation between a dependent variable and one or more independent variables. Given are observations, each subject to random error, greater in number than the parameters in the regression equation, on the dependent variable and the related values of the independent variable(s), which may be known exactly or may also be subject to random error. Related problems are those of choosing the best measures of central tendency and dispersion of the observations. The best solutions of all three problems depend upon the distribution of the random errors. If one assumes that the values of the independent variable(s) are known exactly and that the errors in the observations on the dependent variable are normally distributed, then it is well known that the mean is the best measure of central tendency, the standard deviation is the best measure of dispersion and the method of least squares is the best method of fitting a regression equation. Other assumptions lead to different choices. Most practitioners have tended to make the assumption of normality and not to worry about the consequences when it is not justified. Another problem arises when the data are contaminated by spurious observations (outliers) which come from distributions with different means and/or larger standard deviations. Many methods have been proposed for rejecting outliers or modifying them (or their weights).