Regression is a mathematical procedure for finding the best fitting curve to a given set of observations by minimizing the sum of the squares of the distances (residuals) between the observation points and the curve. The sum of the squares of the residuals is used instead of the absolute values because this allows the residuals to be treated as continuous differentiable quantities. However, because the squares of the distances are used, outlying points can have a disproportionate effect on the fit.

The linear least squares fitting technique is the simplest and most commonly applied form of linear regression and provides a solution to the problem of finding the best fitting straight line through a set of points (independent variables can be numerical or categorical and the dependent variable should be numerical). In fact, if the functional relationship between the two quantities being graphed is known to within additive or multiplicative constants, it is a common practice to transform the data in such a way that the resulting plot is a straight line. Polynomial or logarithmic transformations are commonly used to achieve this. The formula for linear least squares fitting was derived in Legendre, 1805, and Gauss, 1809. The term "least squares" is from the term "moindres quarrés" coined by Legendre.

Linear regression is by far the most widely used data exploration and modeling method. Moreover, it has been adapted to a broad range of situations that had been considered to lie outside its scope. It plays a strong underlying role in many other modeling methods. Books containing complete and comprehensive description of all the techniques related to linear regression are listed in the references section.