Overview of Polynomial Regression

Simple- and multiple linear regression models, Module 2 and Module 3, attempt to model the relationship between one dependent and one or more independent variables (Recall: Dependent vs. independent variables). It was assumed that the relationship between each dependent- and independent variable pair is linear or straight line. We already saw a number of times in the previous modules that that assumption is quite strong and in practice rarely true. In the previous modules we saw that the scatter plots of data points did form distinct curvilinear patterns. Please recall e.g. the income vs. education scatter plot in the Multiple Linear Regression chapter.

Learning Objectives

When you have completed this Module you should be able to

• understand the concept of polynomial regression
• explain the concept of polynomial regression to others
• explain and utilize model assumptions (Ordinary Least Squares assumptions, or OLS)
• estimate parameters
• conduct parameter- and model testing
• conduct a simple residual analysis
• interpret results.

Assumptions for Polynomial Regression Models

For polynomial regression models we assume that:

• the behavior of a dependent variable y can be explained by a linear, or curvilinear, additive relationship between the dependent variable and a set of k independent variables (x_i, i=1 to k)),
• the relationship between the dependent variable y and any independent variable x_i is linear or curvilinear (specifically polynomial),
• the independent variables x_i are independent of each other, and
• the errors are independent, normally distributed with mean zero and a constant variance (OLS).

As you see, there are not too many changes in the assumptions. You already know that these assumptions may, or may not be true. In practice all model assumptions need to be tested, and, in practice, there will be no perfect models. Like in simple- and multiple linear regression we attempt to fit a line, plane or hyperplane to a set of data. (Note: a plane here is synonymous to a surface.) With one dependent variable and two independent variables we are dealing with a 3D space, and therefore the models represent planes, which we can draw if we want. Beyond one dependent- and two independent variables the model represents, what is commonly called, a hyperplane which we cannot draw.

Note: What is a polynomial!? Did you forget! If you throw a ball up in the air, it will follow a path up and down back to the ground. This flight path of the ball is approximately a parabola (recall: y = ax²). The formula of a parabola is an example of a polynomial. Note that the variable power is a constant (2 in this case). In general a polynomial can be written as y = b_nxⁿ + b_n-1x^n-1 + b_n-2x^n-2 + ... + b₃x³ + b₂x² + b₁x + b₀. In this formula all variable exponents are constants. Below you see an animation of a parabola approximating the flight path of that ball.

Note: Sometimes people talk about the polynomial models as either linear or non-linear models. Which one is it!?! Actually, both terms are acceptable. It depends how you want to look at it: The polynomial models are linear with respect to the model parameters _i estimated by b_i , because the parameter exponents are equal to one. However, the polynomial models are nonlinear with respect to the variables, e.g. x², x³, .... All polynomial terms are obtained from the data by raising the data column values to a desired power.

Note: Polynomial terms can be a source for serious multicollinearity. For example, in the polynomial model, e.g. y = b₀ + b₁x + b₂x² the variable x is used twice as an independent variable, first in b₁x and the second time in b₂x² as a squared term. Clearly, x and x² are related, and therefore, multicollinearity can be present.