| School zipcode | \n", "Average ACT Score | \n", "College Eligibility |
|---|---|---|
| 60605 | \n", "25.1 | \n", "80.7 |
| 60607 | \n", "27 | \n", "91.6 |
| ... | \n", "... | \n", "... |
| 60660 | \n", "16.5 | \n", "14.2 | \n", "
![](\"fig1.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```{index} Least-Squares Solutions\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Least-Squares Solutions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Consider a collection of $n$ data points $(x_1,y_1),(x_2,y_2),\\dots,(x_n,y_n)$ in $\\mathbb{R}^2.$ We seek the line $y=c_0+c_1x$ that best fits these data, where $c_0$ is the $y$-intercept of the line and $c_1$ the slope of the line. As shown in the figure below, an OLS loss function $J(c_0,c_1)$ sums the squared vertical separations between the data points $(x_1,y_1),(x_2,y_2),\\dots,(x_n,y_n)$ and the line $y=c_0+c_1x$.\n",
"\n",
"\n",
"![](\"fig2.png\")
\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"If the $n$ data points are collinear, then there exists an exact solution $(c_0,c_1)$ to the system\n",
"\t\n",
"$$\n",
"c_0 + c_1 x_1 = y_1,\n",
"$$\n",
"\n",
"$$\n",
"c_0+c_1x_2 = y_2,\n",
"$$\n",
"\n",
"$$\\vdots$$\n",
"\n",
"$$\n",
"c_0+c_1 x_n = y_n. \n",
"$$\n",
"\n",
"In matrix form, this system of equations becomes \n",
"\n",
"$$\n",
"\\mathbf{A}\\mathbf{c}=\\mathbf{y}, \n",
"$$\n",
"\n",
"where\n",
"\n",
"$$\n",
" \\mathbf{A}=\n",
"\\begin{pmatrix}\n",
"1& x_1\\\\\n",
"1& x_2\\\\\n",
"\\vdots&\\vdots\\\\\n",
"1& x_n\n",
"\\end{pmatrix},\\quad \\mathbf{c}=\n",
"\\begin{pmatrix}\n",
"c_0\\\\\n",
"c_1\n",
"\\end{pmatrix}, \\quad \\, and \\, \\quad \\mathbf{y}=\n",
"\\begin{pmatrix}\n",
"y_1\\\\\n",
"y_2\\\\\n",
"\\vdots\\\\\n",
"y_n\n",
"\\end{pmatrix}.\n",
"$$\n",
"\n",
"\n",
"In general, this linear system with $n>2$ equations and 2 unknowns ($c_0,c_1$) will not have an exact solution. Instead, our goal to find a linear fit to the data is based on a *least-squares solution*, one that solves the following optimization problem:\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```{index} Optimization Problem\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### OLS LINEAR REGRESSION OPTIMIZATION PROBLEM "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
":::::{admonition} OLS Linear Regression Optimization Problem\n",
":class: tip\n",
"\n",
"Find $(c_0,c_1)$ which minimizes the loss function $J(c_0,c_1)$ defined as\n",
"\n",
"$$\n",
"J(c_0,c_1)=\\tfrac{1}{2}\\sum_{i=1}^n (y_i-(c_0+c_1x_i) )^2=\\tfrac{1}{2}\\|\\mathbf{y} - \\mathbf{A}\\mathbf{c}\\|^2.\n",
"$$\n",
"\n",
"(The factor of 1/2 multiplying the sum is introduced to simplify the theoretical analysis of the loss function.)\n",
"\n",
":::::"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The loss function $J$ is zero when the points are collinear and situated on the line $y=c_0+c_1x$; otherwise, $J$ is positive, since it is half the sum of the squared vertical separations between data points and the line $y=c_0+c_1x$, as shown in the previous figure."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```{index} Minimizing the OLS Loss Function via Normal Equations\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Minimizing the OLS Loss Function via Normal Equations"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let $\\mathbf{A}$ be an $n\\times 2$ matrix of real numbers, and let the linear transformation $\\mathbf{L}:\\mathbb{R}^2\\rightarrow\\mathbb{R}^n$ be defined by $\\mathbf{L}(\\mathbf{c})= \\mathbf{A}\\mathbf{c}$, where $\\mathbf{c}$ is a $2\\times 1$ column vector. The range of $\\mathbf{A}$ is the set of all $n\\times 1$ vectors defined by\n",
"\n",
"$$\n",
"range\\, of \\, \\mathbf{A} = \\{ \\mathbf{A}\\mathbf{c} \\mid \\mathbf{c}\\in\\mathbf{R}^2\\}. \n",
"$$\n",
"\n",
"By choosing $\\mathbf{c}=(1,0)^T$, we see that the first column of $\\mathbf{A}$ (denoted $\\mathbf{a}_1$) is in the range of $\\mathbf{A}$. By choosing $\\mathbf{c}=(0,1)^T$, we see that the second column of $\\mathbf{A}$ (denoted $\\mathbf{a}_2$) is also in the range of $\\mathbf{A}$. \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Minimizing the loss function $J(\\mathbf{c}) =\\frac{1}{2}\\|\\mathbf{y} - \\mathbf{A}\\mathbf{c}\\|^2$ is equivalent to minimizing $\\|\\mathbf{y} - \\mathbf{A}\\mathbf{c}\\|$, the latter being the distance between $\\mathbf{y}$ and an arbitrary vector $\\mathbf{A} \\mathbf{c}$ that is in the range of $\\mathbf{A}$ (see figure below). If there is no exact solution (i.e., $\\mathbf{y}$ is not in the range of $\\mathbf{A})$, this minimization is accomplished by choosing $\\mathbf{\\hat{c}}$ such that vector $\\mathbf{y}-\\mathbf{A}\\hat{c}$ is orthogonal to the range of $\\mathbf{A}$. That is, for $\\mathbf{\\hat{c}}$ to be a least-squares solution to $ \\mathbf{A}\\mathbf{c}=\\mathbf{y}$, the vector $\\mathbf{y}-\\mathbf{A}\\mathbf{\\hat{c}}$ must be orthogonal (perpendicular) to each vector in the range of $\\mathbf{A}$. Since both column vectors of $\\mathbf{A}$ (namely, $\\mathbf{a}_1$ and $\\mathbf{a}_2$) are in the range of $\\mathbf{A}$, it follows that $\\mathbf{a}_i\\cdot(\\mathbf{y}-\\mathbf{A}\\mathbf{\\hat{c}})=\\mathbf{a}_i^T(\\mathbf{y}-\\mathbf{A}\\mathbf{\\hat{c}})=0$ for each column vector $\\mathbf{a}_i$ of matrix~$\\mathbf{A}.$\n",
"\n",
"![](\"fig3.png\")
\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Hence,\n",
" \n",
"$$\n",
"\\mathbf{A}^T(\\mathbf{y}-\\mathbf{A}\\mathbf{\\hat{c}})=\\mathbf{0}\\Rightarrow \\label{N1}\n",
"$$\n",
"\n",
"$$\n",
"\\mathbf{A}^T\\mathbf{y}-\\mathbf{A}^T\\mathbf{A}\\mathbf{\\hat{c}} = \\mathbf{0}\\Rightarrow \n",
"$$\n",
"\n",
"$$\n",
"\\mathbf{A}^T\\mathbf{y}=\\mathbf{A}^T\\mathbf{A}\\mathbf{\\hat{c}}\n",
"$$\n",
"\n",
"The last equation is called the **normal equation** for the system $\\mathbf{A}\\mathbf{c}=\\mathbf{y}.$ Solving the normal equations, one obtains optimal values for $c_0,c_1$. \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Example 2.1."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
":::{admonition} Example 2.1\n",
"\n",
"Consider the data $(-1, 1)$, $(0, 0)$, $(1, 2)$, $(2, 3)$. Use the normal equations to find the OLS regression line for the data.\n",
"\n",
":::"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
":::{toggle}\n",
"**Solution.**\n",
"\n",
"The system is\n",
"\n",
"\\begin{align*}\n",
"c_0-c_1&=1,\\\\\n",
"c_0&=0,\\\\\n",
"c_0+c_1&=2,\\\\\n",
"c_0+2c_1&=3;\\\\\n",
"\\end{align*}\n",
"\n",
"\n",
"or, in matrix form,\n",
"\n",
"\\begin{align*}\n",
"\\begin{pmatrix}\n",
"1& -1\\\\\n",
"1& 0\\\\\n",
"1&1\\\\\n",
"1& 2\n",
"\\end{pmatrix}\n",
"\\begin{pmatrix}\n",
"c_0\\\\\n",
"c_1\n",
"\\end{pmatrix}=\n",
"\\begin{pmatrix}\n",
"1\\\\\n",
"0\\\\\n",
"2\\\\\n",
"3\n",
"\\end{pmatrix}.\n",
"\\end{align*}\n",
"The normal equations are \n",
"\\begin{eqnarray}\n",
"\\begin{pmatrix}\n",
"1& 1&1&1\\\\\n",
"-1&0&1&2\n",
"\\end{pmatrix}\n",
"\\begin{pmatrix}\n",
"1\\\\\n",
"0\\\\\n",
"2\\\\\n",
"3\n",
"\\end{pmatrix}=\n",
"\\begin{pmatrix}\n",
"1& 1&1&1\\\\\n",
"-1&0&1&2\n",
"\\end{pmatrix}\n",
"\\begin{pmatrix}\n",
"1& -1\\\\\n",
"1& 0\\\\\n",
"1&1\\\\\n",
"1& 2\n",
"\\end{pmatrix}\n",
"\\begin{pmatrix}\n",
"c_0\\\\\n",
"c_1\n",
"\\end{pmatrix},\n",
"\\label{ne}\n",
"\\end{eqnarray}\n",
"or\n",
"\\begin{eqnarray*}\n",
"\\begin{pmatrix}\n",
"6\\\\\n",
"7\n",
"\\end{pmatrix}=\n",
"\\begin{pmatrix}\n",
"4&2\\\\\n",
"2&6\n",
"\\end{pmatrix}\n",
"\\begin{pmatrix}\n",
"c_0\\\\\n",
"c_1\n",
"\\end{pmatrix}.\n",
"\\end{eqnarray*}\n",
"The least-squares solution is therefore $c_0=11/10$, $c_1=4/5.$ \n",
"\n",
":::"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```{index} Gradient-Based Optimization\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Equivalence of Gradient-Based Optimization"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"Note that the normal equations are equivalent to gradient-based minimization of the OLS linear regression loss function $J(c_0,c_1)=\\frac{1}{2}\\sum_{i=1}^n (y_i-(c_0+c_1x_i) )^2$:\n",
"\n",
"\\begin{align*}\n",
"\\mathbf{A}^T(\\mathbf{y}-\\mathbf{A}\\mathbf{\\hat{c}})&=\n",
"\\begin{pmatrix}\n",
"1 & 1& \\dots& 1\\\\\n",
"x_1&x_2&\\dots&x_n\n",
"\\end{pmatrix}\n",
"\\begin{pmatrix}\n",
"y_1-(c_0+c_1x_1)\\\\\n",
"y_2-(c_0+c_1x_2)\\\\\n",
"\\dots\\\\\n",
"y_n-(c_0+c_1x_n)\n",
"\\end{pmatrix}\\\\\n",
"&=\n",
"\\begin{pmatrix}\n",
"\\sum_{i=1}^n \\bigl(y_i-(c_0+c_1x_i)\\bigr)\\\\\\\\\n",
"\\sum_{i=1}^n x_i\\bigl(y_i-(c_0+c_1x_i)\\bigr)\n",
"\\end{pmatrix}=\n",
"\\begin{pmatrix}\n",
"-\\tfrac{\\partial J}{\\partial c_0}\\\\\n",
"-\\tfrac{\\partial J}{\\partial c_1}\n",
"\\end{pmatrix}=\n",
"\\begin{pmatrix}\n",
"0\\\\\n",
"0\n",
"\\end{pmatrix}.\n",
"\\end{align*}\n",
"The normal equations are equivalent to setting both partial derivatives of $J$ equal to zero, as is required to minimize the loss function $J(c_0,c_1)$.\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Example 2.2."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
":::{admonition} Example 2.2\n",
"\n",
"For the data points in Example 2.1, show how gradient-based optimization of the loss function $J$ gives the same values for $c_0$ and $c_1$.\n",
"\n",
":::"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
":::{toggle}\n",
"\n",
"**Solution:**\n",
"\n",
"The loss function $J(c_0,c_1)$ is \n",
"\n",
"\\begin{align*}\n",
"J(c_0,c_1)&=\\tfrac{1}{2}[\\bigl(1-(c_0-c_1)\\bigr)^2+(0-c_0)^2+\\bigl(2-(c_0+c_1)\\bigr)^2+\\\\\n",
"&\\qquad\\bigl(3-(c_0+2c_1)\\bigr)^2].\n",
"\\end{align*}\n",
"\n",
"To minimize $J$, we solve the linear system for $\\nabla J(c_1,c_2)=\\mathbf{0}$: \n",
"\n",
"$$\n",
"\\frac{\\partial J}{\\partial c_0}\n",
"=4c_0+2c_1-6=0, \\qquad\n",
"\\frac{\\partial J}{\\partial c_1} =2c_0+6c_1-7=0.\n",
"$$\n",
"\n",
"Solving this system, we obtain $c_0=11/10$ and $c_1=4/5$. \n",
"\n",
"\n",
"\n",
"The system used to find these critical values is equivalent to:\n",
"\n",
"\\begin{align}\n",
"%\\scriptstyle\n",
"\\begin{pmatrix}\n",
"%\\scriptstyle\n",
"-\\frac{\\partial J}{\\partial c_0}\\\\\n",
"-\\frac{\\partial J}{\\partial c_1}\n",
"\\end{pmatrix}&=\n",
"\\begin{pmatrix}\n",
"\\scriptstyle\n",
"\\bigl(1-(c_0-c_1)\\bigr)+(0-c_0)+\\bigl(2-(c_0+c_1)\\bigr)+\\bigl(3-(c_0+2c_1\\bigr) \\\\\n",
"\\scriptstyle-\\bigl(1-(c_0-c_1)\\bigr)+0(0-c_0)+\\bigl(2-(c_0+c_1)\\bigr)+2\\bigl(3-(c_0+2c_1)\\bigr)\n",
"\\end{pmatrix}\\nonumber\\\\\n",
"&=\n",
"\\scriptstyle\\begin{pmatrix}\n",
"1& 1&1&1\\\\\n",
"-1&0&1&2\n",
"\\end{pmatrix}\n",
"\\scriptstyle\n",
"\\begin{pmatrix}\n",
"1\\\\\n",
"\\textstyle 0\\\\\n",
"2\\\\\n",
"3\n",
"\\end{pmatrix}-\n",
"\\scriptstyle\\begin{pmatrix}\n",
"1& 1&1&1\\\\\n",
"-1&0&1&2\n",
"\\end{pmatrix}\n",
"\\scriptstyle\\begin{pmatrix}\n",
"1& -1\\\\\n",
"1& 0\\\\\n",
"1&1\\\\\n",
"1& 2\n",
"\\end{pmatrix}\n",
"\\scriptstyle\\begin{pmatrix}\n",
"c_0\\\\\n",
"c_1\n",
"\\end{pmatrix}=\n",
"\\scriptstyle\n",
"\\begin{pmatrix}\n",
"0\\\\\n",
"0\n",
"\\end{pmatrix},\n",
"\\label{ne1}\n",
"\\end{align}\n",
"\n",
"\n",
"Gradient-based optimization expressed is indeed equivalent to the normal equations.\n",
"\n",
":::"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To say that solving the normal equations is mathematically equivalent to gradient-based optimization of the OLS loss function does not imply that the normal equations offer the best numerical method for optimization of the loss function [Epperly 2022]. Beyond the scope of this Module is an assessment of different numerical approaches such as gradient descent and its variants including the Adam method [Sun et. al. 2019], matrix factorizations such as $SVD$ or $QR$ [Aggarwal 2020], and mean-centering and standardizing data [Toth 2020]. \n",
"\n",
"In addition to facilitating numerical analysis, mean-centering simplifies the linear algebra approach. Mean-centered data is obtained from a collection of data points by replacing the original dataset $(x_i,y_i)$ with the data points $(x_i-\\bar{x},y_i-\\bar{y})$, $i=1, \\dots n$, where $\\bar{x}$ and $\\bar{y}$ are the respective means of the $x$ and $y$ coordinates of the original data. One can show that for mean-centered data, $\\hat{c}_0=0$, and the regression line is $y = \\hat{c}_1 x$. Let $\\mathbf{x}=(x_1,x_2,...,x_n)^{tr}$. For mean-centered data, the regression vector ${\\bf \\hat{y}}=\\hat{c}_1\\mathbf{x}$ is obtained by projecting the vector ${\\bf y}$ directly onto the vector ${\\bf x}$ as shown in the figure below.\n",
"\n",
"![](\"fig4.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"The *variation equation* states that the total variation in the y-values is the sum of the explained variation (variation in the corresponding $y$-values on the regression line) and the unexplained variation (the sum of squared residuals or twice the value of the loss function $J$). For mean-centered data, the total variation is $\\mid\\mid \\mathbf{y}\\mid\\mid^2$, the explained variation is $\\mid\\mid \\hat{c}_1\\mathbf{x}\\mid\\mid^2$ and the unexplained variation is $\\mid\\mid \\mathbf{y} -\\hat{c}_1 \\mathbf{x}\\mid\\mid^2$. In other words, the variation equation for mean-centered data is simply the Pythagorean theorem."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Practice Problems"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
":::{admonition} Practice Problems\n",
"\n",
"1) Consider the data $(1, 0)$, $(4, 5)$, $(7, 8)$. Use the normal equations to find the least-squares solution line $y = a + bx$ that best fits the data. \n",
"\n",
"\n",
"2) Consider the data $(-1,1)$, $(0,0)$, $(1,2)$, $(2,3)$. Use the normal equations to find the least-squares solution for the parabola $y=a+bx+cx^2$ that best fits the data.\n",
"\n",
":::"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Solutions to Practice Problems"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
":::{toggle}\n",
"\n",
"Problem 1) \n",
"The system is\n",
"\n",
"\\begin{align*}\n",
"a+b&=0,\\\\\n",
"a + 4b&=5,\\\\\n",
"a+7b&=8;\n",
"\\end{align*}\n",
"\n",
"or, in matrix form,\n",
"\n",
"\\begin{align*}\n",
"\\begin{pmatrix}\n",
"1& 1\\\\\n",
"1& 4\\\\\n",
"1&7\\\\\n",
"\\end{pmatrix}\n",
"\\begin{pmatrix}\n",
"a\\\\\n",
"b\\\\\n",
"\\end{pmatrix}=\n",
"\\begin{pmatrix}\n",
"0\\\\\n",
"5\\\\\n",
"8\\\\\n",
"\\end{pmatrix}.\n",
"\\end{align*}\n",
"\n",
"The normal equations are\n",
"\n",
"\\begin{align*}\n",
"\\begin{pmatrix}\n",
"1& 1&1\\\\\n",
"1&4&7\\\\\n",
"\\end{pmatrix}\n",
"\\begin{pmatrix}\n",
"0\\\\\n",
"5\\\\\n",
"8\\\\\n",
"\\end{pmatrix}=\n",
"\\begin{pmatrix}\n",
"1& 1&1\\\\\n",
"1&4&7\\\\\n",
"\\end{pmatrix}\n",
"\\begin{pmatrix}\n",
"1& 1\\\\\n",
"1& 4\\\\\n",
"1&7\\\\\n",
"\\end{pmatrix}\n",
"\\begin{pmatrix}\n",
"a\\\\\n",
"b\n",
"\\end{pmatrix},\n",
"\\end{align*}\n",
"\n",
"or\n",
"\t\n",
"\\begin{align*}\n",
"\\begin{pmatrix}\n",
"13\\\\\n",
"76\n",
"\\end{pmatrix}=\n",
"\\begin{pmatrix}\n",
"3&12\\\\\n",
"12&66\n",
"\\end{pmatrix}\n",
"\\begin{pmatrix}\n",
"a\\\\\n",
"b\n",
"\\end{pmatrix}.\n",
"\\end{align*}\n",
"\n",
"Solving the normal equations gives the least-squares solution $a=-1$, $b=4/3$. \n",
"\n",
":::\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
":::{toggle}\n",
"\n",
"\n",
"Problem 2)\n",
"\n",
"The system is\n",
"\t\n",
"\\begin{align*}\n",
"a-b+c&=1,\\\\\n",
"a&=0,\\\\\n",
"a+b+c&=2,\\\\\n",
"a+2b+4c&=3;\\\\\n",
"\\end{align*}\n",
"\t\n",
"or, in matrix form,\n",
"\t\n",
"\\begin{align*}\n",
"\\begin{pmatrix}\n",
"1& -1&1\\\\\n",
"1& 0&0\\\\\n",
"1&1&1\\\\\n",
"1& 2&4\n",
"\\end{pmatrix}\n",
"\\begin{pmatrix}\n",
"a\\\\\n",
"b\\\\\n",
"c\n",
"\\end{pmatrix}=\n",
"\\begin{pmatrix}\n",
"1\\\\\n",
"0\\\\\n",
"2\\\\\n",
"3\n",
"\\end{pmatrix}.\n",
"\\end{align*}\n",
"\n",
"The normal equations are\n",
"\n",
"\\begin{align*}\n",
"\\begin{pmatrix}\n",
"1& 1&1&1\\\\\n",
"-1&0&1&2\\\\\n",
"1&0&1&4\n",
"\\end{pmatrix}\n",
"\\begin{pmatrix}\n",
"1\\\\\n",
"0\\\\\n",
"2\\\\\n",
"3\n",
"\\end{pmatrix}=\n",
"\\begin{pmatrix}\n",
"1& 1&1&1\\\\\n",
"-1&0&1&2\\\\\n",
"1&0&1&4\n",
"\\end{pmatrix}\n",
"\\begin{pmatrix}\n",
"1& -1&1\\\\\n",
"1& 0&0\\\\\n",
"1&1&1\\\\\n",
"1& 2&4\n",
"\\end{pmatrix}\n",
"\\begin{pmatrix}\n",
"a\\\\\n",
"b\n",
"\\end{pmatrix},\n",
"\\end{align*}\n",
"or\n",
"\\begin{align*}\n",
"\\begin{pmatrix}\n",
"6\\\\\n",
"7\\\\\n",
"15\n",
"\\end{pmatrix}=\n",
"\\begin{pmatrix}\n",
"4&2&6\\\\\n",
"2&6&8\\\\\n",
"6&8&18\n",
"\\end{pmatrix}\n",
"\\begin{pmatrix}\n",
"a\\\\\n",
"b\\\\\n",
"c\n",
"\\end{pmatrix}.\n",
"\\end{align*}\n",
"\n",
"Solving the normal equations gives the least-squares solution $a=3/5, b=3/10$, and $c=1/2.$\n",
"\n",
"\n",
":::"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Lab"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this lab we will make scatterplots of data which include the OLS regression line.\n",
"\n",
"We begin by importing libraries."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"from sklearn.linear_model import LinearRegression"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Example 1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
":::{admonition} OLS Regression of ACT and College Eligibility\n",
"Datafile: ACTCollegeEligible.csv\n",
":::"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1) Import Libraries, dropping rows with missing data."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | School Name | \n", "Street Address | \n", "ZIP | \n", "Average Score ACT 2012 | \n", "5-Year Cohort Graduation Rate 2012 - Percent | \n", "One-Year Dropout Rate 2012 - Percent | \n", "Freshman On-Track to Graduate 2012 - Percent | \n", "College Eligibility 2012 - Percent | \n", "College Enrollment 2012 - Percent | \n", "College Retention 2010 - Percent | \n", "Misconducts Resulting in Suspensions 2012 - Percent | \n", "Average Days of Suspension 2012 | \n", "Student Attendance 2012 - Percent | \n", "Teacher Attendance 2012 - Percent | \n", "X Coordinate | \n", "Y Coordinate | \n", "Longitude | \n", "Latitude | \n", "Location | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "Chicago Academy High School | \n", "3400 N Austin Ave | \n", "60634 | \n", "19.3 | \n", "71.0 | \n", "4.0 | \n", "82.2 | \n", "35.2 | \n", "75.8 | \n", "71.0 | \n", "4.5 | \n", "2.6 | \n", "93.6 | \n", "95.6 | \n", "1135740.091 | \n", "1922002.941 | \n", "-87.776506 | \n", "41.942154 | \n", "(-87.77650575, 41.94215439) | \n", "
| 3 | \n", "William Jones College Preparatory High School | \n", "606 S State St | \n", "60605 | \n", "25.1 | \n", "87.1 | \n", "1.5 | \n", "98.0 | \n", "80.7 | \n", "92.2 | \n", "90.8 | \n", "10.6 | \n", "4.5 | \n", "93.9 | \n", "95.3 | \n", "1176412.354 | \n", "1897618.088 | \n", "-87.627755 | \n", "41.874419 | \n", "(-87.62775497, 41.87441898) | \n", "
| \n", " | school_id | \n", "legacy_unit_id | \n", "finance_id | \n", "short_name | \n", "long_name | \n", "primary_category | \n", "is_high_school | \n", "is_middle_school | \n", "is_elementary_school | \n", "is_pre_school | \n", "... | \n", "fifth_contact_title | \n", "fifth_contact_name | \n", "seventh_contact_title | \n", "seventh_contact_name | \n", "refugee_services | \n", "visual_impairments | \n", "freshman_start_end_time | \n", "sixth_contact_title | \n", "sixth_contact_name | \n", "hard_of_hearing | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "609966 | \n", "3750 | \n", "23531 | \n", "HAMMOND | \n", "Charles G Hammond Elementary School | \n", "ES | \n", "False | \n", "True | \n", "True | \n", "True | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "
| 1 | \n", "400069 | \n", "4150 | \n", "67081 | \n", "POLARIS | \n", "Polaris Charter Academy | \n", "ES | \n", "False | \n", "True | \n", "True | \n", "False | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "
| 2 | \n", "610191 | \n", "6070 | \n", "29291 | \n", "STONE | \n", "Stone Elementary Scholastic Academy | \n", "ES | \n", "False | \n", "True | \n", "True | \n", "False | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "
| 3 | \n", "400173 | \n", "9648 | \n", "66801 | \n", "PATHWAYS - BRIGHTON PARK HS | \n", "Pathways in Education- Brighton Park | \n", "HS | \n", "True | \n", "False | \n", "False | \n", "False | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "
| 4 | \n", "610153 | \n", "5670 | \n", "25191 | \n", "RYDER | \n", "William H Ryder Math & Science Specialty ES | \n", "ES | \n", "False | \n", "True | \n", "True | \n", "True | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "
5 rows × 92 columns
\n", "| \n", " | school_id | \n", "legacy_unit_id | \n", "finance_id | \n", "short_name | \n", "long_name | \n", "primary_category | \n", "is_high_school | \n", "is_middle_school | \n", "is_elementary_school | \n", "is_pre_school | \n", "... | \n", "fifth_contact_title | \n", "fifth_contact_name | \n", "seventh_contact_title | \n", "seventh_contact_name | \n", "refugee_services | \n", "visual_impairments | \n", "freshman_start_end_time | \n", "sixth_contact_title | \n", "sixth_contact_name | \n", "hard_of_hearing | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "609966 | \n", "3750 | \n", "23531 | \n", "HAMMOND | \n", "Charles G Hammond Elementary School | \n", "ES | \n", "False | \n", "True | \n", "True | \n", "True | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "
| 4 | \n", "610153 | \n", "5670 | \n", "25191 | \n", "RYDER | \n", "William H Ryder Math & Science Specialty ES | \n", "ES | \n", "False | \n", "True | \n", "True | \n", "True | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "
2 rows × 92 columns
\n", "| \n", " | address | \n", "student_count_total | \n", "student_count_black | \n", "student_count_hispanic | \n", "student_count_white | \n", "zip | \n", "
|---|---|---|---|---|---|---|
| 0 | \n", "2819 W 21ST PL | \n", "342 | \n", "33 | \n", "304 | \n", "2 | \n", "60623 | \n", "
| 1 | \n", "620 N SAWYER AVE | \n", "378 | \n", "337 | \n", "33 | \n", "5 | \n", "60624 | \n", "
| \n", " | address | \n", "student_count_total | \n", "student_count_black | \n", "student_count_hispanic | \n", "student_count_white | \n", "zip | \n", "
|---|---|---|---|---|---|---|
| 0 | \n", "2819 W 21ST PL | \n", "342 | \n", "33 | \n", "304 | \n", "2 | \n", "60623 | \n", "
| 11 | \n", "2345 S CHRISTIANA AVE | \n", "559 | \n", "66 | \n", "484 | \n", "9 | \n", "60623 | \n", "
| \n", " | address | \n", "student_count_total | \n", "student_count_black | \n", "student_count_hispanic | \n", "student_count_white | \n", "zip | \n", "
|---|---|---|---|---|---|---|
| 0 | \n", "2819 W 21ST PL | \n", "342 | \n", "33 | \n", "304 | \n", "2 | \n", "60623 | \n", "
| 1 | \n", "2345 S CHRISTIANA AVE | \n", "559 | \n", "66 | \n", "484 | \n", "9 | \n", "60623 | \n", "
| 2 | \n", "3500 W DOUGLAS BLVD | \n", "216 | \n", "212 | \n", "4 | \n", "0 | \n", "60623 | \n", "
| 3 | \n", "3711 W DOUGLAS BLVD | \n", "499 | \n", "480 | \n", "14 | \n", "3 | \n", "60623 | \n", "
| 4 | \n", "4217 W 18TH ST | \n", "316 | \n", "306 | \n", "10 | \n", "0 | \n", "60623 | \n", "
| \n", " | address | \n", "total | \n", "black | \n", "hispanic | \n", "white | \n", "zip | \n", "
|---|---|---|---|---|---|---|
| 0 | \n", "2819 W 21ST PL | \n", "342 | \n", "33 | \n", "304 | \n", "2 | \n", "60623 | \n", "
| \n", " | address | \n", "total | \n", "black | \n", "hispanic | \n", "white | \n", "zip | \n", "%black | \n", "%hispanic | \n", "%white | \n", "
|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "2819 W 21ST PL | \n", "342 | \n", "33 | \n", "304 | \n", "2 | \n", "60623 | \n", "9.6 | \n", "88.9 | \n", "0.0 | \n", "
| 1 | \n", "2345 S CHRISTIANA AVE | \n", "559 | \n", "66 | \n", "484 | \n", "9 | \n", "60623 | \n", "11.8 | \n", "86.6 | \n", "0.0 | \n", "
| 2 | \n", "3500 W DOUGLAS BLVD | \n", "216 | \n", "212 | \n", "4 | \n", "0 | \n", "60623 | \n", "98.1 | \n", "1.9 | \n", "0.0 | \n", "
| 3 | \n", "3711 W DOUGLAS BLVD | \n", "499 | \n", "480 | \n", "14 | \n", "3 | \n", "60623 | \n", "96.2 | \n", "2.8 | \n", "0.0 | \n", "
| 4 | \n", "4217 W 18TH ST | \n", "316 | \n", "306 | \n", "10 | \n", "0 | \n", "60623 | \n", "96.8 | \n", "3.2 | \n", "0.0 | \n", "