Least Squares

Help Questions

Linear Algebra › Least Squares

Questions 1 - 10

$\begin{align*}&\text{Approximate }\begin{bmatrix}x\y\end{bmatrix}\&\text{for the system: }\begin{bmatrix}-12&9\13&14\-5&-16\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}-100\-5\14\end{bmatrix}\end{align*}$

$\begin{bmatrix}4.80\-3.63\end{bmatrix}$

$\begin{bmatrix}3.80\-4.63\end{bmatrix}$

$\begin{bmatrix}0.80\-7.63\end{bmatrix}$

$\begin{bmatrix}12.80\4.37\end{bmatrix}$

Explanation

$\begin{align*}&\text{When the rows of a matrix outnumber the columns in an equation,}\&\text{there are a greater number of equations than variables. In order}\&\text{to approximate the values of these variables, a least-squares}\&\text{approach is appropriate. This is done by multiplying each side}\&\text{of the equation by the transpose of the equation matrix, and}\&\text{solving the resulting equation:}\&Ax=B\&A^{T}Ax=A^{T}B\&(A^{T}A)^{-1}(A^{T}A)x=(A^{T}A)^{-1}A^{T}B\&x=(A^{T}A)^{-1}A^{T}B\&\text{Using this, we can approximate our values:}\&\begin{bmatrix}-12&13&-5\9&14&-16\end{bmatrix}\begin{bmatrix}-12&9\13&14\-5&-16\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}-12&13&-5\9&14&-16\end{bmatrix}\begin{bmatrix}-100\-5\14\end{bmatrix}\&\begin{bmatrix}338&154\154&533\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}1065\-1194\end{bmatrix}\&\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}4.80\-3.63\end{bmatrix}\end{align*}$

$\begin{align*}&\text{Approximate the values of }\begin{bmatrix}x\y\end{bmatrix}\&\text{for the function: }\begin{bmatrix}20&-19\-9&-18\14&-10\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}-8\11\37\end{bmatrix}\end{align*}$

$\begin{bmatrix}0.13\-0.47\end{bmatrix}$

$\begin{bmatrix}-0.87\-1.47\end{bmatrix}$

$\begin{bmatrix}5.13\4.53\end{bmatrix}$

$\begin{bmatrix}-7.87\-8.47\end{bmatrix}$

Explanation

$\begin{align*}&\text{When the rows of a matrix outnumber the columns in an equation,}\&\text{there are a greater number of equations than variables. In order}\&\text{to approximate the values of these variables, a least-squares}\&\text{approach is appropriate. This is done by multiplying each side}\&\text{of the equation by the transpose of the equation matrix, and}\&\text{solving the resulting equation:}\&Ax=B\&A^{T}Ax=A^{T}B\&(A^{T}A)^{-1}(A^{T}A)x=(A^{T}A)^{-1}A^{T}B\&x=(A^{T}A)^{-1}A^{T}B\&\text{Using this, we can approximate our values:}\&\begin{bmatrix}20&-9&14\-19&-18&-10\end{bmatrix}\begin{bmatrix}20&-19\-9&-18\14&-10\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}20&-9&14\-19&-18&-10\end{bmatrix}\begin{bmatrix}-8\11\37\end{bmatrix}\&\begin{bmatrix}677&-358\-358&785\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}259\-416\end{bmatrix}\&\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}0.13\-0.47\end{bmatrix}\end{align*}$

Let $y=(1 \ 2 \ 3 \ 4 \ 5 \10 )$ , and $x=(0 \ 1 \ 2 \3 \4 \5)$ , find the least squares solution for a linear line.

$x=\begin{bmatrix} \frac{5}{21}\ \ \frac{11}{7} \end{bmatrix}$

No solutions exist

$x=\begin{bmatrix} 0\ \ 11 \end{bmatrix}$

$x=\begin{bmatrix} -\frac{7}{21}\ \ \frac{1}{7} \end{bmatrix}$

$x=\begin{bmatrix} -\frac{5}{21}\ \ \frac{11}{7} \end{bmatrix}$

Explanation

The equation for least squares solution for a linear fit looks as follows.

$y=\theta_1+\theta_2x_i$

Recall the formula for method of least squares.

$x=(A^T\cdot A)^{-1}A^T\cdot b$

Remember when setting up the A matrix, that we have to fill one column full of ones.

$A=\begin{bmatrix} 1 & 0 \ 1 & 1\ 1 & 2\ 1 & 3\ 1& 4\ 1& 5 \end{bmatrix}$

$B=\begin{bmatrix} 1 \ 2\ 3\ 4\ 5\10 \end{bmatrix}$

To make things simpler, lets make $C=A^T\cdot A$ , and $D=A^T\cdot b$

$C=\begin{bmatrix} 1 & 1&1 &1 &1 &1 \ 0 & 1& 2& 3& 4& 5 \end{bmatrix}.\begin{bmatrix} 1&0 \ 1& 1\ 1& 2\ 1& 3\ 1& 4\ 1& 5 \end{bmatrix}$

$C=\begin{bmatrix} 6 & 15 \ 15 & 55 \end{bmatrix}$

Now we need to solve for the inverse, we can do this simply by doing the following. We flip the sign on the off diagonal, and change the spots on the main diagonal, then we multiply by $\frac{1}{\det(C)}$ .

$\det(C)=55(6)-(15)(15)=105$

$C^{-1}=\frac{1}{105}\begin{bmatrix}55 & -15 \ -15 & 6 \end{bmatrix}$

$D=\begin{bmatrix} 1&1 &1 &1 &1 & 1\ 0&1 &2 &3 &4 &5 \end{bmatrix} . \begin{bmatrix} 1\ 2\ 3\ 4\ 5\ 10\ \end{bmatrix}$

$D=\begin{bmatrix} 25\ 90 \end{bmatrix}$

$x=C^{-1}\cdot D=\frac{1}{105}\begin{bmatrix} 55& -15\ -15& 6 \end{bmatrix} \cdot \begin{bmatrix} 25\ 90 \end{bmatrix}=\begin{bmatrix} \frac{5}{21}\ \ \frac{11}{7} \end{bmatrix}$

$\begin{align*}&\text{Approximate the values of }\begin{bmatrix}x\y\end{bmatrix}\&\text{for the function: }\begin{bmatrix}15&15\-7&-12\-3&9\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}-120\-4\-180\end{bmatrix}\end{align*}$

$\begin{bmatrix}8.29\-12.69\end{bmatrix}$

$\begin{bmatrix}10.29\-10.69\end{bmatrix}$

$\begin{bmatrix}3.29\-17.69\end{bmatrix}$

$\begin{bmatrix}16.29\-4.69\end{bmatrix}$

Explanation

$\begin{align*}&\text{When the rows of a matrix outnumber the columns in an equation,}\&\text{there are a greater number of equations than variables. In order}\&\text{to approximate the values of these variables, a least-squares}\&\text{approach is appropriate. This is done by multiplying each side}\&\text{of the equation by the transpose of the equation matrix, and}\&\text{solving the resulting equation:}\&Ax=B\&A^{T}Ax=A^{T}B\&(A^{T}A)^{-1}(A^{T}A)x=(A^{T}A)^{-1}A^{T}B\&x=(A^{T}A)^{-1}A^{T}B\&\text{Using this, we can approximate our values:}\&\begin{bmatrix}15&-7&-3\15&-12&9\end{bmatrix}\begin{bmatrix}15&15\-7&-12\-3&9\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}15&-7&-3\15&-12&9\end{bmatrix}\begin{bmatrix}-120\-4\-180\end{bmatrix}\&\begin{bmatrix}283&282\282&450\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}-1232\-3372\end{bmatrix}\&\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}8.29\-12.69\end{bmatrix}\end{align*}$

$\begin{align*}&\text{Using the method of least-squares, approximate x and y values for: }\&\begin{bmatrix}15&-4\19&-15\-2&19\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}-195\-163\15\end{bmatrix}\end{align*}$

$\begin{bmatrix}-11.03\-1.18\end{bmatrix}$

$\begin{bmatrix}-13.03\-3.18\end{bmatrix}$

$\begin{bmatrix}-7.03\2.82\end{bmatrix}$

$\begin{bmatrix}-18.03\-8.18\end{bmatrix}$

Explanation

$\begin{align*}&\text{When the rows of a matrix outnumber the columns in an equation,}\&\text{there are a greater number of equations than variables. In order}\&\text{to approximate the values of these variables, a least-squares}\&\text{approach is appropriate. This is done by multiplying each side}\&\text{of the equation by the transpose of the equation matrix, and}\&\text{solving the resulting equation:}\&Ax=B\&A^{T}Ax=A^{T}B\&(A^{T}A)^{-1}(A^{T}A)x=(A^{T}A)^{-1}A^{T}B\&x=(A^{T}A)^{-1}A^{T}B\&\text{Using this, we can approximate our values:}\&\begin{bmatrix}15&19&-2\-4&-15&19\end{bmatrix}\begin{bmatrix}15&-4\19&-15\-2&19\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}15&19&-2\-4&-15&19\end{bmatrix}\begin{bmatrix}-195\-163\15\end{bmatrix}\&\begin{bmatrix}590&-383\-383&602\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}-6052\3510\end{bmatrix}\&\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}-11.03\-1.18\end{bmatrix}\end{align*}$

$\begin{align*}&\text{Approximate }\begin{bmatrix}x\y\end{bmatrix}\&\text{for the system: }\begin{bmatrix}11&4\-8&0\8&20\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}-42\-66\-36\end{bmatrix}\end{align*}$

$\begin{bmatrix}1.43\-2.84\end{bmatrix}$

$\begin{bmatrix}-1.57\-5.84\end{bmatrix}$

$\begin{bmatrix}5.43\1.16\end{bmatrix}$

$\begin{bmatrix}-7.57\-11.84\end{bmatrix}$

Explanation

$\begin{align*}&\text{When the rows of a matrix outnumber the columns in an equation,}\&\text{there are a greater number of equations than variables. In order}\&\text{to approximate the values of these variables, a least-squares}\&\text{approach is appropriate. This is done by multiplying each side}\&\text{of the equation by the transpose of the equation matrix, and}\&\text{solving the resulting equation:}\&Ax=B\&A^{T}Ax=A^{T}B\&(A^{T}A)^{-1}(A^{T}A)x=(A^{T}A)^{-1}A^{T}B\&x=(A^{T}A)^{-1}A^{T}B\&\text{Using this, we can approximate our values:}\&\begin{bmatrix}11&-8&8\4&0&20\end{bmatrix}\begin{bmatrix}11&4\-8&0\8&20\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}11&-8&8\4&0&20\end{bmatrix}\begin{bmatrix}-42\-66\-36\end{bmatrix}\&\begin{bmatrix}249&204\204&416\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}-222\-888\end{bmatrix}\&\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}1.43\-2.84\end{bmatrix}\end{align*}$

$\begin{align*}&\text{Using the method of least-squares, approximate x and y values for: }\&\begin{bmatrix}14&-9\-10&-4\-8&-6\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}-12\-198\-179\end{bmatrix}\end{align*}$

$\begin{bmatrix}10.91\17.96\end{bmatrix}$

$\begin{bmatrix}11.91\18.96\end{bmatrix}$

$\begin{bmatrix}14.91\21.96\end{bmatrix}$

$\begin{bmatrix}17.91\24.96\end{bmatrix}$

Explanation

$\begin{align*}&\text{When the rows of a matrix outnumber the columns in an equation,}\&\text{there are a greater number of equations than variables. In order}\&\text{to approximate the values of these variables, a least-squares}\&\text{approach is appropriate. This is done by multiplying each side}\&\text{of the equation by the transpose of the equation matrix, and}\&\text{solving the resulting equation:}\&Ax=B\&A^{T}Ax=A^{T}B\&(A^{T}A)^{-1}(A^{T}A)x=(A^{T}A)^{-1}A^{T}B\&x=(A^{T}A)^{-1}A^{T}B\&\text{Using this, we can approximate our values:}\&\begin{bmatrix}14&-10&-8\-9&-4&-6\end{bmatrix}\begin{bmatrix}14&-9\-10&-4\-8&-6\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}14&-10&-8\-9&-4&-6\end{bmatrix}\begin{bmatrix}-12\-198\-179\end{bmatrix}\&\begin{bmatrix}360&-38\-38&133\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}3244\1974\end{bmatrix}\&\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}10.91\17.96\end{bmatrix}\end{align*}$

$\begin{align*}&\text{Approximate the values of }\begin{bmatrix}x\y\end{bmatrix}\&\text{for the function: }\begin{bmatrix}-13&12\-12&1\-2&19\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}-103\-178\-16\end{bmatrix}\end{align*}$

$\begin{bmatrix}12.04\1.51\end{bmatrix}$

$\begin{bmatrix}10.04\-0.49\end{bmatrix}$

$\begin{bmatrix}17.04\6.51\end{bmatrix}$

$\begin{bmatrix}5.04\-5.49\end{bmatrix}$

Explanation

$\begin{align*}&\text{When the rows of a matrix outnumber the columns in an equation,}\&\text{there are a greater number of equations than variables. In order}\&\text{to approximate the values of these variables, a least-squares}\&\text{approach is appropriate. This is done by multiplying each side}\&\text{of the equation by the transpose of the equation matrix, and}\&\text{solving the resulting equation:}\&Ax=B\&A^{T}Ax=A^{T}B\&(A^{T}A)^{-1}(A^{T}A)x=(A^{T}A)^{-1}A^{T}B\&x=(A^{T}A)^{-1}A^{T}B\&\text{Using this, we can approximate our values:}\&\begin{bmatrix}-13&-12&-2\12&1&19\end{bmatrix}\begin{bmatrix}-13&12\-12&1\-2&19\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}-13&-12&-2\12&1&19\end{bmatrix}\begin{bmatrix}-103\-178\-16\end{bmatrix}\&\begin{bmatrix}317&-206\-206&506\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}3507\-1718\end{bmatrix}\&\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}12.04\1.51\end{bmatrix}\end{align*}$

$\begin{align*}&\text{Using the method of least-squares, approximate x and y values for: }\&\begin{bmatrix}1&-17\-6&1\1&-13\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}188\149\19\end{bmatrix}\end{align*}$

$\begin{bmatrix}-26.87\-9.28\end{bmatrix}$

$\begin{bmatrix}-28.87\-11.28\end{bmatrix}$

$\begin{bmatrix}-32.87\-15.28\end{bmatrix}$

$\begin{bmatrix}-35.87\-18.28\end{bmatrix}$

Explanation

$\begin{align*}&\text{When the rows of a matrix outnumber the columns in an equation,}\&\text{there are a greater number of equations than variables. In order}\&\text{to approximate the values of these variables, a least-squares}\&\text{approach is appropriate. This is done by multiplying each side}\&\text{of the equation by the transpose of the equation matrix, and}\&\text{solving the resulting equation:}\&Ax=B\&A^{T}Ax=A^{T}B\&(A^{T}A)^{-1}(A^{T}A)x=(A^{T}A)^{-1}A^{T}B\&x=(A^{T}A)^{-1}A^{T}B\&\text{Using this, we can approximate our values:}\&\begin{bmatrix}1&-6&1\-17&1&-13\end{bmatrix}\begin{bmatrix}1&-17\-6&1\1&-13\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}1&-6&1\-17&1&-13\end{bmatrix}\begin{bmatrix}188\149\19\end{bmatrix}\&\begin{bmatrix}38&-36\-36&459\end{bmatrix}\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}-687\-3294\end{bmatrix}\&\begin{bmatrix}x\y\end{bmatrix}=\begin{bmatrix}-26.87\-9.28\end{bmatrix}\end{align*}$

It is recommended that you use a calculator with matrix arithmetic capability for this question.

Give the equation of the least squares regression line for the following data:

, , , .

Round your coefficients to three decimal digits, if applicable.

Explanation

Form the matrices $X = \begin{bmatrix} 1 & x_{1} \ 1 & x_{2} \1 & x_{3} \1 & x_{4} \end{bmatrix}$ and $Y = \begin{bmatrix} y_{1} \ y _{2} \ y_{3} \ y_{4}\end{bmatrix}$