Least Squares Solutions - Linear Algebra

  1. For a least-squares solution x^\hat{\mathbf{x}}, what is Ax^A\hat{\mathbf{x}}?
    The orthogonal projection of b\mathbf{b} onto Col A\text{Col}\,A, the closest point to b\mathbf{b} in the column space.
  2. AA has columns (1,0,0)(1, 0, 0) and (0,1,0)(0, 1, 0), and b=(2,3,4)\mathbf{b} = (2, 3, 4). What is the least-squares solution?
    x^=(2,3)\hat{\mathbf{x}} = (2, 3). Ax^=(2,3,0)A\hat{\mathbf{x}} = (2, 3, 0) is the projection of b\mathbf{b}, with error 44.
  3. AA has linearly independent columns and b\mathbf{b} is orthogonal to Col A\text{Col}\,A. What is the least-squares solution?
    x^=0\hat{\mathbf{x}} = \mathbf{0}, because the projection of b\mathbf{b} onto Col A\text{Col}\,A is the zero vector.
  4. If Ax=bA\mathbf{x} = \mathbf{b} is consistent, what are its least-squares solutions?
    Exactly its ordinary solutions, with least-squares error 00.
  5. How many least-squares solutions does Ax=bA\mathbf{x} = \mathbf{b} have when the columns of AA are linearly dependent?
    Infinitely many: they differ by vectors in Nul A\text{Nul}\,A, though all give the same Ax^A\hat{\mathbf{x}}.
  6. To which subspace is the residual b−Ax^\mathbf{b} - A\hat{\mathbf{x}} orthogonal?
    Col A\text{Col}\,A: it is orthogonal to every column of AA.
  7. AA is the single column (1,1,1)(1, 1, 1) and b=(1,2,6)\mathbf{b} = (1, 2, 6). What is the least-squares solution x^\hat{x}?
    x^=3\hat{x} = 3, the mean of the entries of b\mathbf{b}: 1+2+63\frac{1 + 2 + 6}{3}.
  8. What is a least-squares solution x^\hat{\mathbf{x}} of Ax=bA\mathbf{x} = \mathbf{b}?
    A vector that minimizes ∥b−Ax∥\|\mathbf{b} - A\mathbf{x}\| over all x\mathbf{x}.
  9. What is an overdetermined linear system?
    One with more equations than unknowns; such systems are usually inconsistent, which is why least squares is used.
  10. When does Ax=bA\mathbf{x} = \mathbf{b} have a unique least-squares solution for every b\mathbf{b}?
    When the columns of AA are linearly independent (equivalently, ATAA^TA is invertible).
  11. What is the least-squares error of Ax=bA\mathbf{x} = \mathbf{b}?
    ∥b−Ax^∥\|\mathbf{b} - A\hat{\mathbf{x}}\|, the distance from b\mathbf{b} to Col A\text{Col}\,A.
  12. AA has orthogonal columns (1,1,0)(1, 1, 0) and (0,0,1)(0, 0, 1), and b=(1,3,5)\mathbf{b} = (1, 3, 5). What is the least-squares solution?
    x^=(2,5)\hat{\mathbf{x}} = (2, 5). With orthogonal columns, x^i=b⋅aiai⋅ai\hat{x}_i = \frac{\mathbf{b} \cdot \mathbf{a}_i}{\mathbf{a}_i \cdot \mathbf{a}_i}, giving 42\frac{4}{2} and 51\frac{5}{1}.

Study these flashcards