Normal Equations - Linear Algebra

  1. How are the normal equations derived from orthogonality?
    The residual b−Ax^\mathbf{b} - A\hat{\mathbf{x}} is orthogonal to every column of AA, so AT(b−Ax^)=0A^T(\mathbf{b} - A\hat{\mathbf{x}}) = \mathbf{0}.
  2. For A=[[1,0],[0,1],[1,1]]A = [[1, 0], [0, 1], [1, 1]] and b=(1,2,3)\mathbf{b} = (1, 2, 3), what is ATbA^T\mathbf{b}?
    (4,5)(4, 5). These are (1,0,1)⋅b=4(1, 0, 1) \cdot \mathbf{b} = 4 and (0,1,1)⋅b=5(0, 1, 1) \cdot \mathbf{b} = 5.
  3. When is ATAA^TA invertible?
    Exactly when the columns of AA are linearly independent.
  4. How is the null space of ATAA^TA related to the null space of AA?
    They are equal: Nul(ATA)=Nul(A)\text{Nul}(A^TA) = \text{Nul}(A).
  5. If AA is 5×35 \times 3, what is the size of the normal-equation matrix ATAA^TA?
    3×33 \times 3. ATA^T is 3×53 \times 5, so ATAA^TA is 3×33 \times 3.
  6. What structural properties does ATAA^TA always have?
    It is square (n×nn \times n for AA of size m×nm \times n), symmetric, and positive semidefinite.
  7. For A=[[1,0],[0,1],[1,1]]A = [[1, 0], [0, 1], [1, 1]], what is ATAA^TA?
    [[2,1],[1,2]][[2, 1], [1, 2]]. The entries are dot products of the columns (1,0,1)(1, 0, 1) and (0,1,1)(0, 1, 1).
  8. AA is the single column (1,2,2)(1, 2, 2) and b=(3,0,3)\mathbf{b} = (3, 0, 3). Solve the normal equation for x^\hat{x}.
    x^=1\hat{x} = 1. The equation is 9x^=3+0+6=99\hat{x} = 3 + 0 + 6 = 9.
  9. What does the word normal refer to in the normal equations?
    Perpendicular: the equations say the residual b−Ax^\mathbf{b} - A\hat{\mathbf{x}} is normal (orthogonal) to Col A\text{Col}\,A.
  10. What are the normal equations for Ax=bA\mathbf{x} = \mathbf{b}?
    ATAx^=ATbA^TA\hat{\mathbf{x}} = A^T\mathbf{b}. Their solutions are exactly the least-squares solutions.
  11. Solve the normal equations [[2,1],[1,2]]x^=(4,5)[[2, 1], [1, 2]]\hat{\mathbf{x}} = (4, 5).
    x^=(1,2)\hat{\mathbf{x}} = (1, 2). Check: 2+2=42 + 2 = 4 and 1+4=51 + 4 = 5.
  12. When ATAA^TA is invertible, what is the formula for the least-squares solution?
    x^=(ATA)−1ATb\hat{\mathbf{x}} = (A^TA)^{-1}A^T\mathbf{b}.

Study these flashcards