1.4 Solutions

1.4.1 Exercise 1.4 - Oblique projections

Suppose that \(\mathsf{span}(\mathbf{X}) \neq \mathsf{span}(\mathbf{W})\), that both \(\mathbf{X}\) and \(\mathbf{W}\) are full-rank \(n \times p\) matrices such that \(\mathbf{X}^\top\mathbf{W}\) and \(\mathbf{W}^\top\mathbf{X}\) are invertible. An oblique projection matrix is of the form \(\mathbf{P}\equiv\mathbf{X}(\mathbf{W}^\top\mathbf{X})^{-1}\mathbf{W}^\top\) and appears in instrumental variable regression. The oblique projection is such that \(\mathrm{im}(\mathbf{P})=\mathsf{span}(\mathbf{X})\), but \(\mathrm{im}(\mathbf{I}-\mathbf{P})=\mathsf{span}(\mathbf{W}^\perp)\). This fact is illustrated below.

We consider two non-parallel vectors in \(\mathbb{R}^2\), \(\mathbf{X}\) and \(\mathbf{W}\).

#Create two vectors (non-parallel)
x <- c(1, 2)
w <- c(-1, 0.1)
#Create oblique projection matrix
P <- x %*% solve(t(w) %*% x) %*% t(w)

isTRUE(all.equal((P %*% P), P)) #P is idempotent

## [1] TRUE

P - t(P) #but not symmetric

##       [,1]   [,2]
## [1,] 0.000 -2.625
## [2,] 2.625  0.000

The figure below shows the projection of a third vector \(\mathbf{v}\) (non-parallel to \(\mathbf{X}\) or \(\mathbf{W}\)) onto the span of \(\mathbf{P}\) (blue), \(\mathbf{P}^\top\) (red), \(\mathbf{I}_2-\mathbf{P}\) (dashed cyan) and \(\mathbf{I}_2-\mathbf{P}^\top\) (dashed orange). The circles indicate the vectors \(\mathbf{W}\) (red) and \(\mathbf{X}\) (blue) on the plane. Notice that \(\mathbf{I}_2-\mathbf{P}^\top \perp \mathbf{P}\), whereas \(\mathbf{I}_2-\mathbf{P} \perp \mathbf{P}^\top\).