1.5 Summary of week 1

Let \(\mathbf{X}\) be an \(n \times p\) full-rank matrix (\(p <n\)). An \(n \times n\) orthogonal projection matrix \(\mathbf{H}\)

• projects on to \(\mathcal{V} \subseteq \mathbb{R}^n\), meaning \(\mathbf{Hv} \in \mathcal{V}\) for any \(\mathbf{v} \in \mathbb{R}^n\);
• is idempotent, meaning \(\mathbf{H} = \mathbf{HH}\);
• is symmetric, meaning \(\mathbf{H} = \mathbf{H}^\top\).

The projection matrix \(\mathbf{H}\) is unique; if \(\mathcal{V} = \mathscr{S}(\mathbf{X})\), then \[\mathbf{H}_{\mathbf{X}} = \mathbf{X}(\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top.\] Since \(\mathbf{X}: \mathbb{R}^n \to \mathbb{R}^p\), \(\mathbf{H}_{\mathbf{X}}\) has rank \(p\). The orthogonal complement \(\mathbf{M}_{\mathbf{X}}\equiv \mathbf{I}_n - \mathbf{H}_{\mathbf{X}}\) projects onto \(\mathscr{S}^{\perp}(\mathbf{X})\).