We consider simulating from a distribution with associated density function proportional to \(p(\cdot)\).
using an algorithm that generates a Markov chain ,
without requiring knowledge of the normalizing factor.
We use a conditional density \(q(\boldsymbol{\theta} \mid \boldsymbol{\theta}^{\text{cur}})\) to generate proposals from the current value \(\boldsymbol{\theta}^{\text{cur}}.\)
Our target is often the unnormalized posterior \(p(\cdot) \equiv p(\boldsymbol{\theta} \mid \boldsymbol{y})\), but the algorithm works for general targets (also frequentist)!
Metropolis–Hastings algorithm
Starting from an initial value \(\boldsymbol{\theta}_0\): for \(t=1, \ldots, T\)
draw a proposal value \(\boldsymbol{\theta}_t^{\star} \sim q(\boldsymbol{\theta} \mid \boldsymbol{\theta}_{t-1})\).
Compute the acceptance ratio \[R = \frac{p(\boldsymbol{\theta}_{t}^{\star} \mid \boldsymbol{y})}{p(\boldsymbol{\theta}_{t-1} \mid \boldsymbol{y})}\frac{q(\boldsymbol{\theta}_{t-1} \mid \boldsymbol{\theta}_t^{\star} )}{q(\boldsymbol{\theta}_t^{\star} \mid \boldsymbol{\theta}_{t-1})}\]
With probability \(\alpha=\min\{R, 1\}\), accept the proposal and set \(\boldsymbol{\theta}_t \gets \boldsymbol{\theta}_t^{\star}\), otherwise set the value to the previous state, \(\boldsymbol{\theta}_t \gets \boldsymbol{\theta}_{t-1}\).
Theory
The Metropolis–Hastings algorithm satisfies a technical condition (detailed balance) that ensures that the Markov chain generated is reversible.
Thus, any draw from the posterior will generate a new realization from the posterior.
Provided the starting value has non-zero probability under the posterior, the chain will converge to the stationarity distribution (albeit perhaps slowly).
Interpretation
If \(R>1\), the proposal has higher posterior density and we always accept the move.
If we reject the move, the Markov chain stays at the current value, which induces autocorrelation.
Since the acceptance probability depends only on the density through ratios, normalizing factors of \(p\) and \(q\) cancel out.
Symmetric proposals and random walk
If the proposal is symmetric, the ratio of proposal densities is \[q(\boldsymbol{\theta}_{t-1} \mid \boldsymbol{\theta}_t^{\star} ) / q(\boldsymbol{\theta}_t^{\star} \mid \boldsymbol{\theta}_{t-1}) = 1.\]
Common examples include random walk proposals \[\boldsymbol{\theta}_t^{\star} \gets \boldsymbol{\theta}_{t-1} + \tau Z,\] where \(Z \sim \mathsf{Gauss}(0,1).\)
Independent proposals
If we pick instead a global proposal, we must ensure that \(q\) samples in far regions (recall rejection sampling), otherwise …
Good proposals include heavy tailed distribution such as Student-\(t\) with small degrees of freedom,
centered at the maximum a posteriori \(\widehat{\boldsymbol{\theta}}_{\mathrm{map}}\) and
with scale matrix proportional to \(-\mathbf{H}^{-1}(\boldsymbol{\theta}_t^{\star})\), where \(\mathbf{H}(\cdot)\) is the Hessian of the (unnormalized) log posterior.
Upworthy data example
We model the Poisson rates for headlines with questions or not. Our model is \[\begin{align*}
Y_{i} &\sim \mathsf{Poisson}(n_i\lambda_i), \qquad (i=1,2)\\
\lambda_1 &= \exp(\beta + \kappa) \\
\lambda_2 &= \exp(\beta) \\
\beta & \sim \mathsf{Gauss}(\log 0.01, 1.5) \\
\kappa &\sim \mathsf{Gauss}(0, 1)
\end{align*}\]
1. Empirical mean and standard deviation for each variable,
plus standard error of the mean:
Mean SD Naive SE Time-series SE
beta -4.51268 0.001697 1.697e-05 6.176e-05
kappa 0.07075 0.002033 2.033e-05 9.741e-05
2. Quantiles for each variable:
2.5% 25% 50% 75% 97.5%
beta -4.51591 -4.51385 -4.51273 -4.51154 -4.50929
kappa 0.06673 0.06933 0.07077 0.07212 0.07463
Metropolis–Hastings algorithm
Starting from an initial value \(\boldsymbol{\theta}_0\):
draw a proposal value \(\boldsymbol{\theta}_t^{\star} \sim q(\boldsymbol{\theta} \mid \boldsymbol{\theta}_{t-1})\).
Compute the acceptance ratio \[
R = \frac{p(\boldsymbol{\theta}_t^{\star} \mid \boldsymbol{y})}{p(\boldsymbol{\theta}_{t-1} \mid \boldsymbol{y})}\frac{q(\boldsymbol{\theta}_{t-1} \mid \boldsymbol{\theta}_t^{\star} )}{q(\boldsymbol{\theta}_t^{\star} \mid \boldsymbol{\theta}_{t-1})}
\]
With probability \(\min\{R, 1\}\), accept the proposal and set \(\boldsymbol{\theta}_t \gets \boldsymbol{\theta}_t^{\star}\), otherwise set the value to the previous state, \(\boldsymbol{\theta}_t \gets \boldsymbol{\theta}_{t-1}\).
Calculations
We compute the log of the acceptance ratio, \(\ln R\), to avoid numerical overflow, with the log posterior difference \[
\ln \left\{\frac{p(\boldsymbol{\theta}_t^{\star} \mid \boldsymbol{y})}{p(\boldsymbol{\theta}_{t-1} \mid \boldsymbol{y})}\right\} = \ell(\boldsymbol{\theta}_t^{\star}; \boldsymbol{y}) + \ln p(\boldsymbol{\theta}_t^{\star}) - \ell(\boldsymbol{\theta}_{t-1}; \boldsymbol{y}) - \ln p(\boldsymbol{\theta}_{t-1})
\]
Compare the value of \(\ln R\) (if less than zero) to \(\log(U)\), where \(U \sim \mathsf{unif}(0,1)\).
What proposal?
The independence Metropolis–Hastings uses a global proposal \(q\) which does not depend on the current state (typically centered at the MAP)
This may be problematic with multimodal targets.
The Gaussian random walk takes \(\boldsymbol{\theta}_t^{\star} =\boldsymbol{\theta}_{t-1}+ \sigma_\text{p}Z\), where \(Z \sim \mathsf{Gauss}(0,1)\) and \(\sigma_\text{p}\) is the proposal standard deviation. Random walks allow us to explore the space.
Breaking down the Markov chain
We distinguish between three phases
burn in period: initial draws allowing the algorithm to converge to it’s stationary distribution (discarded)
warmup adaptation period: tuning period for the proposal std. deviation, etc. (discarded)
sampling: draws post burn in and warmup that are kept for inference
Burn in and warmup
We are guaranteed to reach stationarity with Metropolis–Hastings eventually, but it may take a large number of iterations… Using good starting values help minimize this cost.
Visual inspection of chains can help determining the adequate number of burn in or warmup iterations (typically 1000).
When tuning the algorithm parameters, this number is fixed in advance.
Goldilock principle and proposal variance
Mixing of the chain requires just the right variance (not too small nor too large).
Figure 1: Example of traceplot with proposal variance that is too small (top), adequate (middle) and too large (bottom).
Correlograms of MCMC output
Figure 2: Correlogram for the three Markov chains.
Tuning Markov chain Monte Carlo
Outside of starting values, the variance of the proposal has a huge impact on the asymptotic variance.
We can adapt the variance during warmup by increasing/decreasing proposal variance (if acceptance rate is too large/small).
We can check this via the acceptance rate (how many proposals are accepted).
We can optionally thin the Markov chain by keeping one every \(k\) iterations from the sampling period to reduce the storage cost.
Optimal acceptance rates
The following rules were derived for Gaussian targets under idealized situations.
In 1D, rule of thumb is an acceptance rate of \(0.44\) is optimal, and this ratio decreases to \(0.234\) when \(D \geq 2\)(Sherlock, 2013) for random walk Metropolis–Hastings.
Proposals for \(D\)-variate update should have proposal variance of roughly \((2.38^2/d)\times \boldsymbol{\Sigma}\), where \(\boldsymbol{\Sigma}\) is the posterior variance.
For MALA (see later), we get \(0.574\) rather than \(0.234\).
Block update or one parameter at a time?
As with any accept-reject, proposals become inefficient when the dimension \(D\) increase.
but also leads to more autocorrelation between parameters.
Solutions for strongly correlated coefficients
Reparametrize the model to decorrelate variables (orthogonal parametrization).
Block updates: draw correlated parameters together using a multivariate proposal
using the chain history to learn the correlation, if necessary.
Parameter transformation
Parameters may be bounded, say \(\theta_i \in [a,b]\).
We can ignore this and simply discard proposals outside of the range, by setting the log posterior at \(-\infty\) outside \([a,b].\)
We can do a transformation, e.g., \(\log \theta_i\) if \(\theta_i > 0\) and perform a random walk on the unconstrained space: don’t forget Jacobians for \(q(\cdot)\)!
Another alternative is to use truncated distributions as proposals (useful with more complex algorithms like MALA).
Visual diagnostic: trace plots
Display the Markov chain sample path as a function of the number of iterations.
Ideally, run multiple chains to see if they converge to the same mode (for multimodal behaviour).
Markov chains should look like a fat hairy caterpillar!
Check the bayesplot and codaR packages (trace plot, trace rank, correlograms, marginal densities, etc.)
Checking convergence with multiple chains
Four healthy parallel chains for parameters.
Trace rank plot
A trace rank plot compares the rank of the values of the different chain at a given iteration.
With good mixing, the ranks should switch frequently and be distributed uniformly across integers.
Burn-in and trace rank plot
Figure 3: Traceplots of three Markov chains for the same target with different initial values for the first 500 iterations showing a burn in period (left) and trace rank plot after discarding these (right).
Summary (1/3)
Metropolis–Hastings generalizes rejection sampling by building a Markov chain and providing a mechanism for sampling.
Small proposal variance leads to high acceptance rate, but small step sizes. Large variance proposals leads to many rejections, in which case the previous value is carried forward. Both lead to large autocorrelation.
The proposal density can be anything, but must ideally account for the support and allow for exploration of the state.
Summary (2/3)
Good initial starting values can be obtained by computing maximum a posteriori estimates.
Initializing multiple chains at different starting values can be used to check convergence to the stationary distribution.
Mixing will improve if strongly correlated parameters are sampled together.
Summary (3/3)
The optimal acceptance rate depends on the dimension, but guidelines for random walk Metropolis with Gaussian targets are to have 0.44 for a single parameter model and 0.234 for multivariate targets; see Neal (2011) for a heuristic derivation.
To obtain the target acceptance rate, users must tune the variance of the proposal kernel. This is typically achieved by running the chain for some period, computing the empirical acceptance rate and increasing (respectively decreasing) the variance if the acceptance rate is too high (too low).
References
Neal, R. M. (2011). MCMC using Hamiltonian dynamics. In S. Brooks, A. Gelman, G. Jones, & X. L. Meng (Eds.), Handbook of Markov chain Monte Carlo (pp. 113–162). CRC Press. https://doi.org/10.1201/b10905-5
Sherlock, C. (2013). Optimal scaling of the random walk Metropolis: General criteria for the 0.234 acceptance rule. Journal of Applied Probability, 50(1), 1–15. https://doi.org/10.1239/jap/1363784420