9  Deterministic approximations

So far, we have focused on stochastic approximations of integral. In very large models, Markov chain Monte Carlo suffer from the curse of dimensionality and it is sometimes useful to resort to cheaper approximations. We begin this review by looking at the asymptotic Gaussian limiting distribution of the maximum aposteriori, the Laplace approximations for integrals (Tierney and Kadane 1986), and their applications for model comparison (Raftery 1995) and evaluation of the marginal likelihood. We also discuss integrated nested Laplace approximations (Rue, Martino, and Chopin 2009; Wood 2019), used in hierarchical models with Gaussian components to obtain approximations to the marginal distribution. This material also borrows from Section 8.2 and appendix C.2.2 of Held and Bové (2020).

We make use of Landau’s notation to describe the growth rate of some functions: we write $x=\mathrm{O}(n)$ (big-O) to indicate that the ratio $x/n\to c\in \mathbb{R}$ and $x=\mathrm{o}(n)$ when $x/n\to 0,$ both when $n\to \mathrm{\infty }.$

9.1 Laplace approximation and it’s applications

Proposition 9.1 (Laplace approximation for integrals) The Laplace approximation uses a Gaussian approximation to evaluate integrals of the form $$\begin{array}{r}{I}_n={\int }_a^{b}g(x)\mathrm{d}x={\int }_a^b\exp \{nh(x)\}\mathrm{d}x.\end{array}$$ Assume that $g(x)$ and thus $h(x),$ is concave and and twice differentiable, with a maximum at $x_0\in [a,b].$ We can Taylor expand $h(x)$ to get, $$\begin{array}{r}h(x)=h(x_0)+h^{\prime }(x_0)(x-x_0)+h^{″}(x_0)(x-x_0{)}^2/2+R\end{array}$$ where the remainder $R=\mathrm{O}\{(x-x_0{)}^3\}.$ If $x_0$ is a maximizer and solves $h^{\prime }(x_0)=0,$ then letting $\tau =-n{h}^{″}(x_0),$ we can write ignoring the remainder term the approximation $$\begin{array}{rl}{I}_n& \approx \exp \{nh(x_0)\}{\int }_a^b\exp \{-\frac{1}{2}(x-x_0{)}^2\}\\ & =\exp \{nh(x_0)\}{\left(\frac{2\pi }{\tau }\right)}^{1/2}[\mathrm{\Phi }\{\tau (b-x_0)\}-\mathrm{\Phi }\{\tau (a-x_0)\}]\end{array}$$ upon recovering the unnormalized kernel of a Gaussian random variable centered at $x_0$ with precision $\tau .$ The approximation error is $\mathrm{O}(n^{-1}).$ This quantity reduces to $\exp \{nh(x_0)\}{\left(\frac{2\pi }{\tau }\right)}^{1/2}$ when evaluated over the real line.

The multivariate analog is similar, where now for an integral of the form $\exp \{nh(\mathit{x})\}$ over ${\mathbb{R}}^d,$ we consider the Taylor series expansion $$\begin{array}{rl}h(\mathit{x})& =h({\mathit{x}}_0)+(\mathit{x}-{\mathit{x}}_0{)}^{\mathrm{\top }}{h}^{\prime }({\mathit{x}}_0)+\frac{1}{2}(\mathit{x}-{\mathit{x}}_0{)}^{\mathrm{\top }}{h}^{″}({\mathit{x}}_0)(\mathit{x}-{\mathit{x}}_0)+R.\end{array}$$ We obtain the Laplace approximation at the mode ${\mathit{x}}_0$ satisfying $h^{\prime }({\mathit{x}}_0)={\mathbf{0}}_d,$ $$\begin{array}{r}{I}_n\approx {\left(\frac{2\pi }{n}\right)}^{d/2}|\mathbf{H}({\mathit{x}}_0){|}^{-1/2}\exp \{nh({\mathit{x}}_0)\},\end{array}$$ where $|\mathbf{H}({\mathit{x}}_0)|$ is the determinant of the Hessian matrix of $-h(\mathit{x})$ evaluated at the mode ${\mathit{x}}_0.$

Laplace approximation uses a Taylor series approximation to approximate the density, but since the latter must be non-negative, it performs the approximation on the log scale and back-transform the result. It is important to understand that we can replace $nh(\mathit{x})$ by any $\mathrm{O}(n)$ term.

Corollary 9.1 (Laplace approximation for marginal likelihood) Consider a simple random sample $\mathit{Y}$ of size $n$ from a distribution with parameter vector $\mathit{\theta }\in {\mathbb{R}}^p.$ We are interested in approximating the marginal likelihood. Write (Raftery 1995) $$\begin{array}{r}p(\mathit{y})={\int }_{{\mathbb{R}}^p}p(\mathit{y}\mid \mathit{\theta })p(\mathit{\theta })\mathrm{d}\mathit{\theta }\end{array}$$ and take $$\begin{array}{r}nh(\mathit{\theta })=\log p(\mathit{y}\mid \mathit{\theta })+\log p(\mathit{\theta })\end{array}$$ in Proposition 9.1. Then, evaluating at the maximum a posteriori ${\hat{\mathit{\theta }}}_{\mathrm{MAP}},$ we get $$\begin{array}{r}p(\mathit{y})=p({\hat{\mathit{\theta }}}_{\mathrm{MAP}})p(\mathit{y}\mid {\hat{\mathit{\theta }}}_{\mathrm{MAP}})(2\pi {)}^{d/2}|\mathbf{H}({\hat{\mathit{\theta }}}_{\mathrm{MAP}}){|}^{-1/2}+\mathrm{O}(n^{-1})\end{array}$$ where $-\mathbf{H}$ is the Hessian matrix of second partial derivatives of the unnormalized log posterior. We get the same relationship on the log scale, whence (Tierney and Kadane 1986) $$\begin{array}{r}\log p(\mathit{y})=\log p({\hat{\mathit{\theta }}}_{\mathrm{MAP}})+\log p(\mathit{y}\mid {\hat{\mathit{\theta }}}_{\mathrm{MAP}})+\frac{p}{2}\log (2\pi )-\frac{1}{2}\log |\mathbf{H}({\hat{\mathit{\theta }}}_{\mathrm{MAP}})|+\mathrm{O}(n^{-1})\end{array}$$ If $p(\mathit{\theta })=\mathrm{O}(1)$ and $p(\mathit{y}\mid \mathit{\theta })=\mathrm{O}(n)$ and provided the prior does not impose unnecessary support constraints, we get the same limiting approximation if we replace the maximum a posteriori point estimator ${\hat{\mathit{\theta }}}_{\mathrm{MAP}}$ by the maximum likelihood estimator, and $-\mathbf{H}({\hat{\mathit{\theta }}}_{\mathrm{MAP}})$ by $n\mathit{ı},$ where $\mathit{ı}$ denotes the Fisher information matrix for a sample of size one. We can write the determinant of the $n$-sample Fisher information as $n^p|\mathit{ı}|.$

If we use this approximation instead, we get $$\begin{array}{rl}\log p(\mathit{y})& =\log p(\mathit{y}\mid {\hat{\mathit{\theta }}}_{\mathrm{MLE}})-\frac{p}{2}\log n+\\ & \log p({\hat{\mathit{\theta }}}_{\mathrm{MLE}})-\frac{1}{2}\log |\mathit{ı}|+\frac{p}{2}\log (2\pi )+\mathrm{O}(n^{-1/2})\end{array}$$ where the error is now $\mathrm{O}(n^{-1/2})$ due to replacing the true information by the evaluation at the MLE. The likelihood is $\mathrm{O}(n),$ the second is $\mathrm{O}(\log n)$ and the other three are $\mathrm{O}(1).$ If we take the prior to be a multivariate Gaussian with mean ${\mathit{\theta }}_{\mathrm{MLE}}$ and with variance $\mathit{ı},$ then the approximation error is $\mathrm{O}(n^{-1/2}),$ whereas the marginal likelihood has error $\mathrm{O}(1)$ if we only keep the first two terms. This gives the approximation $$\begin{array}{r}-2\log p(\mathit{y})\approx \mathsf{BIC}=-2\log p(\mathit{y}\mid \mathit{\theta })+p\log n\end{array}$$ If the likelihood contribution dominates the posterior, the $\mathsf{BIC}$ approximation will improve with increasing sample size, so $\exp (-\mathsf{BIC}/2)$ is an approximation fo the marginal likelihood sometimes used for model comparison in Bayes factor, although this derivation shows that the latter neglects the impact of the prior.

Example 9.1 (Bayesian model averaging approximation) Consider the diabetes model from Park and Casella (2008). We fit various linear regression models, considering all best models of a certain type with at most the 10 predictors plus the intercept. In practice, we typically restrict attention to models within some distance of the lowest BIC value, as the weights otherwise will be negligible.

Figure 9.1: BIC as a function of the linear model covariates (left) and Bayesian model averaging approximate weights (in percentage) for the 10 models with the highest posterior weights according to the BIC approximation.

Most of the weight is on a handful of complex models, where the best fitting model only has around $30$% of the posterior mass.

Remark 9.1 (Parametrization for Laplace). Compare to sampling-based methods, the Laplace approximation requires optimization to find the maximum of the function. The Laplace approximation is not invariant to reparametrization: in practice, it is best to perform it on a scale where the likelihood is as close to quadratic as possible in $g(\mathit{\theta })$ and back-transform using a change of variable.

We can also use Laplace approximation to obtain a crude second-order approximation to the posterior. We suppose that the prior is proper.

We can Taylor expand the log prior and log density around their respective mode, say ${\hat{\mathit{\theta }}}_0$ and ${\hat{\mathit{\theta }}}_{\mathrm{MLE}},$ with ${ȷ}_0({\hat{\mathit{\theta }}}_0)$ and $ȷ({\hat{\mathit{\theta }}}_{\mathrm{MLE}})$ denoting negative of the corresponding Hessian matrices evaluated at their mode, meaning the observed information matrix for the likelihood component. Together, these yield $$\begin{array}{rl}\log p(\mathit{\theta })& \approx \log p({\hat{\mathit{\theta }}}_0)-\frac{1}{2}(\mathit{\theta }-{\hat{\mathit{\theta }}}_0{)}^{\mathrm{\top }}{ȷ}_0({\hat{\mathit{\theta }}}_0)(\mathit{\theta }-{\hat{\mathit{\theta }}}_0)\\ \log p(\mathit{y}\mid \mathit{\theta })& \approx \log p(\mathit{y}\mid {\hat{\mathit{\theta }}}_{\mathrm{MLE}})-\frac{1}{2}(\mathit{\theta }-{\hat{\mathit{\theta }}}_{\mathrm{MLE}}{)}^{\mathrm{\top }}ȷ({\hat{\mathit{\theta }}}_{\mathrm{MLE}})(\mathit{\theta }-{\hat{\mathit{\theta }}}_{\mathrm{MLE}})\end{array}$$

In the case of flat prior, the curvature is zero and the prior contribution vanishes altogether. If we apply now Proposition 8.1 to this unnormalized kernel, we get that the approximate posterior must be Gaussian with precision ${ȷ}_n^{-1}$ and mean ${\hat{\mathit{\theta }}}_n,$ where $$\begin{array}{rl}{ȷ}_n& ={ȷ}_0({\hat{\mathit{\theta }}}_0)+ȷ({\hat{\mathit{\theta }}}_{\mathrm{MLE}})\\ {\hat{\mathit{\theta }}}_n& ={ȷ}_n^{-1}\{{ȷ}_0({\hat{\mathit{\theta }}}_0){\hat{\mathit{\theta }}}_0+ȷ({\hat{\mathit{\theta }}}_{\mathrm{MLE}}){\hat{\mathit{\theta }}}_{\mathrm{MLE}}\}\end{array}$$ and note that ${ȷ}_0({\hat{\mathit{\theta }}}_0)=\mathrm{O}(1),$ whereas ${ȷ}_n$ is $\mathrm{O}(n).$

Theorem 9.1 (Bernstein-von Mises theorem) Consider any estimator asymptotically equivalent to the maximum likelihood estimator and suppose that the prior is continuous and positive in a neighborhood of the maximum. Assume further that the regularity conditions for maximum likelihood estimator holds. Then, in the limit as $n\to \mathrm{\infty }$ $$\begin{array}{r}\mathit{\theta }\mid \mathit{y}\stackrel{\cdot }{\sim }\mathsf{Gauss}\{{\hat{\mathit{\theta }}}_{\mathrm{MLE}},{ȷ}^{-1}({\hat{\mathit{\theta }}}_{\mathrm{MLE}})\}\end{array}$$

The conclusions from this result is that, in large samples, the inference obtained from using likelihood-based inference and Bayesian methods will be equivalent: credible intervals will also have guaranteed frequentist coverage.

We can use the statement by replacing the maximum likelihood estimator and the observed information matrix with variants thereof (${\mathit{\theta }}_n$ and ${ȷ}_n,$ or the Fisher information, or any Monte Carlo estimate of the posterior mean and covariance). The differences will be noticeable for small samples, but will vanish as $n$ grows.

Example 9.2 (Gaussian approximations to the posterior) To assess the performance of Laplace approximation, we consider an exponential likelihood $Y_i\mid \lambda \sim \mathsf{expo}(\lambda )$ with conjugate gamma prior $\lambda \sim \mathsf{gamma}(a,b)$. The exponential model has information $i(\lambda )=n/{\lambda }^2$ and the mode of the posterior is $${\hat{\lambda }}_{\mathrm{MAP}}=\frac{n+a-1}{\sum _{i=1}^n{y}_i+b}.$$

Figure 9.2: Gaussian approximation (dashed) to the posterior density (full line) of the exponential rate $\lambda$ for the waiting dataset with an exponential likelihood and a gamma prior with $a=0.01$ and $b=0.01.$ The plots are based on the first $10$ observations (left) and the whole sample of size $n=62$ (right).

Let us now use Laplace approximation to obtain an estimate of the marginal likelihood: because the model is conjugate, the true log marginal likelihood equals $$\begin{array}{r}p(\mathit{y})=\frac{\mathrm{\Gamma }(n+a)}{\mathrm{\Gamma }(a)}\frac{b^a}{{(b+\sum _{i=1}^n{y}_i)}^{n+a}}.\end{array}$$ Recall that while $p(\mathit{y})$ is a function of $\mathit{y}$, we only evaluate it at the observed data so it becomes a normalizing constant for the problem at hand.

n <- length(waiting); s <- sum(waiting)
log_marg_lik <- lgamma(n+a) - lgamma(a) + a*log(b) - (n+a) * log(b+s)
# Laplace approximation
map <- (n + a - 1)/(s + b)
logpost <- function(x){
  sum(dexp(waiting, rate = x, log = TRUE)) +
    dgamma(x, a, b, log = TRUE)
}
# Hessian evaluated at MAP
H <- -numDeriv::hessian(logpost, x = map)
log_marg_laplace <- 1/2*log(2*pi) - c(determinant(H)$modulus) + logpost(map)

For the sample of size $62,$ the exponential model marginal likelihood is $-276.5,$ whereas the Laplace approximation gives $-281.9.$

Proposition 9.2 (Posterior expectation using Laplace method) If we are interested in computing the posterior expectation of a positive real-valued functional $g(\mathit{\theta }):{\mathbb{R}}^d\to {\mathbb{R}}_{+},$ we may write $$\begin{array}{rl}{\mathsf{E}}_{\mathbf{\Theta }\mid \mathit{Y}}(g(\mathit{\theta })\mid \mathit{y})& =\frac{\int g(\mathit{\theta })p(\mathit{y}\mid \mathit{\theta })p(\mathit{\theta })\mathrm{d}\mathit{\theta }}{\int p(\mathit{y}\mid \mathit{\theta })p(\mathit{\theta })\mathrm{d}\mathit{\theta }}\end{array}$$ We can apply Laplace’s method to both numerator and denominator. Let ${\hat{\mathit{\theta }}}_g$ and ${\hat{\mathit{\theta }}}_{\mathrm{MAP}}$ of the integrand of the numerator and denominator, respectively, and the negative of the Hessian matrix of the log integrands $$\begin{array}{rl}{ȷ}_g& =-\frac{{\partial }^2}{\partial \mathit{\theta }\partial {\mathit{\theta }}^{\mathrm{\top }}}\{\log g(\mathit{\theta })+\log p(\mathit{y}\mid \mathit{\theta })+\log p(\mathit{\theta })\},\\ ȷ& =-\frac{{\partial }^2}{\partial \mathit{\theta }\partial {\mathit{\theta }}^{\mathrm{\top }}}\{\log p(\mathit{y}\mid \mathit{\theta })+\log p(\mathit{\theta })\}.\end{array}$$ Putting these together $$\begin{array}{r}{\mathsf{E}}_{\mathbf{\Theta }\mid \mathit{Y}}(g(\mathit{\theta })\mid \mathit{y})=\frac{|ȷ({\hat{\mathit{\theta }}}_{\mathrm{MAP}}){|}^{1/2}}{|{ȷ}_g({\hat{\mathit{\theta }}}_g){|}^{1/2}}\frac{g({\hat{\mathit{\theta }}}_g)p(\mathit{y}\mid {\hat{\mathit{\theta }}}_g)p({\hat{\mathit{\theta }}}_g)}{p(\mathit{y}\mid {\hat{\mathit{\theta }}}_{\mathrm{MAP}})p({\hat{\mathit{\theta }}}_{\mathrm{MAP}})}+\mathrm{O}(n^{-2})\end{array}$$ While the Laplace method has an error $\mathrm{O}(n^{-1}),$ the leading order term of the expansion cancel out from the ratio.

Example 9.3 (Posterior mean for the exponential likelihood) Consider the posterior mean ${\mathsf{E}}_{\mathrm{\Lambda }\mid \mathit{Y}}(\lambda )$ for the model of Example 9.2. Let $s=\sum _{i=1}^n{y}_i$. Then, $$\begin{array}{rl}{\hat{\lambda }}_g& =\frac{(n+a)}{s+b}\\ |{ȷ}_g({\hat{\lambda }}_g){|}^{1/2}& ={\left(\frac{n+a}{{\hat{\lambda }}_g^2}\right)}^{1/2}=\frac{s+b}{(n+a{)}^{1/2}}\end{array}$$

Simplification gives the approximation $$\begin{array}{r}{\hat{\mathsf{E}}}_{\mathrm{\Lambda }\mid \mathit{Y}}(\mathrm{\Lambda })\approx \frac{\exp (-1)}{s+b}\frac{(n+a{)}^{n+a+1/2}}{(n+a-1{)}^{n+a-1/2}}\end{array}$$ which gives $0.03457,$ whereas the true posterior mean is $(n+a)/(s+b)=0.03457.$ The Laplace approximation is equal to the true value up to five significant digits.

9.2 Integrated nested Laplace approximation

In many high dimensional models, use of MCMC is prohibitively expensive and fast, yet accurate calculations are important. One class of models whose special structure is particularly amenable to deterministic approximations.

Consider a model with response $\mathit{y}$ which depends on covariates $\mathbf{x}$ through a latent Gaussian process; typically the priors on the coefficients $\mathit{\beta }\in {\mathbb{R}}^p.$ In applications with splines, or space time processes, the prior precision matrix for $\mathit{\beta }$ will be sparse with a Gaussian Markov random field structure. The dimension $p$ can be substantial (several thousands) with a comparably low-dimensional hyperparameter vector $\mathit{\theta }\in {\mathbb{R}}^m.$ Interest typically then lies in marginal parameters $$\begin{array}{rl}p({\beta }_i\mid \mathit{y})& =\int p({\beta }_i\mid \mathit{\theta },\mathit{y})p(\mathit{\theta }\mid \mathit{y})\mathrm{d}\mathit{\theta }\\ p({\theta }_i\mid \mathit{y})& =\int p(\mathit{\theta }\mid \mathit{y})\mathrm{d}{\mathit{\theta }}_{-i}\end{array}$$ where ${\mathit{\theta }}_{-i}$ denotes the vector of hyperparameters excluding the $i$th element ${\theta }_i.$ The INLA method builds Laplace approximations to the integrands $p({\beta }_i\mid \mathit{\theta },\mathit{y})$ and $p(\mathit{\theta }\mid \mathit{y}),$ and evaluates the integral using quadrature rules over a coarse grid of values of $\mathit{\theta }.$

Write the marginal posterior $p(\mathit{\theta }\mid \mathit{y})$ as $p(\mathit{\beta },\mathit{\theta }\mid \mathit{y})=p(\mathit{\beta }\mid \mathit{\theta },\mathit{y})p(\mathit{\theta }\mid \mathit{y})$ and perform a Laplace approximation for fixed value of $\mathit{\theta }$ for the term $p(\mathit{\beta }\mid \mathit{\theta },\mathit{y}),$ whose mode we denote by $\hat{\mathit{\beta }}.$ This yields $$\begin{array}{r}\tilde{p}(\mathit{\theta }\mid \mathit{y})\propto \frac{p(\hat{\mathit{\beta }},\mathit{\theta }\mid \mathit{y})}{p_G(\hat{\mathit{\beta }}\mid \mathit{y},\mathit{\theta })}=\frac{p(\hat{\mathit{\beta }},\mathit{\theta }\mid \mathit{y})}{|\mathbf{H}(\hat{\mathit{\beta }}){|}^{1/2}}\end{array}$$ and the Laplace approximation has kernel $$p_G(\mathit{\beta }\mid \mathit{y},\mathit{\theta })\propto |\mathbf{H}(\hat{\mathit{\beta }}){|}^{1/2}\exp \{-(\mathit{\beta }-\hat{\mathit{\beta }}{)}^{\mathrm{\top }}\mathbf{H}(\hat{\mathit{\beta }})(\mathit{\beta }-\hat{\mathit{\beta }})/2\};$$ since it is evaluated at $\hat{\mathit{\beta }},$ we retrieve only the determinant of the negative Hessian of $p(\mathit{\beta }\mid \mathit{\theta },\mathit{y}),$ namely $\mathbf{H}(\hat{\mathit{\beta }}).$ Note that the latter is a function of $\mathit{\theta }.$

To obtain $p({\theta }_i\mid \mathit{y})$, we then proceed with

1. finding the mode of $\tilde{p}(\mathit{\theta }\mid \mathit{y})$ using a Newton’s method, approximating the gradient and Hessian via finite differences.
2. Compute the negative Hessian at the mode to get an approximation to the covariance of $\mathit{\theta }.$ Use an eigendecomposition to get the principal directions $\mathit{z}$.
3. In each direction of $\mathit{z}$, consider drops in $\tilde{p}(\mathit{\theta }\mid \mathit{y})$ as we move away from the mode and define a coarse grid based on these, keeping points where the difference in $\tilde{p}(\mathit{\theta }\mid \mathit{y})$ relative to the mode is less than some numerical tolerance $\delta .$
4. Retrieve the marginal by numerical integration using the central composition design outline above. We can also use directly avoid the integration and use the approximation at the posterior mode of $\tilde{p}(\mathit{\theta }\mid \mathit{y}).$

To approximate $p({\beta }_i\mid \mathit{y})$, Rue, Martino, and Chopin (2009) proceed instead by building an approximation of it based on maximizing ${\mathit{\beta }}_{-i}\mid {\beta }_i,\mathit{\theta },\mathit{y}$ to yield ${\hat{\mathit{\beta }}}_{(i)}$ whose $i$th element is ${\beta }_i,$ yielding $$\begin{array}{r}\tilde{p}({\beta }_i\mid \mathit{\theta },\mathit{y})\propto \frac{p({\hat{\mathit{\beta }}}_{(i)},\mathit{\theta }\mid \mathit{y})}{\tilde{p}({\hat{\mathit{\beta }}}_{(i),-i}\mid {\beta }_i,\mathit{\theta },\mathit{y})},\end{array}$$ with a suitable renormalization of $\tilde{p}({\hat{\mathit{\beta }}}_{(i),-i}\mid {\beta }_i,\mathit{\theta },\mathit{y}).$ Such approximations are reminiscent of profile likelihood.

While we could use the Laplace approximation $p_G(\hat{\mathit{\beta }}\mid \mathit{y},\mathit{\theta })$ and marginalize the latter directly, this leads to evaluation of the Laplace approximation to the density far from the mode, which is often inaccurate. One challenge is that $p$ is often very large, so calculation of the Hessian $\mathbf{H}$ is costly to evaluate. Having to evaluate it repeatedly for each marginal ${\beta }_i$ for $i=1,\dots ,p$ is prohibitive since it involves factorizations of $p\times p$ matrices.

To reduce the computational costs, Rue, Martino, and Chopin (2009) propose to use the approximate mean to avoid optimizing and consider the conditional based on the conditional of the Gaussian approximation with mean $\hat{\mathit{\beta }}$ and covariance $\mathbf{\Sigma }={\mathbf{H}}^{-1}(\hat{\mathit{\beta }}),$ $$\begin{array}{r}{\mathit{\beta }}_{-i}\mid {\beta }_i,\mathit{\theta },\mathit{y}\approx {\mathsf{Gauss}}_{p-1}\{{\tilde{\mathit{\beta }}}_{(i)}={\hat{\mathit{\beta }}}_{-i}+{\mathbf{\Sigma }}_{i,i}^{-1}{\mathbf{\Sigma }}_{i,-i}({\beta }_i-{\hat{\beta }}_i,{\mathbf{M}}_{-i,-i}^{-1}\};\end{array}$$ cf. Proposition 1.6. This only requires a rank-one update. Wood (2019) suggest to use a Newton step to correct ${\tilde{\mathit{\beta }}}_{(i)},$ starting from the conditional mean. The second step is to exploit the local dependence on $\mathit{\beta }$ using the Markov structure to build an improvement to the Hessian. Further improvements are proposed in Rue, Martino, and Chopin (2009), who used a simplified Laplace approximation to correct the Gaussian approximation for location and skewness, a necessary step when the likelihood itself is not Gaussian. This leads to a Taylor series approximation to correct the log determinant of the Hessian matrix. Wood (2019) consider a BFGS update to ${\mathbf{M}}_{-i,-i}^{-1}$ directly, which works less well than the Taylor expansion near ${\hat{\beta }}_i$, but improves upon when we move far from this value. Nowadays, the INLA software uses a low-rank variational correction to Laplace method, proposed in van Niekerk and Rue (2024).

The INLA R package provides an interface to fit models with Gaussian latent random effects. While the software is particularly popular for spatio-temporal applications using the SPDE approach, we revisit two examples in the sequel where we can exploit the Markov structure.

Example 9.4 (Stochastic volatility model with INLA) Financial returns $Y_t$ typically exhibit time-varying variability. The stochastic volatility model is a parameter-driven model that specifies $$\begin{array}{rl}{Y}_t& =\exp (h_t/2)Z_t\\ h_t& =\gamma +\varphi (h_{t-1}-\gamma )+\sigma U_t\end{array}$$ where $U_t\stackrel{\mathrm{iid}}{\sim }\mathsf{Gauss}(0,1)$ and $Z_t\sim \stackrel{\mathrm{iid}}{\sim }\mathsf{Gauss}(0,1).$ The INLA documentation provides information about which default prior and hyperparameters are specified. We use a $\mathsf{gamma}(1,0.001)$ prior for the precision.

library(INLA)
# Stochastic volatility model
data(exchangerate, package = "hecbayes")
# Compute response from raw spot exchange rates at noon
y <- 100*diff(log(exchangerate$dexrate))
# 'y' is now a series of percentage of log daily differences
time <- seq_along(y)
data <- data.frame(y = y, time = time)
# Stochastic volatility model
# https://inla.r-inla-download.org/r-inla.org/doc/likelihood/stochvolgaussian.pdf
# The model uses a log link, and a (log)-gamma prior for the precision
f_stochvol <- formula(y ~ f(time, model = "ar1", param = list(prec = c(1, 0.001))))
mod_stochvol <- inla(f_stochvol, family = "stochvol", data = data)
# Obtain summary
summary <- summary(mod_stochvol)
# plot(mod_stochvol)
marg_prec <- mod_stochvol$marginals.hyperpar[[1]]
marg_phi <- mod_stochvol$marginals.hyperpar[[2]]

Figure 9.3: Marginal densities of precision and autocorrelation parameters from the Gaussian stochastic volatility model.

Figure 9.3 shows that the correlation $\varphi$ is nearly one, leading to random walk behaviour and high persistence over time (this is also due to the frequency of observations). This strong serial dependence in the variance is in part responsible for the difficulty in fitting this model using MCMC.

We can use the marginal density approximations to obtain quantiles for summary of interest. The software also includes utilities to transform the parameters using the change of variable formula.

# Compute density, quantiles, etc. via inla.*marginal
## approximate 95% credible interval and marginal post median
INLA::inla.qmarginal(marg_phi, p = c(0.025, 0.5, 0.975))

[1] 0.9706630 0.9847944 0.9929106

# Change of variable to get variance from precision
marg_var <- INLA::inla.tmarginal(
  fun = function(x) { 1 / x }, 
  marginal = marg_prec)
INLA::inla.qmarginal(marg_var, p = c(0.025, 0.975))

[1] 0.2864908 0.7396801

# Posterior marginal mean and variance of phi
mom1 <- INLA::inla.emarginal(
    fun = function(x){x}, 
    marginal = marg_phi)
mom2 <- INLA::inla.emarginal(
    fun = function(x){x^2}, 
    marginal = marg_phi)
c(mean = mom1, sd = sqrt(mom2 - mom1^2))

      mean         sd 
0.98405272 0.00576251 

Example 9.5 (Tokyo binomial time series) We revisit Example 7.6, but this time fit the model with INLA. We specify the mean model without intercept and fit a logistic regression, with a second-order cyclic random walk prior for the coefficients, and the default priors for the other parameters.

data(Tokyo, package = "INLA")
# Formula (removing intercept)
formula <- y ~ f(time, model = "rw2", cyclic = TRUE) - 1
mod <- INLA::inla(
   formula = formula, 
   family = "binomial",
   Ntrials = n, 
   data = Tokyo)

Figure 9.4: Posterior probability per day of the year with posterior median and 95% credible interval for the Tokyo rainfall binomial time series.

Figure 9.4 shows posterior summaries for the $\mathit{\beta },$ which align with the results for the probit model.

If we wanted to obtain predictions, we need to augment the model matrix and set missing values for the response variable. These then get imputed alongside with the other parameters.

Held, Leonhard, and Daniel Sabanés Bové. 2020. Likelihood and Bayesian Inference: With Applications in Biology and Medicine. 2nd ed. Heidelberg: Springer Berlin. https://doi.org/10.1007/978-3-662-60792-3.

Park, Trevor, and George Casella. 2008. “The Bayesian Lasso.” Journal of the American Statistical Association 103 (482): 681–86. https://doi.org/10.1198/016214508000000337.

Raftery, Adrian E. 1995. “Bayesian Model Selection in Social Research.” Sociological Methodology 25: 111–63. https://doi.org/10.2307/271063.

Rue, Håvard, Sara Martino, and Nicolas Chopin. 2009. “Approximate Bayesian Inference for Latent Gaussian Models by Using Integrated Nested Laplace Approximations.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 71 (2): 319–92. https://doi.org/10.1111/j.1467-9868.2008.00700.x.

Tierney, Luke, and Joseph B. Kadane. 1986. “Accurate Approximations for Posterior Moments and Marginal Densities.” Journal of the American Statistical Association 81 (393): 82–86. https://doi.org/10.1080/01621459.1986.10478240.

van Niekerk, Janet, and Håavard Rue. 2024. “Low-Rank Variational Bayes Correction to the Laplace Method.” Journal of Machine Learning Research 25 (62): 1–25. http://jmlr.org/papers/v25/21-1405.html.

Wood, Simon N. 2019. “Simplified Integrated Nested Laplace Approximation.” Biometrika 107 (1): 223–30. https://doi.org/10.1093/biomet/asz044.