This Title All WIREs
How to cite this WIREs title:
WIREs Comp Stat

# Splines, knots, and penalties

Can't access this content? Tell your librarian.

Penalized splines have gained much popularity as a flexible tool for smoothing and semi‐parametric models. Two approaches have been advocated: (1) use a B‐spline basis, equally spaced knots, and difference penalties [Eilers PHC, Marx BD. Flexible smoothing using B‐splines and penalized likelihood (with Comments and Rejoinder). Stat Sci 1996, 11:89–121.] and (2) use truncated power functions, knots based on quantiles of the independent variable and a ridge penalty [Ruppert D, Wand MP, Carroll RJ. Semiparametric Regression. New York: Cambridge University Press; 2003]. We compare the two approaches on many aspects: numerical stability, quality of the fit, interpolation/extrapolation, derivative estimation, visual presentation and extension to multidimensional smoothing. We discuss mixed model and Bayesian parallels to penalized regression. We conclude that B‐splines with difference penalties are clearly to be preferred. WIREs Comp Stat 2010 2 637–653 DOI: 10.1002/wics.125

Figure 1.

Truncated power function bases with equally spaced knots. (a) Linear, (b) cubic.

[ Normal View | Magnified View ]
Figure 2.

B‐spline bases with equally spaced knots. (a) Linear, (b) quadratic.

[ Normal View | Magnified View ]
Figure 3.

Logarithm of the absolute value of one cubic B‐spline, computed from the fourth difference of cubic truncated polynomials. The B‐splines has been scaled to a maximum of 1.

[ Normal View | Magnified View ]
Figure 4.

Components of a fit with 18 cubic B‐splines and a second‐order penalty to simulated data (squares). The encircled dots show the coefficients of the B‐splines. (a) λ = 0.01, (b) λ = 10.

[ Normal View | Magnified View ]
Figure 5.

Components of a fit with 15 (κ = 0.1) linear truncated power functions to simulated data.

[ Normal View | Magnified View ]
Figure 6.

Smoothing and interpolation of simulated data with a large basis of cubic B‐splines and a second‐order penalty (λ = 10). The scaled B‐splines are shown on the bottom of the graph. Their sum gives the full line, which is the fitted curve. The encircled dots represent the value of the B‐spline coefficients.

[ Normal View | Magnified View ]
Figure 7.

Smoothing and interpolation of simulated data with a basis of linear truncated power functions, with 100 equally spaced knots. Ridge penalty with κ = 0.1.

[ Normal View | Magnified View ]
Figure 8.

Smoothing and interpolation of simulated data with a basis of linear truncated power functions, with knots at unique values of x. Ridge penalty with κ = 0.1.

[ Normal View | Magnified View ]
Figure 9.

An illustration of optimal smoothing and interpolation with many B‐splines and a large gap. The upper left panel shows the numerical condition, and the lower left panel the leave‐one‐out cross‐validation profile. The panels show results of smoothing for the approximately optimal λ and for small λ, the latter showing overshoot. The number of cubic B‐splines is 53 and the order of the penalty is 2.

[ Normal View | Magnified View ]
Figure 10.

Condition numbers of three types of bases: truncated power functions (crosses), B‐splines (squares) and Z‐matrices for mixed models (diamonds). The sample sizes are 100, 200, 500, or 1000, but this is difficult to see, because the lines and symbols overlap strongly.

[ Normal View | Magnified View ]
Figure 11.

Automatic choice of the smoothing parameter with the hybrid algorithm for variance estimation. Simulated data. Broken line: true curve; full line: automatically estimated smooth.

[ Normal View | Magnified View ]
Figure 12.

A standard B‐spline basis (top) and the corresponding wrapped basis.

[ Normal View | Magnified View ]
Figure 13.

Smoothing of simulated data (dots) with and without exponentially varying weights on the differences in the penalty. (a) Uniform weights; (b) varying weights. Parameters optimized with grid search and leave‐one‐out cross validation. Full line: fitted curve (100 cubic B‐splines, second‐order penalty); broken line: true curve.

[ Normal View | Magnified View ]
Figure 14.

(a) Impulse response of a P‐spline smoother with a only a first or second‐order difference penalty; (b) impulse response of a P‐spline smoother with second and first‐order penalties: \documentclass{article}\usepackage{amsmath}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{amsfonts}\pagestyle{empty}\begin{document} ${\rm {pen}} = \lambda ||D_{2}\alpha||^{2} + 2 \sqrt{\lambda} ||D_{1}\alpha||^{2}$\end{document}.

[ Normal View | Magnified View ]
Figure 15.

Interpolation and extrapolation with a combination of penalties of first and second order: \documentclass{article}\usepackage{amsmath}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{amsfonts}\pagestyle{empty}\begin{document} ${\rm {pen}} = \lambda ||D_{2}\alpha||^{2} + \gamma \sqrt{\lambda} ||D_{1}\alpha||^{2}$\end{document}. The values of γ are 0, 0.01, 0.02, 0.05, and 0.1, A larger γ, gives a tighter curve.

[ Normal View | Magnified View ]

### Browse by Topic

Computational Intensive Statistical Methods > Density Estimation and Curve Fitting
Statistical Methods > Mathematical Methods