What Is Camera Calibration? A Scientific Introduction

A camera does not record pixels — it records light. Every pixel value in an image is the result of light reflecting off some 3D surface in the world, passing through a lens, and being measured by a sensor element. Camera calibration is the inverse problem: given the pixels, can we describe the geometric function $f : \mathbb{R}^3 \to \mathbb{R}^2$ that produced them?

The key to understanding that function is realizing it is not one transformation but a chain of four, passing through four different coordinate systems — each with its own origin. A point travels from the world frame, into the camera frame, onto the image plane, and finally into pixel coordinates. Throughout this article we use a strict convention: capital letters denote the axes of a coordinate system ($X_W$, $Y_W$, $Z_W$), while lowercase letters denote the coordinates of a specific point measured along those axes ($x_w$, $y_w$, $z_w$).

Step 1: the world frame. The world coordinate system is arbitrary — we choose it. In calibration it is typically anchored to the target: the origin $O_W$ sits at a corner of the checkerboard, with $X_W$ and $Y_W$ spanning the board plane and $Z_W$ perpendicular to it. A point $P$ in the scene has world coordinates $(x_w, y_w, z_w)$; for a point on the board itself, $z_w = 0$. Nothing about the camera has entered the picture yet — this frame describes the scene, not the sensor.

Step 2: the extrinsic transform. The camera has its own coordinate system: its origin is the optical center $C$ (the point all rays pass through), with $X_C$ pointing right, $Y_C$ pointing down, and $Z_C$ pointing forward along the optical axis into the scene. The extrinsic parameters — a rotation $R$ and a translation $\mathbf{t}$ — re-express the point in this frame: $$\begin{bmatrix} x_c \\ y_c \\ z_c \end{bmatrix} = R \begin{bmatrix} x_w \\ y_w \\ z_w \end{bmatrix} + \mathbf{t}$$ It is worth pausing on what this does and does not do. The point $P$ has not moved. Only the reference frame has changed: the same physical point now carries new coordinates $(x_c, y_c, z_c)$ — its position as seen from the camera. In particular, $z_c$ is the point's depth along the camera's viewing direction, a quantity that simply does not exist in the world frame.

Two properties of the extrinsics deserve precision, because they are the source of endless bugs. First, $R$ is not nine free numbers: it is an orthonormal rotation matrix ($R^\top R = I$, $\det R = +1$) with only three degrees of freedom, which is why libraries store it compactly as an axis–angle vector (OpenCV's rvec, expanded via the Rodrigues formula). Second — and this is the classic trap — $\mathbf{t}$ is not the camera's position in the world. Setting $(x_w, y_w, z_w) = (0,0,0)$ in the equation above shows that $\mathbf{t}$ is the world origin expressed in the camera frame. The camera's actual position in world coordinates is $C = -R^\top \mathbf{t}$. Report tvec as "camera position" and every downstream measurement will be silently wrong.

Step 3: the perspective divide. This is the step where a photograph actually happens. An ideal pinhole projects the point onto a plane at unit distance in front of $C$ by dividing out the depth: $$x_n = \frac{x_c}{z_c}, \qquad y_n = \frac{y_c}{z_c}$$ These normalized image coordinates are dimensionless and independent of any particular sensor. The divide is the only nonlinear operation in the entire chain, and it is where information is destroyed: 3D collapses to 2D, depth is gone, and "farther away" becomes "smaller" as a direct consequence of the arithmetic. It is also why a single pixel can never be inverted back to a 3D point — only to a ray of candidate points through $C$, all of which share the same $(x_n, y_n)$.

Step 4: the intrinsic matrix. So far no property of the physical camera has appeared — the normalized coordinates would be identical for any ideal pinhole. The intrinsics convert this idealized description into the actual pixel grid of a specific sensor: $$u = f_x\, x_n + c_x, \qquad v = f_y\, y_n + c_y$$ Here $f_x$ and $f_y$ are the focal lengths expressed in pixels — the lens focal length multiplied by the sensor's pixel density — which is why a "35 mm" lens produces $f_x$ values in the thousands. They control how many pixels one unit of normalized coordinate spans: the magnification. The principal point $(c_x, c_y)$ handles a bookkeeping fact: the projection is naturally centered on the optical axis, but images are indexed from the top-left corner. The offset $(c_x, c_y)$ is the pixel location where the optical axis pierces the sensor — near the image center, but never exactly, which is why it is estimated rather than assumed. Packed as a matrix, with a skew term $s$ that is effectively zero on modern sensors: $$K = \begin{bmatrix} f_x & s & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}$$

Chaining all four steps in homogeneous coordinates gives the full ideal pinhole model in a single line: $$z_c \begin{bmatrix} u \\ v \\ 1 \end{bmatrix} = K\, [\, R \mid \mathbf{t}\, ] \begin{bmatrix} x_w \\ y_w \\ z_w \\ 1 \end{bmatrix}$$ Note the $z_c$ on the left-hand side: that is the perspective divide from step three, hiding in plain sight. One computes the right-hand side and divides by the third component to obtain the pixel $(u, v)$. Counting parameters, the model has 11 degrees of freedom — 5 intrinsic ($f_x, f_y, c_x, c_y, s$) and 6 extrinsic (3 for $R$, 3 for $\mathbf{t}$).

One deliberate omission: real lenses are not ideal pinholes. Between the perspective divide and the intrinsic matrix, the lens bends the normalized coordinates — radial and tangential distortion — and this bending is nonlinear, so it cannot be absorbed into $K$. We treat it in detail in a later section; for the structure of the mapping, the ideal model above is the skeleton everything else hangs on.

Calibration, then, is the estimation problem this chain defines: recover the intrinsic parameters (and distortion coefficients) that are fixed properties of the camera, and — usually as a byproduct — the per-image extrinsics $[R \mid \mathbf{t}]$ describing where the camera stood for each frame used in the solve. Once the intrinsics are known, the chain can be run in both directions: undistorting images, projecting world points to pixels, triangulating from stereo pairs, and estimating pose from known features. None of those are possible without first answering: what camera am I dealing with?

The simplest geometric model of a camera is the pinhole: imagine all light passing through a single infinitesimal aperture at the camera centre $O$, then landing on a planar sensor at distance $f$ (the focal length) behind it. Using the virtual-image convention, where the sensor sits between $O$ and the world, similar triangles give the projection equations $$x' = f \cdot \frac{X}{Z}, \qquad y' = f \cdot \frac{Y}{Z}$$

Written in homogeneous coordinates and rolled together with the extrinsics, this gives the standard pinhole equation $$\mathbf{x} = K \, [\, R \mid \mathbf{t} \,] \, \mathbf{X}$$ where $\mathbf{X} = (X, Y, Z, 1)^\top$ is a homogeneous world point and $\mathbf{x} = (u, v, 1)^\top$ is its homogeneous pixel coordinate. The $3 \times 3$ intrinsic matrix $K$ does the work of converting metric image-plane coordinates into pixel coordinates.

In pixels rather than metres, the focal length splits into $f_x$ and $f_y$ (the metric focal length divided by the horizontal and vertical pixel pitch). The principal point $(c_x, c_y)$ is where the optical axis pierces the sensor — close to but rarely exactly at the image centre. Together, $f_x, f_y, c_x, c_y$ are the four intrinsic parameters of an ideal pinhole camera.

A real lens is not an infinitesimal pinhole. It is a stack of curved glass elements designed to gather more light and project a sharp image. That design introduces deviations from the pure pinhole projection — most prominently, radial distortion, where points appear shifted along the radial direction from the optical centre. Wide-angle and fisheye lenses bow lines outward (barrel distortion), while telephoto lenses can bow them inward (pincushion).

OpenCV models radial distortion with a polynomial in the radial distance $r$ from the principal point: $$r_d = r \cdot \bigl(1 + k_1 r^2 + k_2 r^4 + k_3 r^6\bigr)$$ Most reasonable lenses are captured by $k_1$ and $k_2$; $k_3$ is added for wide-angle. The rational model extends this with $k_4, k_5, k_6$ in a denominator polynomial for cameras with stronger distortion. Fisheye lenses use a different model entirely — the equidistant projection — because the polynomial approach breaks down near 180°.

There is also tangential distortion, parameterised by $p_1$ and $p_2$, which appears when the lens elements are not perfectly parallel to the sensor. It is usually small for modern cameras but matters for high-precision work. The full distortion vector $D = (k_1, k_2, p_1, p_2, k_3)$ is what calibration estimates alongside $K$.

Calibration is an optimisation. We capture $N$ images of a planar target whose 3D coordinates $\mathbf{X}_j$ are known by construction (a chessboard, ChArUco board, or circles grid). In each image $i$ we detect the projections $\mathbf{x}_{ij}$ of those known points. The unknowns are: the intrinsics $K$ and distortion $D$ (shared across all images), and a per-image rigid pose $(R_i, \mathbf{t}_i)$.

The goal is to find the parameters that make the predicted projections $\pi(K, D, R_i, \mathbf{t}_i, \mathbf{X}_j)$ match the observations $\mathbf{x}_{ij}$ as closely as possible, in the sense of squared pixel distance. Concretely: $$\min_{K, D, \{R_i, \mathbf{t}_i\}} \;\; \sum_{i, j} \bigl\lVert \mathbf{x}_{ij} - \pi(K, D, R_i, \mathbf{t}_i, \mathbf{X}_j) \bigr\rVert^2$$

This is a nonlinear least-squares problem because $\pi$ is nonlinear in the unknowns (the distortion polynomial is the obvious source of nonlinearity, but rotation is also). The standard solver is Levenberg–Marquardt, initialised from a closed-form solution that ignores distortion. The closed-form initialisation is Zhang's contribution from his 2000 paper: he showed that the homographies between a planar target and its images give linear constraints on $K$, after which distortion can be added back and the whole thing refined nonlinearly.

The standard quality metric is the root mean square reprojection error: $$e_{\text{rms}} = \sqrt{\, \frac{1}{N} \sum_{i} \bigl\lVert \mathbf{x}_i - \pi(\boldsymbol\theta, \mathbf{X}_i) \bigr\rVert^2 \,}$$ where $N$ is the total number of point observations and $\boldsymbol\theta$ is the full parameter vector. This is the same quantity the optimiser was minimising, evaluated at the converged solution. It has units of pixels.

What counts as "low"? It depends on the application. For typical machine-vision lenses on a moderate-resolution sensor, an RMS below $0.5\,\text{px}$ is excellent, $0.5$–$1.0\,\text{px}$ is good, and above $2\,\text{px}$ usually means weak frames, wrong board parameters, or a model mismatch (e.g. trying to fit a fisheye lens with a pinhole model). For sub-pixel measurement work in metrology, sub-$0.2\,\text{px}$ is the target.

The trap is that one number can hide a lot. A run can show a respectable average while one or two outlier frames carry most of the error, or while the error is concentrated near the edges of the sensor where the distortion model is failing. That is why analytics that show per-image error, sensor coverage, and residual heatmaps are essential — and why CalibrX surfaces all three.

In practice, "the pinhole model" is one of several models. Choosing the right one matters: a wide-angle lens fit with a strict pinhole will end up with a high RMS and ugly undistortion at the edges, while a normal lens fit as fisheye will produce parameters with strange physical meaning.

The three families most commonly used: the standard pinhole (Brown-Conrady distortion, OpenCV’s calibrateCamera) for moderate lenses up to roughly 90° field of view; the rational pinhole (additional k₄, k₅, k₆ in a denominator polynomial) for wide-angle lenses up to about 120°; and the equidistant fisheye model (Kannala–Brandt or OpenCV’s fisheye namespace) for true fisheye projections approaching 180°. CalibrX surfaces all three so you can solve the same captures against each and compare RMS and undistorted previews directly.

• Pinhole — standard rectilinear lenses, 4 to 8 distortion coefficients.
• Pinhole wide — rational distortion model for wide-angle and action cameras.
• Fisheye — equidistant projection (Kannala–Brandt) for dome and 180° lenses.

Once calibration has converged, K and D are the keys to every geometric operation the camera can support. Undistorting an image becomes a function call that takes pixels in and pixels out with straight lines preserved. Projecting a 3D world point becomes deterministic. Triangulating depth from a stereo pair requires both cameras’ intrinsics and the rectification homography that aligns their epipolar lines.

For applications that consume the calibration — robotics, drones, AR, 3D reconstruction, machine vision — the export needs more than the raw numbers. It needs the model identifier (pinhole, pinhole_wide, fisheye), the image size used during solving, and ideally the RMS and per-image quality metadata. That structured export is what CalibrX writes to JSON or YAML, and what the calibrx Python SDK reads back to drive undistortion locally with the correct model-specific routine.