Optimization for Bundle Adjustment

Motivation

Bundle Adjustment is a core optimization problem in 3D computer vision and robotics. It jointly estimates and refines camera poses and 3D point structures by minimizing reprojection error between projected points and image pixels.

Due to the nonlinear nature of the pinhole camera model, the problem is typically solved using Gauss–Newton and Levenberg–Marquardt (LM), which behave like second-order optimization methods without explicitly computing second derivatives.

---

The Bundle Adjustment Problem

Notation Reference Table

Symbol	Meaning	Dimension / Space
$C_i$	Pose of camera $i$ (rotation + translation)	$SE(3)$
$R_i$	Rotation matrix of camera $i$	$3 \times 3$, $SO(3)$
$t_i$	Translation vector of camera $i$	$\mathbb{R}^3$
$T_i$	Homogeneous camera pose matrix	$4 \times 4$, $SE(3)$
$X_j$	3D world point $j$	$\mathbb{R}^3$
$u_{ij}$	Observed 2D pixel of point $j$ in camera $i$	$\mathbb{R}^2$
$\pi(\cdot)$	Camera projection function	$\mathbb{R}^3 \to \mathbb{R}^2$
$r_{ij}$	Reprojection residual for observation $(i,j)$	$\mathbb{R}^2$
$\theta$	All optimization parameters (poses + points)	$\mathbb{R}^{6M + 3N}$
$\Delta \theta$	Parameter update vector	$\mathbb{R}^{6M + 3N}$
$J$	Jacobian of residuals w.r.t. parameters	Sparse matrix
$H$	Hessian of the cost function	$(6M + 3N) \times (6M + 3N)$
$J^T J$	Gauss–Newton Hessian approximation	Same as $H$
$\lambda$	Levenberg–Marquardt damping parameter	$\mathbb{R}^+$
$\xi$	Pose perturbation (twist coordinates)	$\mathbb{R}^6$
$\hat{\xi}$	Lie algebra matrix form of twist	$\mathfrak{se}(3)$
$\exp(\hat{\xi})$	Exponential map to update pose	$SE(3)$
$U, W, V$	Block Hessian matrices (camera, cross, point)	Block matrices
$S = U - W V^{-1} W^T$	Schur complement	$6M \times 6M$
$b$	Right-hand side of normal equations	Vector

---

Problem Setup

Given:

• A set of camera poses ${C_i}$
• A set of 3D points in world coordinates ${X_j}$
• 2D observation measurements $u_{ij}$

Lightbox2

---

PhotoSwipe

---

Spotlight JS

---

Venobox