This chapter discusses the use of regression forests for the automatic detection and simultaneous localization of multiple anatomical regions within Computed Tomography (CT) and Magnetic Resonance (MR) three-dimensional images. Important applications include: organ-specific tracking of radiation dose over time; selective retrieval of patient images from radiological database systems; semantic visual navigation; and the initialization of organ-specific image processing operations. We present a continuous parametrization of the anatomy localization problem, which allows it to be addressed effectively by multivariate random regression forests. A single pass of our probabilistic algorithm enables the direct mapping from voxels to organ location and size, with training focusing on maximizing the confidence of output predictions. As a by-product, our method produces salient anatomical landmarks, i.e. automatically selected “anchor” regions which help localize organs of interest with high confidence. This chapter builds upon the work in [80, 277] and demonstrates the flexibility of forests in dealing with both CT or multi-channel MR images. Quantitative validation is performed on two groundtruth labelled databases: i) a database of 400 highly variable CT scans, and ii) a database of 33 full-body, multi-channel MR scans. In both cases localization errors are shown to be lower and more stable than those from more conventional atlas-based registration approaches. The simplicity of the regressor’s context-rich visual features yield typical run-times of only 4 seconds per volume. This anatomy recognition algorithm is now part of the commercial product Microsoft Amalga Unified Intelligence System.