We introduce a powerful non-parametric image labelling framework, Regression Tree Fields (RTFs), and discuss its application to image restoration. The conditional structure and the parameters of our model are estimated from training data so as to directly optimize for popular performance measures, resulting in excellent predictive performance at low computational cost.