Automatic human face detection from images in surveillance and biometric applications is a challenging task due to the variances in image background, view, illumination, articulation, and facial expression. In this paper, we propose a novel three-step face detection approach to addressing this problem. The approach adopts a simple-to-complex strategy. First, a linear-filtering algorithm is applied to enhance detection performance by remove most non-face-like candidate rapidly. Second, a boosting chain algorithm is adopted to combine the boosting classifiers into a hierarchy “chain” structure. By utilizing the inter-layer discriminative information, this algorithm reveals higher efficiency than the original cascade approaches . Last, a post-filtering algorithm consists of image pre-processing; SVM-filter and color-filter are applied to refine the final prediction. As only small amount of candidate windows remain in the final stage, this algorithm greatly improves the detection accuracy with small computation cost. Compared with conventional approaches, this three-step approach is shown to be more effective and capable of handling more pose variations. Moreover, together with a two-level hierarchy in-plane pose estimator, a rapid multi-view face detector is therefore built. The experimental results demonstrate the significant performance improvement using the proposed approach over others.