An office presence detection system is presented in this paper. Context information from multi-sensory inputs is integrated to infer a user’s activities in an office. We design a layered architecture to model human activities with different granularities. An IHDR (Incremental Hierarchical Discriminant Regression) tree is used to automatically generate models for acoustic signals from unsegmented auditory streams, with a high adaptive capability to new settings. Hidden Markov Models (HMM) are implemented to detect human motion patterns. The outputs of the above two components are fed into high-level HMMs to analyze human activities. Experimental results of the real-time prototype system are reported.