We present a novel scheme for dynamically recovering a background image from consecutive frames of a video sequence based on spatial and temporal continuities. The proposed algorithm applies a boundary-level spatial continuity constraint in order to detect and correct ghosting, which corresponds to incorrectly classified foreground regions due to fast moving objects. The proposed method can be applied successfully to sequences with deformable foreground objects and non-uniform motion. Simulation results show that the extracted background, when used for foreground detection, results in a higher performance in terms of recall and precision as compared to existing popular schemes.