A novel highly efficient macroblock-based progressive fine granularity scalable (MBPFGS) video coding scheme that can optimally balance drifting errors and coding efficiency is proposed in this paper. In order to improve coding efficiency, the existing PFGS video coding scheme uses a set of high quality references to predict the enhancement layers. However, potential drifting errors may arise since high quality references may not always be available at the decoder when channel bandwidth fluctuates. To solve this problem, we first analyze when drifting errors occur and how they will propagate. Then, an iterative model is established to estimate the drifting errors at the encoder. Meanwhile, three new INTER modes are proposed for the coding of macroblocks at the enhancement layers. One of these modes provides an effective and flexible method to reduce the drifting errors at low enhancement bit rates. Furthermore, we present a decision-making mechanism to select the proper coding mode for each macroblock at the enhancement layers. Experimental results show that the proposed techniques can dramatically reduce drifting errors at low enhancement bit rates, while significantly improving coding efficiency, up to 2.3dB in average PSNR, compared with MPEG-4 FGS at moderate or high bit rates.