This paper presents a new RD optimal motion estimation and mode decision for fast H.264/AVC bit-rate reduction. In our proposed RD model, the prediction error is estimated according to the input motion information and the picture self property. This prediction error estimation is free from computing SSD/SAD. Along with the macroblock partition process, our proposed method significantly reduces the transcoding complexity. Experimental results show that our proposed method can improve the coding efficiency about 5 dB with similar speed, compared with the motion reuse transcoder.