This paper proposes an image compression approach, in which we incorporate primal sketch based learning into the mainstream image compression framework. The key idea of our approach is to use primal sketch information to enhance the quality of distorted images. With this method, we only encode the down-sampled image and use the primal sketch based learning to recover the high frequency information which has been removed by down-sampling. Experimental results demonstrate that our scheme achieves better objective visual quality as well as subjective quality compared with JPEG2000 at the same bit-rates.