FxpNet: Training deep convolutional neural network in fixed-point representation

We introduce FxpNet, a framework to train deep convolutional neural networks with low bit-width arithmetics in both forward pass and backward pass. During training procedure, FxpNet further reduces the bit-width of stored parameters (also known as primal parameters) by adaptively updating their fixed-point formats. These primal parameters are usually represented in full resolution of floating-point values in previous binarized and quantized neural networks. In FxpNet, during forwarding pass, fixed-point primal weights and activations will first be binarized before computation, while in backward pass all gradients are represented as low-resolution fixed-point values and then accumulated to corresponding fixed-point primal parameters. To have highly efficient implementations in FPGAs, ASICs and other dedicated devices, FxpNet introduces Integer Batch Normalization (IBN) and Fixed-point ADAM (FxpADAM) methods to further reduce the required floating-point operations, which will save considerable power and chip area. The evaluation on CIFAR-10 dataset indicates the effectiveness that FxpNetwith 12-bit primal parameters and 12-bit gradients achieve comparable prediction accuracy with state-of-art binarized and quantized neural networks. The corresponding training code implemented in TensorFlow is available online.