Belief tracking is a promising technique for adding robustness to spoken dialog systems, but current research is fractured across different teams, techniques, and domains. This paper amplifies past informal discussions (Raux, 2011) to call for a belief tracking challenge task, based on the Spoken dialog challenge corpus (Black et al., 2011). Benefits, limitations, evaluation design issues, and next steps are presented.