We propose that the task of question generation should incorporate not only measures of grammaticality, but also a measure of the importance of a question automatically generat-ed. Necessarily, importance of a given question can be judged only in context, so we propose that the data for a shared task be larger than a single sentence, data point or statement in a knowledge base. By focusing on the importance of questions, the community will be in a position to address the shortcomings of existing systems that employ question generation.