Many major cities are suffering from noise pollution, which compromises people’s working efficiency and even mental health. New York City (NYC) has opened a platform titled 311 to allow people to complain the imperfect of the city by using a mobile app or making a phone call; noise is the third largest category of complaints in the 311 data. As each complaint about noises is associated with a location, time stamp, and a fine-grained noise category, such as loud music or construction noises, the data is actually a result of “human as a sensor” and “crowd sensing”, containing rich human intelligence that can help diagnose urban noises. In this paper we infer the fine-grained noise situation (consisting of a noise pollution indicator and the composition of noises) of different time of day for each region of NYC, by using the 311 complaint data together with social media, road network data, and Points of Interests (POIs). We model the noise situation of NYC with a three dimension tensor, where the three dimensions stand for regions, noise categories, and time slots, respectively. By filling in the missing entries of the tensor through a context-aware tensor decomposition approach, we recover the noise situation throughout NYC. The information of noise can inform people and officials’ decision making. We evaluated our method with four real datasets, demonstrating the advantages of our method beyond four baselines, such as interpolation-based approach.