Mismatch in speaker verification systems can originate from several sources. A mismatch occurs whenever certain conditions vary between the enrollment and verification sessions. It is well known that certain types of mismatch can lead to degraded performance (e.g., mismatched handsets in telephonebased speaker verification). Many studies conducted on several databases have reported on the impacts of mismatch and on various methods to reduce their effects to increase performance robustness. To our knowledge, this is one of the first comparative studies that tries to establish the relative impact of the different potential sources of mismatch. The experiments described in this paper have been constructed in a way that easily allows to isolate the impact of the different causes of mismatch on a single large database in very controlled conditions. We will compare the impact of lexical Mismatch with other types of mismatch on the true speaker and imposter populations. Our results show that total lexical mismatch induces an increase of the Equal Error Rate (EER) by a factor of 5 while the EER increases by a factor of 1.7 for channel mismatch and 1.4 for SNR mismatch.