One of the enduring challenges facing HIV vaccine design is the remarkable rate of viral mutation and adaptation that limits the ability of the immune system to mount a lasting effective response. This rapid rate of mutation leads to extensive within- and between-host viral diversity that makes creation of a broadly reactive vaccine difficult. A first step in overcoming this challenge is to identify consistent patterns in viral adaptation. Recently, several studies have analyzed large groups of HIV-infected individuals and looked for correlations between HIV polymorphisms and the HLA class I alleles that restrict the cellular immune response. Here, we point out a limitation of previous approaches: correlations among HLA alleles and HIV codons lead to statistical confounding if not taken into consideration. In response, we develop two statistical models of evolution that explicitly represent stochastic selection pressure from multiple sources. After validating these models on synthetic data, we analyze the patterns of immune escape in a multicenter cohort of over 1000 individuals. Our results identify a dense network of interactions between HLA alleles and HIV codons, as well as among HIV codons, reflecting both a complexity and a promising consistency in the way that HIV adapts to the human immune response.