Visions of smart homes have long caught the attention of researchers and considerable effort has been put toward enabling home automation. However, these technologies have not been widely adopted despite being available for over three decades. To gain insight into this state of affairs, we conducted semi-structured home visits to 14 households with home automation. The long term experience, both positive and negative, of the households we interviewed illustrates four barriers that need to be addressed before home automation becomes amenable to broader adoption. These barriers are high cost of ownership, inflexibility, poor manageability, and difficulty achieving security. Our findings also provide several directions for further research, which include eliminating the need for structural changes for installing home automation, providing users with simple security primitives that they can confidently configure, and enabling composition of home devices.