Previously, human computer interaction (HCI) issues in ubiquitous or pervasive computing and virtual and augmented reality have been largely carried out in separate research communities. Given major advancements and increased maturity within the individual research areas, it may be fruitful to discuss combined approaches with a unifying perspective toward the next-generation interaction space; we initiated the Tangible Space Initiative (TSI) toward this goal, a multi-campus international research program funded by KIST for the next 10 years.. We have identified three major components that form mutually cooperative but distinct areas in TSI: tangible interface (TI), responsive cyberspace (RCS), and tangible agents (TA). TI mainly provides the interface between the human and the cyberspace. RCS is a virtual space that is responsive to user situation and intentions. In this talk, I will overview the TSI project and discuss our experience in building immersive virtual environments and current research interests in context-aware multimodal interaction issues in virtual and physical interaction environment both.