Multiple-input multiple-output (MIMO) techniques have evolved as one of the key enabling technologies to address the ever-increasing demand for high-speed wireless data access. In this dissertation, we explore some important problems in point-to-point and multiuser MIMO systems. Antenna selection provides a new form of diversity in MIMO systems with low cost and relatively small overhead increase. In the first part, we investigate this technique, concerning (1) associated diversity property analysis in MIMO spatial multiplexing systems with practical transmitter/receiver; (2) fast algorithm design, especially in correlated fading channels; (3) practical implementation. The first two problems above are addressed via novel geometric tools, which are also extended to analyze some open problems in MIMO study, in particular the diversity-multiplexing tradeoff in V-BLAST and SDMA systems employing ordered successive interference cancellation (SIC). Finally we focus on the study of interference mitigation in multiuser multicell MIMO downlink, and investigate the potential of cooperative transmission among adjacent base stations (BS) for effectively mitigating co-channel interference. Our study starts with a quasi-synchronous model to obtain performance upper bounds, by which we also explore some other advantages like channel rank/conditioning improvement and macro-diversity protection. When considering a more practical channel model, in which the inter-cell interfering signals from different BS?s in the downlink are by nature asynchronous at each MS, we propose some novel and effective pre-coding algorithms achieving different levels of tradeoffs between interference mitigation and computational complexity. In summary, we have tackled some open and interesting problems in MIMO study, in particular, the diversity analysis for MIMO spatial multiplexing systems with antenna selection and practical coding and decoding schemes, and the impact of ordering on the performance of S.