We study two important generalizations of dynamic portfolio choice problems: a portfolio choice problem with market impact costs and a portfolio choice problem under the Hidden Markov Model. In the first problem, we allow the presence of market impact and illiquidity. Illiquidity and market impact refer to the situation where it may be costly or difficult to trade a desired quantity of assets over a desire period of time. In this work, we formulate a simple model of dynamic portfolio choice that incorporates liquidity effects. The resulting problem is a stochastic linear quadratic control problem where liquidity costs are modeled as a quadratic penalty on the trading rate. Though easily computable via Riccati equations, we also derive a multiple time scale asymptotic expansion of the value function and optimal trading rate in the regime of vanishing market impact costs. This expansion reveals an interesting but intuitive relationship between the optimal trading rate for the illiquid problem and the classical Merton model for dynamic portfolio selection in perfectly liquid markets. It also gives rise to the notion of a liquidity time scale. Furthermore, the solution to our illiquid portfolio problem shows promising performance and robustness properties. In the second problem, we study dynamic portfolio choice problems under regime switching market. We assume the market follows the Hidden Markov Model with unknown transition probabilities and unknown observation statistics. The main difficulty of this dynamic programming problem is its high-dimensional state variables. The joint probability density function of the hidden regimes and the unknown quantities is part of the state variables, and this makes the problem suffer from the curse of dimensionality. Though the problem cannot be solved by any standard fashions, we propose approximate methods that tractably solve the problem. The key is to approximate the value function by that of a simpler problem where the regime is not hidden and the parameters are observable (the C-problem). This approximation allows the optimal portfolio to be computed in a semi-explicit way. The approximate solution shares the same structure with the solution of C-problem, but at the same time it provides clear insight into the unobservable extension. In addition, the performance of the proposed methods is reasonably close to the upper-bound obtained from the information relaxation problem.