A central theme in applied and computational statistics is the accurate and efficient methods of inference. The Bayesian paradigm performs inference based on the posterior distribution of unknown quantities. Throughout decades, there has been an enormous literature on computational Bayesian methods. Practical implementations, while succussful to different degrees, usually impose certain restrictions on the specific model structure. As more applications rely on complex model dynamics, more challenges remain to tackle the curse of high dimensionality and the analytical intractability of many non-Gaussian distributions. This thesis builds on existing research in the field of sequential Bayesian estimation for a general class of state-space models. We establish recursive Bayesian simulation algorithms to estimate parameters and states for a variety of diffusion and jump stochastic models. Our main work and contribution are two-fold. First, we build a particle filter framework for Levy-type state-space models. Particle filters are efficient numerical simulation techniques ideally suitable for highly nonlinear models, with a significant computational advantage over the standard Markov Chain Monte Carlo. Our particle filters can effectively estimate parameters and state variables for non-Gaussian dynamics. We perform empirical testing on financial time series, and find that certain Levy-type small jump processes can be a substitute of the usual Brownian motion-based random walk models. In addition, we propose a general Variational Bayes Particle Filter framework. It is applicable to a wider class of models with a large number of dimensions. Secondly, we build a Variational Bayes estimator for Hidden Markov Models with observational jumps. This is a typical setup for numerous biostatistical data analysis, where huge amounts of streaming data need to be sequentially filtered for potential evidence of the existence of quantitative traits or genetic features. Our algorithm works to identify and classify different responses. The hidden Markov estimator is robust and highly adaptable. In addition, this thesis also includes a self-contained chapter on the technique of Markovian projection. It reduces a complicated multi-dimensional dynamics to a one-dimensional simple Markovian process with identical marginal distributions, therefore keeping certain path-independent expectation values invariant. The projection has certain implications in the pricing of European-style options in financial mathematics. We provide a theorem generalizing existing results to the general Levy jump models, and discuss calibration issues.