This book develops systematically and rigorously, yet in an expository and lively manner, the evolution of general random processes and their large time properties such as transience, recurrence, and convergence to steady states. The emphasis is on the most important classes of these processes from the viewpoint of theory as well as applications, namely, Markov processes. The book features very broad coverage of the most applicable aspects of stochastic processes, including sufficient material for self-contained courses on random walk in one and multiple dimensions; Markov chains in discrete and continuous times, including birth-death processes; Brownian motion and diffusions; stochastic optimization; and stochastic differential equations. Audience: this book can be used for a number of different courses for graduate students of mathematics, statistics, economics, engineering, and other fields who have some background in probability and analysis. It is also intended as a reference for researchers and professionals in many areas of science and technology whose work involves the application of probability.