Markov Chain

Markov Chain is a stochastic process that moves between a set of states, where the probability of moving to the next state depends only on the current state (the Markov Property). It's described by the set of states and a transition matrix of probabilities between them.

Probabilistic counterpart of Finite Automaton.

A Markov Decision Process (MDP) extends a Markov chain with actions and rewards. See also Markov Chain Sampling.