MOMENT GENERATING FUNCTIONS

Moment generating functions, and their close relatives (probability generating functions and characteristic functions) provide an alternative way of representing a probability distribution by means of a certain function of a single variable. These functions turn out to be useful in many different ways:

(a) They provide an easy way of calculating the moments of a distribution.

(b) They provide some powerful tools for addressing certain counting and com binatorial problems.

(c) They provide an easy way of characterizing the distribution of the sum of independent random variables.

(d) They provide tools for dealing with the distribution of the sum of a random number of independent random variables.

(e) They play a central role in the study of branching processes.

(f) They play a key role in large deviations theory, that is, in studying the asymptotics of tail probabilities of the form P(X ≥ c), when c is a large number.

(g) They provide a bridge between complex analysis and probability, so that complex analysis methods can be brought to bear on probability problems.

(h) They provide powerful tools for proving limit theorems, such as laws of large numbers and the central limit theorem.

MOMENT GENERATING FUNCTIONS 

Definition 

The moment generating function associated with a random variable X is a function

  M_X( : R → [0,∞] defined by '

                  M_X((s) = E[e^sX]. 

The domain D_X of MX is defined as the  set D_X = {s | M_X(s) < ∞}.

If X is a discrete random variable, with PMF pX, then
                                          M_X(s) = ∑ e^sXpx(x)
If X is a continuous random variable with PDF fx, then
                                          M_X(s) =∫e^sXfx(x) dx.
Note 
that this is essentially the same as the definition of the Laplace transform of a function fx, except that we are using s instead of −s in the exponent.

The domain of the moment generating function

Note that 0 ∈  D_X, because M_X(0) = E[e^0X] = 1. For a discrete random variable that takes only a finite number of different values, we have  D_X = R. For example, if X takes the values 1 , 2 and 3, with probabilities 1/2, 1/3, and 1/6, respectively, then

                                      M_X(s) = 1/2 * e^s + 1/3 * e^2s + 1/6 * e^3s, '................................ (1)

which is finite for every s ∈ R. On the other hand, for the Cauchy distribution,

 fX(x) = 1/(π(1 + x²)), for all x, 

it is easily seen that M_X(s) = ∞, for all s = 0. 

In general, D_X is an interval (possibly infinite or semi-infinite) that contains zero. 

Exercise 1. 

Suppose that M_X(s) < ∞ for some s > 0. Show that M_X(t) < ∞ for all t ∈ [0, s]. Similarly, suppose that M_X(s) < ∞ for some s < 0. Show that M_X(t) < ∞ for all t ∈ [s, 0]

Exercise 2. 

Suppose that 

lim _x→∞^{log P(X > x)}⁄_x  ≜ −ν < 0. x→∞ x 

Establish that M_X(s) < ∞ for all s ∈ [0, ν).

Inversion of transforms 

By inspection of the formula for M_X(s) in Eq. (1), it is clear that the distribution of X is readily determined. The various powers e^sX indicate the possible values of the random variable X, and the associated coefficients provide the corresponding probabilities. At the other extreme, if we are told that M_X(s) = ∞ for every s = 0, this is certainly not enough information to determine the distribution of X. On this subject, there is the following fundamental result. It is intimately related to the inversion properties of Laplace transforms. Its proof requires sophisticated analytical machinery and is omitted. 

Theorem 1. 

Inversion theorem

 (a) Suppose that M_X(s) is finite for all s in an interval of the form [−a, a], where a is a positive number. Then, M_X determines uniquely the CDF of the random variable X. 

(b) If M_X(s) = M_Y (s) < ∞, for all s ∈ [−a, a], where a is a positive number, then the random variables X and Y have the same CDF. 

There are explicit formulas that allow us to recover the PMF or PDF of a random variable starting from the associated transform, but they are quite difficult to use (e.g., involving “contour integrals”). In practice, transforms are usually inverted by “pattern matching,” based on tables of known distribution-transform pairs.

Moment generating properties 

There is a reason why M_X is called a moment generating function. Let us consider the derivatives of M_X at zero. Assuming for a moment we can interchange 

the order of integration and differentiation, we obtain 

 dM_X(s)/ds |_s=0 = d E[e^sX] /ds|_s=0= E[Xe^sX] |_s=0= E[X], 

d^mM_X(s)/ds^m|_s=0  = d^m E[e^sX] /ds^m|_s=0  = E[X^m e^sX] |_s=0= E[X^m], 

Thus, knowledge of the transform M_X allows for an easy calculation of the moments of a random variable X. 

Justifying the interchange of the expectation and the differentiation does require some work. The steps are outlined in the following exercise. For simplicity, we restrict to the case of non negative random variables. 

Exercise 3.

 Suppose that X is a non negative random variable and that M_X(s) < ∞ for all s ∈ (−∞, a], where a is a positive number.

 (a) Show that E[X^k] < ∞, for every k. 

(b) Show that E[X^k e^sk] < ∞, for every s < a. 

(c) Show that (e^hX − 1)/h ≤ Xe^hX. 

(d) Use the DCT to argue that 

E[ E[X] = E [lim _h↓0 (e^hX − 1)/h] = lim _h↓0  (E[e^hX] -1) / h

The probability generating function 

For discrete random variables, the following probability generating function is sometimes useful. It is defined by
                         gx(s) = E[s^X],

 with s usually restricted to positive values. It is of course closely related to the moment generating function in that, for s > 0, we have gx(s) = M_X(log s). One difference is that when X is a positive random variable, we can define gx(s), as well as its derivatives, for s = 0. So, suppose that X has a PMF px(m), for m = 1, . . ..

                        Then,  gx(s) = ∑_(m→1,∞) s^mpx(m),

resulting in
                      d^m/ds^m  gx(s)| _s=0  = m! px(m).

(The interchange of the summation and the differentiation needs justification, but is indeed legitimate for small s.) Thus, we can use gx to easily recover the PMF px, when X is a positive integer random variable.

Properties of moment generating functions

 We record some useful properties of moment generating functions.

Theorem 2. 

(a) If Y = aX + b, then M_Y (s) = e^sbM_X(as).
(b) If X and Y are independent, then M_X+Y (s) = M_X(s)M_Y (s).
(c) Let X and Y be independent random variables. Let Z be equal to X, with probability p, and equal to Y , with probability 1 − p. Then, M_Z(s) = pM_X(s) + (1 − p)M_Y (s).

 Proof: 

For part (a), we have M_X(aX + b) = E[exp(saX + sb)] = exp(sb)E[exp(saX)] = exp(sb)M_X(as).
For part (b), we have M_X+Y (s) = E[exp(sX + sY )] = E[exp(sX)]E[exp(sY )] = M_X(s)M_Y (s).
For part (c), by conditioning on the random choice between X and Y , we have M_Z(s) = E[esZ] = pE[esX] + (1 − p)E[esY ] = pM_X(s) + (1 − p)M_Y (s).

Example : (Normal random variables)

 (a) Let X be a standard normal random variable, and let Y = σX +µ, which we know to have a N(µ, σ2) distribution. We then find that MY (s) = exp(sµ + 1 2 s2σ2).

(b) Let X = d N(µ1, σ1 2) and Y = N(µ2, σ2 2). Then,
                                         M_X+Y (s)= exp( s(µ1 + µ2) + 2 s2(σ1 2 + σ2 2) .
 Using the inversion property of transforms, we conclude that X + Y = d N(µ1 + µ2, σ1 2 + σ2 2), thus corroborating a result we first obtained using convolutions.