Sums of independent random variables

This lecture discusses how to derive the distribution of the sum of two independent random variables. We explain first how to derive the distribution function of the sum and then how to derive its probability mass function (if the commands are discrete) or its probability density function (if the commands are continuous).

Distribution function of a sum

The following proposition characterizes the distribution function of the sum in terms of the distribution functions of the two commands.

Proposition Let  and  be two independent random variables and denote by  and  their distribution functions. Let

                                             z=x+y

and denote the distribution function of z by F_z(z) The following holds:

                                              F_z(z) =E[F_x(z-X) ]

or                                           F_z(z) =E[F_y(z-Y) ]

Proof

Example Let X be a uniform random variable with support R_x=[0 , 1] and probability density function

                                    1 if x ϵ R_x

          F_x(x)={

                                     0    otherwise

and Y another uniform random variable, independent of X with support  R_x=[0 , 1] and probability density function

                                   1 if y ϵ R_y

          F_y(y)={

                                   0    otherwise

The distribution function of X is

                                                         1 if x ≤ 0

          F_x(x)=∫_lim(x,-∞) ={        x if 0≤1

                                                                               1 if x>1

The distribution function of Z=X+Y is

                  F_z(Z)=E[F_x(z-Y)

              =∫_lim(x,-∞)F_x(z-y)f_{y(y)dy
              =∫lim(1,0)Fx(z-y)dy
              =∫lim(z , z-1)Fx(z-y)tdt           (by a change of variable t=z-y)}
              =∫_lim(z-1,z)F_X(t)_{dt                  (exchanging the bounds of integration)}

There are four cases to consider:

If z ≤ 0 then

                       F_z(Z)=∫_{lim(z-1 , z)}F_x(t)_dt

                              _{=∫lim(z-1 , z)0 dt}

                              ₌₀

If 0 < z ≤ 1, then

                       F_z(Z)=∫_{lim(z-1 , z)}F_x(t)_dt

                                     =∫_{lim(z-1 , 0)}F_x(t)_dt + ∫_{lim(0 , z)}F_x(t)_dt

                                      =∫_{lim(z-1 , 0)0 dt +} ∫_{lim(0 , z)}t dt

                                      =_{0 + [(1/2)t}2_{] = (1/2)z}2

If ,1< z ≤ 2 then

                       F_z(Z)=∫_{lim(z-1 , z)}F_x(t)_dt

                             _{=∫lim(z-1 , 1)Fx(t)dt  +} ∫_{lim(1 , z)}F_x(t)_dt

                             _{=∫lim(z-1 , 1)tdt  +} ∫_{lim(1 , z)1dt}

                             _=(1/2)-(z-1)2+z-1

                             _=1/2-(1/2)z2+(1/2)2z-1/2+z-1

                     =(1/2)z2+2z-1

1. If z>2, then

                       F_z(Z)=∫_{lim(z-1 , z)}F_x(t)_dt

                       _{Fz(Z)=∫lim(z-1 , z)1dt}

                              _=z-(z-1)=1

Probability Mass Function (PMF)

If X$X$ is a discrete random variable then its range RX$R_X$ is a countable set, so, we can list the elements in RX$R_X$. In other words, we can write

RX={x1,x2,x3,...}.$$R_X=\{x_1,x_2,x_3,...\}.$$

Note that here x1,x2,x3,...$x_1,x_2,x_3,...$ are possible values of the random variable X$X$. While random variables are usually denoted by capital letters, to represent the numbers in the range we usually use lowercase letters such as x$x$, x1$x_1$, y$y$, z$z$, etc. For a discrete random variable X$X$, we are interested in knowing the probabilities of X=xk$X=x_k$. Note that here, the event A={X=xk}$A=\{X=x_k\}$ is defined as the set of outcomes s$s$ in the sample space S$S$ for which the corresponding value of X$X$ is equal to xk$x_k$. In particular,

A={s∈S|X(s)=xk}.$$A=\{s\in S|X(s)=x_k\}.$$

The probabilities of events {X=xk}$\{X=x_k\}$ are formally shown by the probability mass function (pmf) of X$X$.
Definition 
Let X$X$ be a discrete random variable with range RX={x1,x2,x3,...}$R_X=\{x_1,x_2,x_3,...\}$ (finite or countably infinite). The function

PX(xk)=P(X=xk), for k=1,2,3,...,$$P_X(x_k)=P(X=x_k), for k=1,2,3,...,$$

is called the probability mass function (PMF) of X$X$.

Thus, the PMF is a probability measure that gives us probabilities of the possible values for a random variable. While the above notation is the standard notation for the PMF of X$X$, it might look confusing at first. The subscript X$X$ here indicates that this is the PMF of the random variable X$X$. Thus, for example, PX(1)$P_X(1)$ shows the probability that X=1$X=1$. To better understand all of the above concepts, let's look at some examples.

---

Example 

I toss a fair coin twice, and let X$X$ be defined as the number of heads I observe. Find the range of X$X$, RX$R_X$, as well as its probability mass function PX$P_X$.

Solution: 

Here, our sample space is given by

S={HH,HT,TH,TT}.$$S=\{HH,HT,TH,TT\}.$$

The number of heads will be 0$0$, 1$1$ or 2$2$. Thus

RX={0,1,2}.$$R_X=\{0,1,2\}.$$

Since this is a finite (and thus a countable) set, the random variable X$X$ is a discrete random variable. Next, we need to find PMF of X$X$. The PMF is defined as

PX(k)=P(X=k) for k=0,1,2.$$P_X(k)=P(X=k) for k=0,1,2.$$

We have

PX(0)=P(X=0)=P(TT)=14,$$P_X(0)=P(X=0)=P(TT)=\frac{1}{4},$$

PX(1)=P(X=1)=P({HT,TH})=14+14=12,$$P_X(1)=P(X=1)=P(\{HT,TH\})=\frac{1}{4}+\frac{1}{4}=\frac{1}{2},$$

eriment (tossing a coin twice) a large number of times, then about half of the times we observe X=1X=1, about a quarter of times we observe X=0X=0, and about a quarter of times we observe X=2X=2.

Although the PMF is usually defined for values in the range, it is sometimes convenient to extend the PMF of X$X$ to all real numbers. If x∉RX$x\notin R_X$, we can simply write PX(x)=P(X=x)=0$P_X(x)=P(X=x)=0$. Thus, in general we can write

PX(x)={P(X=x)0if x is in RXotherwise$$P_X(x)=\{\begin{array}{ll}P(X=x)& \text{if }x\text{ is in }{R}_X\\ 0& \text{otherwise}\end{array}$$

To better visualize the PMF, we can plot it. Figure 3.1 shows the PMF of the above random variable X$X$. As we see, the random variable can take three possible values 0,1$0,1$ and 2$2$. The figure also clearly indicates that the event X=1$X=1$ is twice as likely as the other two possible values. The Figure can be interpreted in the following way: If we repeat the random experiment (tossing a coin twice) a large number of times, then about half of the times we observe X=1$X=1$, about a quarter of times we observe X=0$X=0$, and about a quarter of times we observe X=2$X=2$.

Example 

I have an unfair coin for which P(H)=p$P(H)=p$, where 0<p<1$0<p<1$. I toss the coin repeatedly until I observe a heads for the first time. Let Y$Y$ be the total number of coin tosses. Find the distribution of Y$Y$.

• • First, we note that the random variable Y$Y$ can potentially take any positive integer, so we have RY=N={1,2,3,...}$R_Y=\mathbb{N}=\{1,2,3,...\}$. To find the distribution of Y$Y$, we need to find PY(k)=P(Y=k)$P_Y(k)=P(Y=k)$ for k=1,2,3,...$k=1,2,3,...$. We have
    PY(1)=P(Y=1)=P(H)=p,$$P_Y(1)=P(Y=1)=P(H)=p,$$
    PY(2)=P(Y=2)=P(TH)=(1−p)p,$$P_Y(2)=P(Y=2)=P(TH)=(1-p)p,$$
    PY(3)=P(Y=3)=P(TTH)=(1−p)2p,$$P_Y(3)=P(Y=3)=P(TTH)=(1-p{)}^{2}p,$$
    ....$$....$$
    ....$$....$$
    ....$$....$$
    PY(k)=P(Y=k)=P(TT...TH)=(1−p)k−1p.$$P_Y(k)=P(Y=k)=P(TT...TH)=(1-p{)}^{k-1}p.$$
    Thus, we can write the PMF of Y$Y$ in the following way
    PY(y)={(1−p)y−1p0for y=1,2,3,...otherwise$$P_Y(y)=\{\begin{array}{ll}(1-p{)}^{y-1}p& \text{for }y=1,2,3,...\\ 0& \text{otherwise}\end{array}$$
  • 
    

Consider a discrete random variable X$X$ with Range(X)=RX$(X)=R_X$. Note that by definition the PMF is a probability measure, so it satisfies all properties of a probability measure. In particular, we have

• 0≤PX(x)≤1$0\le P_X(x)\le 1$ for all x$x$, and
• ∑x∈RXPX(x)=1$\sum _{x\in R_X}{P}_X(x)=1$.

Also note that for any set A⊂RX$A\subset R_X$, we can find the probability that X∈A$X\in A$ using the PMF

P(X∈A)=∑x∈APX(x).$$P(X\in A)=\sum _{x\in A}{P}_X(x).$$

Properties of PMF:

• 0≤PX(x)≤1$0\le P_X(x)\le 1$ for all x$x$;
• ∑x∈RXPX(x)=1$\sum _{x\in R_X}{P}_X(x)=1$;
• for any set A⊂RX,P(X∈A)=∑x∈APX(x)$A\subset R_X,P(X\in A)=\sum _{x\in A}{P}_X(x)$.

---

Example 

For the random variable Y$Y$ in Example 3.4,

1. Check that ∑x∈RYPY(y)=1$\sum _{x\in R_Y}{P}_Y(y)=1$.
2. If p=12$p=\frac{1}{2}$, find P(2≤Y<5)$(2\le Y<5)$.

In Example 3, we obtained

PY(k)=P(Y=k)=(1−p)k−1p, for k=1,2,3,...$$P_Y(k)=P(Y=k)=(1-p{)}^{k-1}p, for k=1,2,3,...$$

Thus,

1. to check that ∑y∈RYPY(y)=1$\sum _{y\in R_Y}{P}_Y(y)=1$, we have
   
   

   ∑y∈RYPY(y)$\sum _{y\in R_Y}{P}_Y(y)$	=∑∞k=1(1−p)k−1p$=\sum _{k=1}^{\mathrm{\infty }}(1-p{)}^{k-1}p$
   	=p∑∞j=0(1−p)j$=p\sum _{j=0}^{\mathrm{\infty }}(1-p{)}^j$
   	=p11−(1−p)$=p\frac{1}{1-(1-p)}$	Geometric sum$\text{ Geometric sum}$
   	=1$=1$;

2. if p=12$p=\frac{1}{2}$, to find P(2≤Y<5)$(2\le Y<5)$, we can write
   
   

   P(2≤Y<5)$\text{P}(2\le Y<5)$	=∑4k=2PY(k)$=\sum _{k=2}^4{P}_Y(k)$
   	=∑4k=2(1−p)k−1p$=\sum _{k=2}^4(1-p{)}^{k-1}p$
   	=12(12+14+18)$=\frac{1}{2}(\frac{1}{2}+\frac{1}{4}+\frac{1}{8})$
   	=716$=\frac{7}{16}$.