To introduce the method of maximum likelihood, consider a very simple estimation
problem. Suppose that an urn contains a number of black and a number
of white balls, and suppose that it is known that the ratio of the numbers is
3/1 but that it is not known whether the black or the white balls are more
numerous. That is, the probability of drawing a black ball is either * or i.
If n balls are drawn with replacement from the urn, the distribution of x, the
number of black balls, is given by the binomial distribution
\[f(x;p)=(_{x}^{n}){{p}^{x}}{{q}^{n-x}}\]
where q = 1 - p and p is the probability of drawing a black ball. Here p = 1/4, or p = 3/4 We shall draw a sample of three balls, that is, n = 3, with replacement and
attempt to estimate the unknown parameter p of the distribution. The estimation problem is particularly simple in this case because we have only to choose between the two numbers .25 and .75. Let us anticipate the results of the drawing of the sample. The possible outcomes and their probabilities are given below:
In the present example, if we found x = 0 in a sample of 3, the estimate .25 for
p would be preferred over .75 because the probability it is greater than 6
1
4'
. i.e., because a sample with x = 0 is more likely (in the sense of having larger
probability) to arise from a population with p ! than from one with p i.
And in general we should estimate p by .25 when x = 0 or 1 and by .75 when
x = 2 or 3. The estimator may be defined as
\[\hat{p}=\hat{p}(x)=\{_{.75}^{.25_{for,x=2,3}^{for,x=0,1}}\]
The estimator thus selects for every possible x the value of p, say ,\[\hat{p}\] such that
f(x;\[\hat{p}\]) > f(x;p')
where p' is the alternative value of p. More generally, if several alternative values of p were possible, we might reasonably proceed in the same manner. Thus if we found x = 6 in a sample of 25 from a binomial population, we should substitute all possible values of p in the expression
\[f(6;p)=(_{6}^{25}){{p}^{6}}{{q}^{25-6}}\]
anti choose as our estimate that value of p which maximized f(6; p). For the given possible values of p we should find our estimate to be 265- The position of its maximum value can be found by putting the derivative of the function defined in Eq. (2) with respect to p equal to 0 and solving the resulting equation for p. Thus
\[\frac{d}{dp}f(6;p)=(_{6}^{25}){{p}^{5}}{{\left( 1-p \right)}^{18}}\left[ 6\left( 1-p \right)-19p \right]\]
and on putting this equal to 0 and solving for p, we find that p = 0, 1, 6/25 are the roots. The first two roots give a minimum, and so our estimate is therefore p = 6/25' This estimate has the property that
f(6;\[\hat{p}\])>f(6;p')
where pi is any other value of p in the interval 0 ≤ p < 1<1 .="" define="" estimators="" first="" function.="" in="" likelihood="" maximum-likelihood="" nbsp="" order="" p="" shall="" the="" to="" we="">
The likelihood function L(,θ; x1 ' ••• , xn) gives the likelihood that the random variables assume a particular value x1, x2 ... , xn. The likelihood is the value of a density function; so for discrete random variables it is a probability. Suppose for a moment that ,θ is known; denote the value by ,θ0 , The particular value of the random variables which is "most likely to occur" is that value x'1, x'2 ... , x'n such that f x1, ... , xn (X1 , ••• , Xn ; ,θ0) is a maximum. For example, for simplicity let us assume that n = 1 and X1 has the normal density with mean 6 and variance 1. Then the value of the random variable which is most likely to occur is X1 = 6. By" most likely to occur" we mean the value x'1 of X1 such that ∅6,1(x'1) > ∅6,1(x1)' Now let us suppose that the joint density of n random variables is fx1, ...• xn(X1, ••• , Xn;,θ), where 8 is unknown. Let the particular values which are observed be represented by x1' , x2' ., ., xn' . We want to know from which density is this particular set of values most likely to have come. We want to know from which density (what value of 0) is the likelihood largest that the set x1' ..., ., xn' .was obtained. In other words, we want to find the value of 0 in 9, denoted by {\[\hat{,θ}\], which maximizes the likelihood function L(,θ; x1' ..., ., xn' ). The value {J which maximizes the likelihood function is, in general, a function of x1 ••• , xn, say\[\hat{,θ}\] =\[\hat{,θ}\] (x1, x2 ... , xn) When this is the case, the random variable \[\hat{,θ}\] =,\[\hat{θ}\] (X1 , ••• , Xn) is called the maximum likelihood estimator of ,θ. (We are assuming throughout that the maximum of the likelihood function exists.) We shall now formalize the definition of a maximum-likelihood estimator
\[\prod\limits_{i=1}^{n}{f\left( {{x}_{i}};{{\theta }_{1}},{{\theta }_{2}},...............{{\theta }_{k}} \right)}\]
then the maximum-likelihood estimators of the parameters ,θ1, ,θ2, ... , ,θk are the random variables,\[{{\hat{\theta }}_{1}}\] = ,\[{{\hat{\theta }}_{1}}\](X1 , X2 , .......Xn), ,\[{{\hat{\theta }}_{2}}\] = ,\[{{\hat{\theta }}_{2}}\](X1 , X2 , .......Xn ) ... , ,\[{{\hat{\theta }}_{k}}\]= ,\[{{\hat{\theta }}_{k}}\](X1 , X2 , .......Xn)' where ,\[{{\hat{\theta }}_{1}}\], ,\[{{\hat{\theta }}_{2}}\], ....... ,\[{{\hat{\theta }}_{k}}\] are the values in e which maximize L(,θ1, , 2, ..........,θk)' If certain regularity conditions are satisfied, the.point where the likelihood is a maximum is a solution of the k equations
If certain regularity conditions are satisfied, the.point where the likelihood is a maximum is a solution of the k equations
\[\frac{dL({{\theta }_{1}},...........{{\theta }_{k}})}{d{{\theta }_{1}}}=0\]
\[\frac{dL({{\theta }_{1}},...........{{\theta }_{k}})}{d{{\theta }_{2}}}=0\]
.
.
.
.
\[\frac{dL({{\theta }_{1}},...........{{\theta }_{k}})}{d{{\theta }_{k}}}=0\]
In this case it ~ay also be easier to work with the logarithm of the likelihood.
We shall Illustrate these definitions with some examples.
\[f(x;p)=(_{x}^{n}){{p}^{x}}{{q}^{n-x}}\]
where q = 1 - p and p is the probability of drawing a black ball. Here p = 1/4, or p = 3/4 We shall draw a sample of three balls, that is, n = 3, with replacement and
attempt to estimate the unknown parameter p of the distribution. The estimation problem is particularly simple in this case because we have only to choose between the two numbers .25 and .75. Let us anticipate the results of the drawing of the sample. The possible outcomes and their probabilities are given below:
Outcome
: x
|
0
|
1
|
2
|
3
|
f(x;3/4)
|
1/64
|
9/64
|
27/64
|
27/64
|
f(x;1/4)
|
27/64
|
27/64
|
9/64
|
1/64
|
\[\hat{p}=\hat{p}(x)=\{_{.75}^{.25_{for,x=2,3}^{for,x=0,1}}\]
The estimator thus selects for every possible x the value of p, say ,\[\hat{p}\] such that
f(x;\[\hat{p}\]) > f(x;p')
where p' is the alternative value of p. More generally, if several alternative values of p were possible, we might reasonably proceed in the same manner. Thus if we found x = 6 in a sample of 25 from a binomial population, we should substitute all possible values of p in the expression
\[f(6;p)=(_{6}^{25}){{p}^{6}}{{q}^{25-6}}\]
anti choose as our estimate that value of p which maximized f(6; p). For the given possible values of p we should find our estimate to be 265- The position of its maximum value can be found by putting the derivative of the function defined in Eq. (2) with respect to p equal to 0 and solving the resulting equation for p. Thus
\[\frac{d}{dp}f(6;p)=(_{6}^{25}){{p}^{5}}{{\left( 1-p \right)}^{18}}\left[ 6\left( 1-p \right)-19p \right]\]
and on putting this equal to 0 and solving for p, we find that p = 0, 1, 6/25 are the roots. The first two roots give a minimum, and so our estimate is therefore p = 6/25' This estimate has the property that
f(6;\[\hat{p}\])>f(6;p')
where pi is any other value of p in the interval 0 ≤ p < 1<1 .="" define="" estimators="" first="" function.="" in="" likelihood="" maximum-likelihood="" nbsp="" order="" p="" shall="" the="" to="" we="">
Definition 2
Likelihood function
The likelihood function of n random variables X1 , X2 , .......Xn is defined to be the joint density of the n random variables, say fx1, ... ,xn (X1 , ••• , Xn; 0), which is considered to be a function of 8. In particular, if X1, ... , Xn is a random sample from the density f(x;,θ), then the likelihood function is f(x1; ,θ)f(X1; ,θ)·..............'f(Xn; ,θ)Notation
To remind ourselves to think of the likelihood function as a function of ,θ, we shall use the notation L(,θ; x1 ••• , xn) or L(. ; x1 ..• , xn ) for the likelihood function.The likelihood function L(,θ; x1 ' ••• , xn) gives the likelihood that the random variables assume a particular value x1, x2 ... , xn. The likelihood is the value of a density function; so for discrete random variables it is a probability. Suppose for a moment that ,θ is known; denote the value by ,θ0 , The particular value of the random variables which is "most likely to occur" is that value x'1, x'2 ... , x'n such that f x1, ... , xn (X1 , ••• , Xn ; ,θ0) is a maximum. For example, for simplicity let us assume that n = 1 and X1 has the normal density with mean 6 and variance 1. Then the value of the random variable which is most likely to occur is X1 = 6. By" most likely to occur" we mean the value x'1 of X1 such that ∅6,1(x'1) > ∅6,1(x1)' Now let us suppose that the joint density of n random variables is fx1, ...• xn(X1, ••• , Xn;,θ), where 8 is unknown. Let the particular values which are observed be represented by x1' , x2' ., ., xn' . We want to know from which density is this particular set of values most likely to have come. We want to know from which density (what value of 0) is the likelihood largest that the set x1' ..., ., xn' .was obtained. In other words, we want to find the value of 0 in 9, denoted by {\[\hat{,θ}\], which maximizes the likelihood function L(,θ; x1' ..., ., xn' ). The value {J which maximizes the likelihood function is, in general, a function of x1 ••• , xn, say\[\hat{,θ}\] =\[\hat{,θ}\] (x1, x2 ... , xn) When this is the case, the random variable \[\hat{,θ}\] =,\[\hat{θ}\] (X1 , ••• , Xn) is called the maximum likelihood estimator of ,θ. (We are assuming throughout that the maximum of the likelihood function exists.) We shall now formalize the definition of a maximum-likelihood estimator
Maximum-likelihood estimator
Let
L(,θ)= L(,θ; x1 ••• , xn)
be the likelihood function for the random variables X1, ••• , Xn If \[\hat{,θ}\] [where \[\hat{,θ}\] = \[\hat{,θ}\](x1 ••• , xn) is a function of the observations x1 ••• , xn]
is the value of ,θ in \[\hat{,θ}\] which maximizes L(,θ), then \[\hat{,θ}\] = \[\hat{,θ}\](X1 , ••• , Xn)
is the maximum-likelihood estimator of \[\hat{,θ}\] = \[\hat{,θ}\](x1, x2 ... , xn) is the
maximum-likelihood estimate of ,θ for the sample X1' ....... Xn'
The most important cases which we shall consider are those in which X1 , X2 , .......Xn is a random sample from some density f(x; ,θ), so that the
likelihood function is
L(,θ)= f(x1;,θ)f(x2;,θ)............f(xn;,θ)
Many likelihood functions satisfy regularity conditions; so the maximum likelihood
estimator is the solution of the equation
dL(,θ)/d(,θ) = 0
Also L(O) and log L(O) have their maxima at the same value of 0, and it is sometimes
easier to find the maximum of the logarithm of the likelihood.
If the likelihood function contains k parameters, that is, if
L(,θ1, , ,θ2, ..........,θn) =
\[\prod\limits_{i=1}^{n}{f\left( {{x}_{i}};{{\theta }_{1}},{{\theta }_{2}},...............{{\theta }_{k}} \right)}\]
then the maximum-likelihood estimators of the parameters ,θ1, ,θ2, ... , ,θk are the random variables,\[{{\hat{\theta }}_{1}}\] = ,\[{{\hat{\theta }}_{1}}\](X1 , X2 , .......Xn), ,\[{{\hat{\theta }}_{2}}\] = ,\[{{\hat{\theta }}_{2}}\](X1 , X2 , .......Xn ) ... , ,\[{{\hat{\theta }}_{k}}\]= ,\[{{\hat{\theta }}_{k}}\](X1 , X2 , .......Xn)' where ,\[{{\hat{\theta }}_{1}}\], ,\[{{\hat{\theta }}_{2}}\], ....... ,\[{{\hat{\theta }}_{k}}\] are the values in e which maximize L(,θ1, , 2, ..........,θk)' If certain regularity conditions are satisfied, the.point where the likelihood is a maximum is a solution of the k equations
If certain regularity conditions are satisfied, the.point where the likelihood is a maximum is a solution of the k equations
\[\frac{dL({{\theta }_{1}},...........{{\theta }_{k}})}{d{{\theta }_{1}}}=0\]
\[\frac{dL({{\theta }_{1}},...........{{\theta }_{k}})}{d{{\theta }_{2}}}=0\]
.
.
.
.
\[\frac{dL({{\theta }_{1}},...........{{\theta }_{k}})}{d{{\theta }_{k}}}=0\]
No comments:
Post a Comment