Estimation theory

1. Estimation Theory 1

2. Estimation Theory We seek to determine from a set of data, a set of parameters such that their values would yield the highest probability of obtaining the observed data. The unknown parameters may be seen as deterministic or random variables There are essentially two alternatives to the statistical case When no a priori distribution assumed then Maximum Likelihood When a priori distribution known then Bayes

3. Maximum Likelihood Principle: Estimate a parameter such that for this value the probability of obtaining an actually observed sample is as large as possible. I.e. having got the observation we “look back” and compute probability that the given sample will be observed, as if the experiment is to be done again. This probability depends on a parameter which is adjusted to give it a maximum possible value. Reminds you of politicians observing the movement of the crowd and then move to the front to lead them?

4. Estimation Theory Let a random variable have a probability distribution dependent on a parameter The parameter lies in a space of all possible parameters Let be the probability density function of Assume the the mathematical form of is known but not

5. Estimation Theory The joint pdf of sample random variables evaluated at each the sample points Is given as The above is known as the likelihood of the sampled observation

6. Estimation Theory The likelihood function is a function of the unknown parameter for a fixed set of observations The Maximum Likelihood Principle requires us to select that value of that maximises the likelihood function The parameter may also be regarded as a vector of parameters

7. Estimation Theory It is often more convenient to use The maximum is then at

8. An example Let be a random sample selected from a normal distribution The joint pdf is We wish to find the best and

9. Estimation Theory Form the log-likelihood function Hence or

10. Fisher and Cramer-Rao The Fisher Information helps in placing a bound on estimators Cramer-Rao Lower Bound:“If is any unbiased estimator of based on maximum likelihood then Ie provides a lower bound on the covariance matrix of any unbiased estimator

11. Estimation Theory It can be seen that if we model the observations as the output of an AR process driven by zero mean Gaussian noise then the Maximum Likelihood estimator for the variance is also the Least Squares Estimator.

12. The Cramer-Rao Lower Bound This is an important theorem which establishes the superiority of the ML estimate over all others. The Cramer-Rao lower bound is the smallest theoretical variance which can be achieved. ML gives this so any other estimation technique can at best only equal it. this is the Cramer-Rao inequality.

14. Inverse of the Fisher Matrix:

15. lowest possible variance

16. Purpose of CRB analysis:

17. indicate the performance bounds of a particular problem.

18. facilitate analysis of factors that impact most on the performance of an algorithm.

19. Fisher Matrix of our Energy Decay ModelThe Cramer-Rao Lower Bound

21. Intuition: We can discriminate values of s better if r(s) is changing rapidly. If r(s) does not change much with s, then we don’t learn much about s.The Cramer-Rao Lower Bound

23. Fisher information is high when:

24. High negative curvature at s for all r (on average)

25. Rapid change in p(r|s) at s for all r (on average)

26. Implies easy to discriminate different values of sThe Cramer-Rao Lower Bound

28. The best unbiased estimator is only so good.

29. The best unbiased estimator is the ML estimate

30. Show a relation between variance of estimators and an information measureThe Cramer-Rao Lower Bound