next up previous
Next: Leave-one-out Error Estimation [4 Up: MLA_Exercises_160106 Previous: Binary Decision Trees [2

Confidence Intervals [3 P]

a)
Hypothesis $ H$ makes $ r = 10$ mistakes on a sample of $ N = 80$ randomly drawn test instances. Calculate the two-sided 95% confidence interval for the true error. Also calculate the 95% one-sided confidence interval (i.e. an upper bound $ U$ such that the true error is smaller than $ U$ with 95% confidence).
b)
You know that the true error of hypothesis $ H$ lies between 0.1 and 0.4. How many test examples would you need at least to make the width of the two-sided 95% confidence interval smaller than 0.05?
c)
A nearest-neighbour and a SVM-classifier were trained for a classification problem on the same training set. The nearest-neighbour classifier makes 9 mistakes on test set $ S_1$ , which consists of 85 instances. The SVM-classifier makes 5 mistakes on an independent test set $ S_2$ of 65 examples. With what probability is the SVM hypothesis better than the nearest-neighbour hypothesis? Can you decide from the given information whether SVMs are more suitable for this problem than nearest-neighbour algorithms? Explain your answer.


next up previous
Next: Leave-one-out Error Estimation [4 Up: MLA_Exercises_160106 Previous: Binary Decision Trees [2
Pfeiffer Michael 2006-01-18