Computational Intelligence, SS08
2 VO 442.070 + 1 RU 708.070 last updated:
General
Course Notes (Skriptum)
Online Tutorials
Practical Course Slides
Homework
Assignments
Scores
Guidelines
Archive
Exams
Animated Algorithms
Interactive Tests
Key Definitions
Downloads
Literature and Links
News
mailto:webmaster

Homework 7: Backprop and cross-validation



[Points: 10; Issued: 2003/04/04; Deadline: 2003/05/09; Tutor: Thomas Mauthner; Infohour: 2003/05/07, 14:00-15:00, Seminarraum IGI; Einsichtnahme: 2003/05/21, 14:00-15:00, Seminarraum IGI; Download: pdf; ps.gz]





This homework assignment asks you to apply cross-validation to determine the optimal number of hidden units in a two layer neural network.

  1. Use the Pima Indians data set. The file diabetes.zip contains the data as the file diabetes.mat. See the file diabetes-description.txt which is also contained in diabetes.zip for more information on the data set. The task is a classification problem.
  2. Initialize the random number generator using the Matlab commands rand('state',<MatrNmr>); and randn('state',<MatrNmr>);.
  3. Split the data randomly into a training set $ L$ (70$ \%$) and a test set $ T$ (30%) (use the function randperm). Make sure that each set consists of an equal number of examples of each class. Normalize the data with prestd.
  4. Perform a $ k$-fold cross-validation ($ k=10$) on the data set $ L$ for a two layer network with $ n_H$ hidden units. Do this respectively for $ n_H=1,...,10$. For the training use the Levenberg-Marquant (trainlm) algorithm and a method of your choice to avoid overfitting.
  5. Report the used training parameters net.trainParam and net.performParam
  6. Generate a diagram $ A$ showing the dependence of $ E_{xval} = \frac{1}{k}\sum_{i=1}^{k}e_i$ on $ n_H$, where $ e_i$ is the percentage of incorrectly classified examples on the validation set in the $ i$-th iteration of the cross validation.
  7. The optimal value for the number of hidden units $ n^*_H \in \{1,...,10 \}$ is that, for which $ E_{xval}$ in diagram $ A$ is minimal. Determine $ n^*_H$. Explain the meaning of $ n^*_H$ in the context of model selection, hypothesis classes and generalization.
  8. Generate a diagram $ B$ showing the dependence of $ E_{T}$ on $ n_H$, where $ E_{T}$ is the classification error on the test set $ T$ of a network of $ n_H$ hidden units trained on the training set $ L$ as outlined above.
  9. The optimal value for the number of hidden units $ n^+_H \in \{1,...,10 \}$ is that, for which $ E_{T}$ in diagram $ B$ is minimal. Determine $ n^+_H$. Explain the difference between $ n^+_H$ and $ n^*_H$, in particular the relevance of the test data $ T$.

Remarks

  • Present your results clearly, structured and legible. Document them in such a way that anybody can reproduce them effortless.
  • Please hand in the print out of the Matlab program you have used.