Analyse heuristics to avoid overfitting for the training of
multilayer neural networks with backpropagation.
Use the Boston Housing dataset housing.mat
contained in the archive housing.zip See also
housing-description.txt for more information on the
Initialize the random number generator using the Matlab
commands rand('state',<MatrNmr>); and
Split the dataset randomly (a useful command is
randperm) in a training set (75%) and a validation set for
Perform a -fold
on the data set for a
two layer network with hidden units. Train the network with the
Quasi-Newton method trainbfg
without heuristics to avoid overfitting.
by adding to the training data in the -th iteration of the cross validation 3 noisy
versions of . No noise
should be added to the validation data of the cross-validation.
Noise should be drawn from a normal distribution with mean 0 and a
standard deviation of 0.1 (use 0.1*randn(size(Di)) to
generate a noisy version of ).
with early stopping (hand over the validation set
to the function
with weight decay (use net.performFcn =
'msereg' and net.performParam.ratio = 0.5).
Repeat these four points with
the default parameters and train for maximal 500 epochs.
Create a plot which shows for (a) - (d) the dependence of
mse performance on the validation set in the
-th iteration of the
Interpret the plot. How big is the benefit of each method?
Which method seems to be most favorable. What are the advantages
and disadvantages of each method? Could the dataset be used better
for the weight decay heuristics?
Hand in the first 10 elements of the data sets and .