Tutorial: Mixtures of Gaussians

• Formulas and Definitions
• Training of Gaussian mixtures
• Training of parameters with the EM algorithm

Mixtures of Gaussians

Formulas and Definitions

Gaussian Mixtures are combinations of Gaussian, or `normal', distributions. A mixture of Gaussians can be written as a weighted sum of Gaussian densities.

Recall the -dimensional Gaussian probability density function (pdf):

(1)

A weighted mixture of Gaussians can be written as

(2)

and   for

(3)

In Figure 1 an example is given for an one dimensional Gaussian mixture, consisting of three single Gaussians.

Figure 1: One dimensional Gaussian mixture pdf, consisting of 3 single Gaussians

By varying the number of Gaussians , the weights , and the parameters and of each Gaussian density function, Gaussian mixtures can be used to describe any complex probability density function.

Training of Gaussian mixtures

The parameters of a probability density function are the number of Gaussians , their weighting factors , and the mean vector and covariance matrix of each Gaussian function.

To find these parameters to optimally fit a certain probability density function for a set of data, an iterative algorithm, the expectation-maximization (EM) algorithm can be used. We have introduced this algorithm in the tutorial about Gaussian statistics to train parameters of the Gaussian pdfs for several classes in unsupervised classification. Starting with initial values for all parameters they are re-estimated iteratively.

It is crucial to start with `good' initial parameters as the algorithm only finds a local, and not a global optimum. Therefore the solution (to where the algorithm converges) strongly depends on the initial parameters.

Experiment: Back and front vowels

The data used in this experiment have already been used in previous tutorials. Load the data file BackFront.mat. The file contains simulated 2-dimensional speech features in the form of artificially generated pairs of formant frequency values (the first and the second spectral formants, ). The 5 vowels have been grouped into two classes, the back vowels containing /a/ and /o/, and the front vowels, containing /e/, /i/, /y/. Since the distribution for each vowel is Gaussian (cf. tutorial `Gaussian Statistics and Unsupervised Learning'), each of the two classes, back vowels and front vowels, shall constitute a Gaussian mixture distribution. The data samples for the back vowels and front vowels are saved in backV and frontV.

You can visualize the data by making a 2-dimensional plot of the data:

» plot(frontV(1,:),frontV(2,:))

You can also plot a 2-dimensional histogram, with the function histo.

» histo(frontV(1,:),frontV(2,:))



Recall that you can change the viewpoint in a 3-dimensional MATLAB plot with the mouse, choosing the rotate option in MATLAB.

It is possible to depict the `real' pdf of the Gaussian mixtures according to equation 2, considering the parameters for the pdfs of the single vowels. The mean vectors and covariance matrices are saved in means_front and vars_front.

The means are saved in a 3-dimensional array. First dimension covers the mean of and , the second dimension is of size 1, and the third dimension contains the number of Gaussian mixtures. The second dimension is used if the pdf is part of a Hidden Markov Model (HMM) and denotes the number of the state in the HMM (see BNT toolkit [1]).

To find the mean for for the second Gaussian for the front vowels (i.e., the mean of the Gaussian distribution of vowel /i/) use:

» means_front(:,:,2)

The covariance matrices are stored in a 4-dimensional array. First and second dimension contains the covariance matrix , the third dimension is of size 1 (for HMMs, see above), and the forth dimension gives the number of Gaussian mixtures. To get the covariance matrix for and for the second Gaussian for the front vowels use:

» vars_front(:,:,:,2)

You can verify the values given in means_front and vars_front, resp. means_back and vars_back, by computing the mean vector and the covariance matrix for each vowel using the MATLAB commands mean ans cov.

The weighting factors (see equation 2) for the front/back vowels are stored in mm_front/mm_back, in the form of a row vector.

With these parameters we can plot the `true' pdf using the function mgaussv. It is advisable to specify an x-range and a y-range for plotting.

» % we take the x- and y-range according to the 2-dimensional histogram

» figure;mgaussv(means_front,vars_front,mm_front,71:20:980,633:45:2953)

Training of parameters with the EM algorithm

We like to estimate the parameters of our vowel groups with the EM algorithm. Use the Gaussian-Mixtures-EM-Explorer to do that. To call the explorer type

» BW(data,initial_parameters)

The first input argument are the data that should be modeled by the pdf. The second input argument is either a set of initial parameters (consisting of initial means , covariances , and weighting factors ) as a cell (see the data structure of hmmF or type help BW in MATLAB), or the number of Gaussian mixtures that should be used for the pdf. In the latter case a first guess of the initial parameters , , and is done automatically (using the command init_mhmm from the Bayes net toolbox [1]).

Experiment

Do several runs of the Gaussian-Mixtures-EM-Explorer with different initializations:

1. Choose some initial parameters for the means, variances and weighting factors. You have to generate them as specified above, and connect them as a cell object. See the data structure of hmmF which gives the true parameter set for the front vowels.
   
   » BW(frontV,hmmIni)
   
   
2. The true parameters:
   
   » BW(frontV,hmmF)
   
   
3. Let the initial parameters be determined automatically and specify only the number of Gaussian mixtures:
   
   » BW(frontV,3)
   
   

Compare with the plots you generated before (histogram, pdf with true parameters). Describe your observations.

All the examples here were given for the front vowels. Do the same things for the back vowels.

Tutorial: Mixtures of Gaussians

HMMs with Gaussian mixture emission pdfs