# Gaussian Mixture Models¶

A Gaussian mixture model is a probabilistic model that assumes that data are generated from a finite mixture of Gaussians with unknown parameters. The model likelihood can be written as:

$p(x|\theta) = \sum_{i=1}^{K}{\pi_i \mathcal{N}(x|\mu_i, \Sigma_i)}$

where $$p(x|\theta)$$ is probability distribution given $$\theta:=\{\pi_i, \mu_i, \Sigma_i\}_{i=1}^K$$, $$K$$ denotes number of mixture components, $$\pi_i$$ denotes weight for $$i$$-th component, $$\mathcal{N}$$ denotes a multivariate normal distribution with mean vector $$\mu_i$$ and covariance matrix $$\Sigma_i$$.

The expectation maximization (EM) algorithm is used to learn parameters of the model, via finding a local maximum of a lower bound on the likelihood.

See Chapter 20 in [Bar12] for a detailed introduction.

## Example¶

We start by creating CDenseFeatures (here 64 bit floats aka RealFeatures) as

features_train = RealFeatures(f_feats_train)

features_train = RealFeatures(f_feats_train);

RealFeatures features_train = new RealFeatures(f_feats_train);

features_train = Shogun::RealFeatures.new f_feats_train

features_train <- RealFeatures(f_feats_train)

features_train = shogun.RealFeatures(f_feats_train)

RealFeatures features_train = new RealFeatures(f_feats_train);

auto features_train = some<CDenseFeatures<float64_t>>(f_feats_train);


We initialize CGMM, passing the desired number of mixture components.

num_components = 3
gmm = GMM(num_components)

num_components = 3;
gmm = GMM(num_components);

int num_components = 3;
GMM gmm = new GMM(num_components);

num_components = 3
gmm = Shogun::GMM.new num_components

num_components <- 3
gmm <- GMM(num_components)

num_components = 3
gmm = shogun.GMM(num_components)

int num_components = 3;
GMM gmm = new GMM(num_components);

auto num_components = 3;
auto gmm = some<CGMM>(num_components);


We provide training features to the CGMM object, train it by using EM algorithm and sample data-points from the trained model.

gmm.set_features(features_train)
gmm.train_em()
output = gmm.sample()

gmm.set_features(features_train);
gmm.train_em();
output = gmm.sample();

gmm.set_features(features_train);
gmm.train_em();
DoubleMatrix output = gmm.sample();

gmm.set_features features_train
gmm.train_em
output = gmm.sample

gmm$set_features(features_train) gmm$train_em()
output <- gmm$sample()  gmm:set_features(features_train) gmm:train_em() output = gmm:sample()  gmm.set_features(features_train); gmm.train_em(); double[] output = gmm.sample();  gmm->set_features(features_train); gmm->train_em(); auto output = gmm->sample();  We extract parameters like $$\pi$$, $$\mu_i$$ and $$\Sigma_i$$ for any componenet from the trained model. component_num = 1 nth_mean = gmm.get_nth_mean(component_num) nth_cov = gmm.get_nth_cov(component_num) coef = gmm.get_coef()  component_num = 1; nth_mean = gmm.get_nth_mean(component_num); nth_cov = gmm.get_nth_cov(component_num); coef = gmm.get_coef();  int component_num = 1; DoubleMatrix nth_mean = gmm.get_nth_mean(component_num); DoubleMatrix nth_cov = gmm.get_nth_cov(component_num); DoubleMatrix coef = gmm.get_coef();  component_num = 1 nth_mean = gmm.get_nth_mean component_num nth_cov = gmm.get_nth_cov component_num coef = gmm.get_coef  component_num <- 1 nth_mean <- gmm$get_nth_mean(component_num)
nth_cov <- gmm$get_nth_cov(component_num) coef <- gmm$get_coef()

component_num = 1
nth_mean = gmm:get_nth_mean(component_num)
nth_cov = gmm:get_nth_cov(component_num)
coef = gmm:get_coef()

int component_num = 1;
double[] nth_mean = gmm.get_nth_mean(component_num);
double[,] nth_cov = gmm.get_nth_cov(component_num);
double[] coef = gmm.get_coef();

auto component_num = 1;
auto nth_mean = gmm->get_nth_mean(component_num);
auto nth_cov = gmm->get_nth_cov(component_num);
auto coef = gmm->get_coef();


We obtain log likelihood of belonging to clusters and being generated by this model.

log_likelihoods = gmm.cluster(nth_mean)

log_likelihoods = gmm.cluster(nth_mean);

DoubleMatrix log_likelihoods = gmm.cluster(nth_mean);

log_likelihoods = gmm.cluster nth_mean

log_likelihoods <- gmm$cluster(nth_mean)  log_likelihoods = gmm:cluster(nth_mean)  double[] log_likelihoods = gmm.cluster(nth_mean);  auto log_likelihoods = gmm->cluster(nth_mean);  We can also use Split-Merge Expectation-Maximization algorithm [UNGH00] for training. gmm.train_smem()  gmm.train_smem();  gmm.train_smem();  gmm.train_smem  gmm$train_smem()

gmm:train_smem()

gmm.train_smem();

gmm->train_smem();


## References¶

Wikipedia: Mixture_model

Wikipedia: Expectation–maximization_algorithm

 [Bar12] D. Barber. Bayesian reasoning and machine learning. Cambridge University Press, 2012.
 [UNGH00] N. Ueda, R. Nakano, Z. Ghahramani, and G.E. Hinton. Smem algorithm for mixture models. Neural Computation, 12(9):2109–2128, 2000.