Text Classification Using Mixture Models

The goal of text document classification is to assign a new document into one class from the predefined classes based on its contents.

Mixture Models

The usage of finite mixtures for class-conditional probability functions is a useful method in pattern recognition, because mixture models are able to represent arbitrarily complex probability functions. Mixtures are flexible enough for finding an appropriate tradeoff between model complexity and the amount of the training data available.

Our approach to learning on text document is based on the fact that documents in the same class are often mixtures of multiple topics. Although mixture models are important for continuous data, some research shows, that these models perform, with discrete data in text classification, very well too.

We focus on the application of the mixture of multivariate Bernoulli distributions Bernoulli mixture model and on the mixture of multinomial distributions Multinomial mixture model. The estimation of mixture parameters is done by EM-algorithm.

For performance evaluation, we carried out the experiments with the Reuters-21578 data collection and the Newsgroups data set

Experimental results on the Reuters and the Newsgroups data sets indicate the effectiveness of the multinomial mixture model.

Bernoulli mixture and multinomial mixture models have been used as class-conditional models for the task of text document classification to relax the naive Bayes class-conditional independence assumption. This generalization of naive Bayes tries to properly model significant class-conditional dependencies by spreading them over different mixture components.

An observation of the performance of Bayes classifier for text classification on Reuters-21578 and Newsgroups data sets suggests that learning methods based on Bernoulli mixture and multinomial mixture models for class-conditional probability functions of the documents performed better than the corresponding standard models.

Feature Selection

We propose to use the multiclass Bhattacharrya distance (MBD) for multinomial model as a feature selection criterion to measure the ability of feature subsets in discriminating between classes and to take into consideration how features work together (opposed to information gaincriterion).

The performance of FS on small training sets was observed. We used a few documents for training the multinomial model; ten documents from each class of the Reuters data set and hundred documents from each class of the Newsgroups data set. The rest of documents was employed for testing.

The behaviour of both methods on Reuters data set is similar on an extremely small number of words (less than 25) but after this point the classification accuracy of MBD is considerably higher.
Novovičová, J., and A. Malík, "Application of finite mixtures to text document classification", Znalosti 2003. Sborník příspěvků 2. ročníku konference, Ostrava, VŠB, pp. 23-32, February, 2003.
Novovičová, J., and A. Malík, "Application of multinomial mixture model to text classification", Pattern Recognition and Image Analysis, Berlin, Springer, pp. 646-653, June, 2003.