modnlp.tc.classify

Class DSNormalisedBayes

java.lang.Object

modnlp.tc.classify.DSNormalisedBayes

public class DSNormalisedBayes
extends java.lang.Object

Document-size normalised Naive Bayes classifier for documents represented as Boolean vectors. This class uses the CSV described in (Sebastiani, 2002). CSV values are not probabilities but the function is monotonically increasing on the estimated probability function. In standard BVBayes, larger documents will tend to get disproportionally large CSV scores. DSNormalisedBayes tackles this problem by taking into account the size to the document when computing CSV. NB: For aggressively reduced term sets, document-size normalisation (dsn) doesn't seem to yield any improvement in classification. On the contrary, it seems to worsen things, slightly. However, for lightly reduced term sets (say, agrr = 100), dsn greatly improves accuracy. Categorise each news item in corpus_list according to categ using Boolean Vector Naive Bayes (see lecture notes ctinduction.pdf, p 7) Usage:

 DSNormalisedBayes corpus_list categ prob_model [parser [smoothing]]

SYNOPSIS:
  Categorise each news item in corpus_list according to categ using
  Boolean Vector Naive Bayes (see lecture notes ctinduction.pdf, p 7)

ARGUMENTS
 corpus_list: list of files to be classified

 categ: target category (e.g. 'acq'.) The classifier will define CSV 
        as CSV_{categ}

 pmfile: file containing a  probability model generated via, say, 
         modnlp.tc.induction.MakeProbabilityModel.

 parser:      LingspamEmailParser, NewsParser [default: NewsParser]

 smoothing:   0: MLE (no smoothing), 1: Laplace, ...
  

See Also:

BVProbabilityModel, NewsParser

Constructor Summary

Constructors

Constructor and Description
DSNormalisedBayes(java.lang.String clist, java.lang.String pmfile) Set up the main user interface items

Method Summary

Methods

Modifier and Type	Method and Description
double	computeCSV(java.lang.String cat, ParsedDocument pni) CSV_i(d_j) = \sum_0^T tkj log p(t|c) * (1 - p(t|?c) / p(t|?c) * (1 - p(t|c) (where tkj \in {0, 1} is the binary weight at position k in vector d_j; multiplying by it causes terms that do not occur in the document to be ignored)
static void	main(java.lang.String[] args)
ParsedCorpus	parse(java.lang.String filename, java.lang.String plugin) pars: Set up a new parser object of type plugin, perform parsing, and return a ParsedCorpus

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DSNormalisedBayes

public DSNormalisedBayes(java.lang.String clist,
                 java.lang.String pmfile)

Set up the main user interface items

Method Detail

parse

public ParsedCorpus parse(java.lang.String filename,
                 java.lang.String plugin)
                   throws java.lang.Exception

pars: Set up a new parser object of type plugin, perform parsing, and return a ParsedCorpus

Parameters:

filename - the file to be parsed

plugin - the parser class name

Returns:

a ParsedCorpus

Throws:

java.lang.Exception - if an error occurs

computeCSV

public double computeCSV(java.lang.String cat,
                ParsedDocument pni)

CSV_i(d_j) = \sum_0^T tkj log p(t|c) * (1 - p(t|?c) / p(t|?c) * (1 - p(t|c) (where tkj \in {0, 1} is the binary weight at position k in vector d_j; multiplying by it causes terms that do not occur in the document to be ignored)

main

public static void main(java.lang.String[] args)