modnlp.tc.induction

Class MakeProbabilityModel

java.lang.Object

modnlp.tc.induction.MakeProbabilityModel

public class MakeProbabilityModel
extends java.lang.Object

Parse Reuters file (or any other type of file, if parser argument specified), perform term filtering (local to a specific category) and generate a probability table containing joint term-category probabilities and a few extra things, and saves it to disk (as a serialized object.) Usage:

 MakeProbabilityModel corpus_list stopwdlist aggr tf_method categ pmfile [parser]

SYNOPSIS:
  Tokenise each file in corpus_list, remove words in stopwdlist
  and reduce the term set by a factor of aggr.

ARGUMENTS
 tf_method: term filtering method. One of: 
         'df': document frequency, local,
         'dfg': document frequency, global,
         'ig': information gain.

 categ: target category (e.g. 'acq'.) for local term filtering OR
         a method for combining local scores. One of:
            '_DFG' (global document frequency),
            '_MAX' (maximum local score),
            '_SUM' (sum of local scores),
            '_WAVG' (sum of local scores wbeighted by category generality.)

 pmfile: name of output file for probability model.

 PARSER: parser to be used [default: 'NewsParser']
           'LingspamEmailParser': Androutsopoulos' lingspam corpus
           'NewsParser':          REUTERS-21578 corpus, XML version.


  

See Also:

BVProbabilityModel, TermFilter, NewsParser

Constructor Summary

Constructors

Constructor and Description
MakeProbabilityModel(java.lang.String clist, java.lang.String swlist, java.lang.String aggr) Set up the main user interface items

Method Summary

Methods

Modifier and Type	Method and Description
java.util.Set	filter(java.lang.String method, BVProbabilityModel pm, java.lang.String categ)
static void	main(java.lang.String[] args)
ParsedCorpus	parse(java.lang.String filename, java.lang.String plugin) pars: Set up a new parser object of type plugin, perform parsing, and return a ParsedCorpus

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MakeProbabilityModel

public MakeProbabilityModel(java.lang.String clist,
                    java.lang.String swlist,
                    java.lang.String aggr)

Set up the main user interface items

Method Detail

parse

public ParsedCorpus parse(java.lang.String filename,
                 java.lang.String plugin)
                   throws java.lang.Exception

pars: Set up a new parser object of type plugin, perform parsing, and return a ParsedCorpus

Parameters:

filename - the file to be parsed

plugin - the parser class name

Returns:

a ParsedCorpus

Throws:

java.lang.Exception - if an error occurs

filter

public java.util.Set filter(java.lang.String method,
                   BVProbabilityModel pm,
                   java.lang.String categ)
                     throws java.lang.Exception

Throws:

java.lang.Exception

main

public static void main(java.lang.String[] args)