modnlp.idx.inverted

Class TokeniserGNU

java.lang.Object

modnlp.util.Tokeniser

modnlp.idx.inverted.TokeniserGNU

public class TokeniserGNU
extends Tokeniser

Tokenise a chunk of text and record the position of each token

See Also:

Field Summary

Fields inherited from class modnlp.util.Tokeniser

encoding, indexPuntuation, originalText, SEPTKARR, SEPTOKEN, tagIndexing, tokenMap, verbose

Constructor Summary

Constructors

Constructor and Description
TokeniserGNU(java.io.File t, java.lang.String e)
TokeniserGNU(java.lang.String t)
TokeniserGNU(java.net.URL t, java.lang.String e)

Method Summary

Methods

Modifier and Type	Method and Description
java.lang.String	getIgnoredElements() Get the IgnoredElements value.
boolean	getTagIndexing()
static void	main(java.lang.String[] args)
void	setIgnoredElements(java.lang.String newIgnoredElements) Set the IgnoredElements value.
void	setTagIndexing(boolean v)
void	tokenise() tokenise: Very basic tokenisation; Serious tokenisers must override this method.

Methods inherited from class modnlp.util.Tokeniser

disbar, fixType, getEncoding, getIndexPuntuation, getOriginalText, getTokenIndex, getTokenMap, getVerbose, isBar, setEncoding, setIndexPuntuation, setTokenMap, setVerbose, split, splitWordOnly

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokeniserGNU

public TokeniserGNU(java.lang.String t)

TokeniserGNU

public TokeniserGNU(java.io.File t,
            java.lang.String e)
             throws java.io.IOException

Throws:

java.io.IOException

TokeniserGNU

public TokeniserGNU(java.net.URL t,
            java.lang.String e)
             throws java.io.IOException

Throws:

java.io.IOException

Method Detail

getIgnoredElements

public final java.lang.String getIgnoredElements()

Get the IgnoredElements value.

Returns:

a String value

setIgnoredElements

public final void setIgnoredElements(java.lang.String newIgnoredElements)

Set the IgnoredElements value.

Overrides:

setIgnoredElements in class Tokeniser

Parameters:

newIgnoredElements - The new IgnoredElements value.

tokenise

public void tokenise()

Description copied from class: Tokeniser

tokenise: Very basic tokenisation; Serious tokenisers must override this method. Note that positions in the tokenMap here correspond to the ORDER in which the token appears in originalText not its actual OFFSET.

Overrides:

tokenise in class Tokeniser

See Also:

for a proper implementation.

getTagIndexing

public boolean getTagIndexing()

Overrides:

getTagIndexing in class Tokeniser

setTagIndexing

public void setTagIndexing(boolean v)

Overrides:

setTagIndexing in class Tokeniser

main

public static void main(java.lang.String[] args)