modnlp.idx.inverted

Class TokeniserJP

java.lang.Object

modnlp.util.Tokeniser

modnlp.idx.inverted.TokeniserJP

public class TokeniserJP
extends Tokeniser

Tokenise a chunk of *Japanese laguage* text and record the position of each token

See Also:

Field Summary

Fields inherited from class modnlp.util.Tokeniser

encoding, indexPuntuation, originalText, SEPTKARR, SEPTOKEN, tagIndexing, tokenMap, verbose

Constructor Summary

Constructors

Constructor and Description
TokeniserJP(java.io.File t, java.lang.String e)
TokeniserJP(java.lang.String t)
TokeniserJP(java.net.URL t, java.lang.String e)

Method Summary

Methods

Modifier and Type	Method and Description
java.lang.String	getIgnoredElements() Get the IgnoredElements value.
TokenIndex	getTokenIndex(java.lang.String str)
void	setIgnoredElements(java.lang.String newIgnoredElements) Set the IgnoredElements value.
java.util.List<java.lang.String>	split(java.lang.String s)
void	tokenise() tokenise: Very basic tokenisation; Serious tokenisers must override this method.

Methods inherited from class modnlp.util.Tokeniser

disbar, fixType, getEncoding, getIndexPuntuation, getOriginalText, getTagIndexing, getTokenMap, getVerbose, isBar, setEncoding, setIndexPuntuation, setTagIndexing, setTokenMap, setVerbose, splitWordOnly

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokeniserJP

public TokeniserJP(java.lang.String t)

TokeniserJP

public TokeniserJP(java.io.File t,
           java.lang.String e)
            throws java.io.IOException

Throws:

java.io.IOException

TokeniserJP

public TokeniserJP(java.net.URL t,
           java.lang.String e)
            throws java.io.IOException

Throws:

java.io.IOException

Method Detail

getIgnoredElements

public final java.lang.String getIgnoredElements()

Get the IgnoredElements value.

Returns:

a String value

setIgnoredElements

public final void setIgnoredElements(java.lang.String newIgnoredElements)

Set the IgnoredElements value.

Overrides:

setIgnoredElements in class Tokeniser

Parameters:

newIgnoredElements - The new IgnoredElements value.

tokenise

public void tokenise()
              throws java.io.IOException

Description copied from class: Tokeniser

tokenise: Very basic tokenisation; Serious tokenisers must override this method. Note that positions in the tokenMap here correspond to the ORDER in which the token appears in originalText not its actual OFFSET.

Overrides:

tokenise in class Tokeniser

Throws:

java.io.IOException

See Also:

for a proper implementation.

split

public java.util.List<java.lang.String> split(java.lang.String s)

Overrides:

split in class Tokeniser

getTokenIndex

public TokenIndex getTokenIndex(java.lang.String str)

Overrides:

getTokenIndex in class Tokeniser