modnlp.idx.database

Class Dictionary

java.lang.Object

modnlp.idx.database.Dictionary

public class Dictionary
extends java.lang.Object

Mediate access to all databases (called Dictionary for 'historical' reasons; see tec-server)

See Also:

MakeTECIndex

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	NOITEMS_LABEL
static java.lang.String	TTOKENS_LABEL
static java.lang.String	TTRATIO_LABEL
protected boolean	verbose

Constructor Summary

Constructors

Constructor and Description
Dictionary() Open a new Dictionary in read-only mode with default DictProperties ("dictionary.properties" in current directory or, failing that, hardcoded defaults).
Dictionary(boolean write) Open a new Dictionary.
Dictionary(boolean write, DictProperties dp) Open a new Dictionary.
Dictionary(DictProperties dp) Open a new Dictionary in read-only mode with DictProperties dp.

Method Summary

Methods

Modifier and Type	Method and Description
int	addToDictionary(TokenMap tm, java.lang.String fou) Add each token in tm (extracted from fou) to the index N.B.: currently, addToDictionary operations aren't atomic; if the program crashes the index could be left in an inconsistent state.
void	cleanLog()
void	close()
void	compress()
void	dump()
void	finalize()
java.util.Vector	getAllFileNames(java.lang.String key) Return a vector containing all filenames where KEY occurs in the corpus.
CaseTable	getCaseTable()
java.lang.String	getCorpusDir()
DictProperties	getDictProps()
com.sleepycat.je.Environment	getEnvironment()
java.lang.String	getExtract(java.lang.String fn, int ctx, long offset, boolean ignx)
FrequencyHash	getFileFrequencyTable(int fno, boolean nocase)
int	getFileKey(java.lang.String fou)
FileTable	getFileTable()
int	getFrequency(WordForms wforms)
int	getFrequency(WordForms wforms, SubcorpusConstraints sbc)
int[]	getIndexedFileKeys()
java.lang.String	getIndexedFileName(int k)
java.lang.String[]	getIndexedFileNames()
LogStream	getLogStream()
IntegerSet	getOccurringFiles(java.lang.String word)
int	getTotalNoOfTokens()
double	getTypeTokenRatio()
double	getTypeTokenRatio(boolean nocase)
static double	getTypeTokenRatio(int notypes, int notokens)
boolean	getVerbose()
void	init(boolean write)
boolean	isIndexed(java.lang.String fou) Check if file or URI fou is in the index.
boolean	matchConcordance(PrepContextQuery pcq, int pos, int[] posa) matchConcordance match cline against this query (represented after parseQuery() by queryArray and intervArray)
void	printConcordances(WordQuery query, int ctx, boolean ignx, java.io.PrintWriter os)
void	printConcordances(WordQuery query, int ctx, boolean ignx, java.io.PrintWriter os, SubcorpusConstraints sbc)
void	printCorpusStats(java.io.PrintWriter os)
void	printCorpusStats(java.io.PrintWriter os, boolean nocase)
void	printNoItems(java.io.PrintWriter os, int noitems)
void	printSortedFreqList(java.io.PrintWriter os) Print the entire frequency list onto os
void	printSortedFreqList(java.io.PrintWriter os, int max) Print the max topmost frequent types onto os.
void	printSortedFreqList(java.io.PrintWriter os, int from, int max, boolean nocase) Print the max topmost frequent types onto os.
void	printSortedFreqList(java.io.PrintWriter os, int from, int max, SubcorpusConstraints sbc, boolean nocase) Print the max topmost frequent types occurring in the subcorpus denoted by sbc onto os.
static void	printSubCorpusStats(java.io.PrintWriter os, int notypes, int notokens)
void	removeFromDictionary(java.lang.String fou) removeFromDictionary de-indexes file or URL fou N.B.: currently, removeFromDictionary operations aren't atomic; if the program crashes the index could be left in an inconsistent state.
void	setVerbose(boolean v)
void	sync()

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TTOKENS_LABEL

public static final java.lang.String TTOKENS_LABEL

See Also:

Constant Field Values

TTRATIO_LABEL

public static final java.lang.String TTRATIO_LABEL

See Also:

Constant Field Values

NOITEMS_LABEL

public static final java.lang.String NOITEMS_LABEL

See Also:

Constant Field Values

verbose

protected boolean verbose

Constructor Detail

Dictionary

public Dictionary()

Open a new Dictionary in read-only mode with default DictProperties ("dictionary.properties" in current directory or, failing that, hardcoded defaults).

Dictionary

public Dictionary(DictProperties dp)

Open a new Dictionary in read-only mode with DictProperties dp.

Dictionary

public Dictionary(boolean write)

Open a new Dictionary.

Parameters:

write - a boolean value: false opens the dictionary in read-only mode; true opens it for writing (enabling creation of new tables). Use default DictProperties ("dictionary.properties" in current directory or, failing that, hardcoded defaults).

Dictionary

public Dictionary(boolean write,
          DictProperties dp)

Open a new Dictionary.

Parameters:

write - a boolean value: false opens the dictionary in read-only mode; true opens it for writing (enabling creation of new tables). Use default DictProperties dp.

Method Detail

init

public void init(boolean write)

addToDictionary

public int addToDictionary(TokenMap tm,
                  java.lang.String fou)
                    throws AlreadyIndexedException,
                           EmptyFileException

Add each token in tm (extracted from fou) to the index N.B.: currently, addToDictionary operations aren't atomic; if the program crashes the index could be left in an inconsistent state. In future, implement it using JE transactions

Parameters:

tm - a TokenMap: multiset of tokens

fou - a String: the file whose TokenMap is tm

Returns:

founo: the sequential number identiying the file added

Throws:

AlreadyIndexedException - if an error occurs

EmptyFileException

removeFromDictionary

public void removeFromDictionary(java.lang.String fou)
                          throws NotIndexedException

removeFromDictionary de-indexes file or URL fou N.B.: currently, removeFromDictionary operations aren't atomic; if the program crashes the index could be left in an inconsistent state. In future, implement it using JE transactions

Parameters:

fou - a String value

Throws:

NotIndexedException - if an error occurs

isIndexed

public boolean isIndexed(java.lang.String fou)

Check if file or URI fou is in the index.

Parameters:

fou - a String value

Returns:

a boolean value

getFileKey

public int getFileKey(java.lang.String fou)

getFileTable

public FileTable getFileTable()

getIndexedFileNames

public java.lang.String[] getIndexedFileNames()

getIndexedFileKeys

public int[] getIndexedFileKeys()

getIndexedFileName

public java.lang.String getIndexedFileName(int k)

getFileFrequencyTable

public FrequencyHash getFileFrequencyTable(int fno,
                                  boolean nocase)

getEnvironment

public com.sleepycat.je.Environment getEnvironment()

getFrequency

public int getFrequency(WordForms wforms)

getFrequency

public int getFrequency(WordForms wforms,
               SubcorpusConstraints sbc)

getCaseTable

public CaseTable getCaseTable()

matchConcordance

public boolean matchConcordance(PrepContextQuery pcq,
                       int pos,
                       int[] posa)

matchConcordance match cline against this query (represented after parseQuery() by queryArray and intervArray)

N.B.: used binary search in order to speed it up to logarithmic levels (avoiding the current O(n) worst-case behaviour)

Parameters:

pcq - the 'pre-processed' query object, containing the search horizons (Horizon) objects, and a set of byte offsets per wordform for a specific file (WordPositionTable).

pos - the position (as byte offset) of the keyword on the file

posa - the entire word position array for a specified file (obtained through TPosTable.getPosArray(int)).

Returns:

true if cline matches, false otherwise.

getAllFileNames

public java.util.Vector getAllFileNames(java.lang.String key)

Return a vector containing all filenames where KEY occurs in the corpus.

Parameters:

key - the keyword to search for

printCorpusStats

public void printCorpusStats(java.io.PrintWriter os)

printCorpusStats

public void printCorpusStats(java.io.PrintWriter os,
                    boolean nocase)

printSubCorpusStats

public static final void printSubCorpusStats(java.io.PrintWriter os,
                       int notypes,
                       int notokens)

printNoItems

public void printNoItems(java.io.PrintWriter os,
                int noitems)

getTotalNoOfTokens

public int getTotalNoOfTokens()

getTypeTokenRatio

public double getTypeTokenRatio(boolean nocase)

getTypeTokenRatio

public double getTypeTokenRatio()

getTypeTokenRatio

public static final double getTypeTokenRatio(int notypes,
                       int notokens)

printSortedFreqList

public void printSortedFreqList(java.io.PrintWriter os)

Print the entire frequency list onto os

Parameters:

os - a PrintWriter value

printSortedFreqList

public void printSortedFreqList(java.io.PrintWriter os,
                       int max)

Print the max topmost frequent types onto os.

Parameters:

os - a PrintWriter value

max - an int value

printSortedFreqList

public void printSortedFreqList(java.io.PrintWriter os,
                       int from,
                       int max,
                       boolean nocase)

Print the max topmost frequent types onto os.

Parameters:

os - a PrintWriter value

max - an int value

printSortedFreqList

public void printSortedFreqList(java.io.PrintWriter os,
                       int from,
                       int max,
                       SubcorpusConstraints sbc,
                       boolean nocase)

Print the max topmost frequent types occurring in the subcorpus denoted by sbc onto os.

Parameters:

os - a PrintWriter value

max - an int value

sbc - a SubcorpusConstraints value

getCorpusDir

public java.lang.String getCorpusDir()

printConcordances

public void printConcordances(WordQuery query,
                     int ctx,
                     boolean ignx,
                     java.io.PrintWriter os)

printConcordances

public void printConcordances(WordQuery query,
                     int ctx,
                     boolean ignx,
                     java.io.PrintWriter os,
                     SubcorpusConstraints sbc)

getExtract

public java.lang.String getExtract(java.lang.String fn,
                          int ctx,
                          long offset,
                          boolean ignx)

getDictProps

public DictProperties getDictProps()

getOccurringFiles

public IntegerSet getOccurringFiles(java.lang.String word)

Parameters:

word - a String the query word (type)

Returns:

an IntegerSet the set of keys to files in which word occurs

dump

public void dump()

sync

public void sync()

compress

public void compress()

cleanLog

public void cleanLog()

close

public void close()

finalize

public void finalize()

Overrides:

finalize in class java.lang.Object

getLogStream

public LogStream getLogStream()

getVerbose

public boolean getVerbose()

setVerbose

public void setVerbose(boolean v)