
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
˵

BasePoS: ӢĴԱע

1. 

basePoS һܵӢĴԱעߡĿϣǾ޵ڷӯоͽĿģͬʱӭһд󱨸ԼĽ 

ע⣺ڰȨ⣬ҲṩκصѵϣһֱӺԵЩҪṩδȨϵʼ


2. 

basePoS (ǰ汾1.0)ӢĴԱעߡıҪͽ£
    (1) ܴзֵı֮ÿոԺ֮ǰķִʹbaseSegʹáκηִзֱ׼ԴڱһǽȿCTBзֱע롣
    (2) ܴGBıҪģת
    (3) ıȫȫǻĴܽһơ

ӢģҪ£
    (1) еıŶӦöǰĿո
    (2) ȻǱҪģȻ飬ÿһеľӾĶЩڸûһҪ󣩡

3. 

basePoS һطдɡģʹBakeoff-4CTBԱעѵϣӢģͲûϵSection 02-21

4. 

ıע0.941(Bakeoff-4CTB)ڻSection 24ϵӢıע0.966

ԱȣBakeoff-4ղԵӦõĴԱעɼ0.943ױӢĵóɼ0.971(ͬһԼ) Section 24һΪѵĲԼ


5. ļ

basePoS ĺӢĴֱҪģļ֧֣ģļchinese.modelchinese.dictӢģļenglish.modelenglish.dictȷӦıʱ򣬶ӦģļͬĹĿ¼֮¡

6. ÷

basePoS һĿǰṩwindowsµ3264λlinux64λƽ̨ĶƷ汾÷ʽ

basePoS -l [chinese|english] [input] -o [output]

Щʵʹõ:

basePoS.x86.win32 -l chinese input.gb.txt -o output.txt

basePoS.x86.win64 input.txt -o output.txt -l english 

basePoS.x86.linux64 -l chinese -o output.txt input.txt 


Ժ

haizhao@cityu.edu.hk

2008.06.02


+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


English Manual

BasePoS: Part-of-Speech tagger for Chinese and English


1. Introduction

basePoS is a simplified version of our state-of-the-art part-of-speech tagger for both Chinese and English. 

This software are distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY. Itcan be used freely for non-commercial research and educational purposes. 

I welcome all comments, bug reports, and suggestions for improvements. 

Note that I have no right to provide any related training corpus because of license issue.


2. Function 

basePoS (current version 1.0) is a part-of-speech tagger for English and segmented ONLY Chinese text.

Requirements for Chinese input text:
(1) The text should be segmented first and words are separated by blanks. You may use baseSeg that I released to perform this work. Though any segmentation standard is acceptable by this program, I recommend that CTB standard of bakeoff is used.
(2) Only simplified Chinese in GB encoding is accepted by this program. Otherwise you need transform the input text into this style.
(3) Unified the input, especially for punctuations, digitals and letters. This may cause better performance.

Requirements for English input text:
(1) All punctuations should be separated with blanks, otherwise they will be regarded as part of words.
(2) Though it is not necessary, split too long sentence and put the shorter ones to multiple lines. This may cause better performance and higher tagging speed.

3. Techniques

This is a maximum entropy tagger. The Chinese model is trained from CTB PoS training corpus of Bakeoff-4, and the English model is trained from WSJ corpus with sections 02-21. The training details will be available later.

4. Performance

The evaluation results are 0.941 (tagging precision) in CTB test corpus of Bakeoff-4 for Chinese and 0.966 in WSJ section 24 for English, while the best result of Bakeoff-4 in closed track of CTB corpus is 0.943 and the best reported result is 0.971 for English (not the same test corpus). 

5. Files

basePoS needs two supported files for both Chinese and English text tagging. As for Chinese, these two files are
chinese.model
chinese.dict

and for English, they are
english.model
english.dict

Be sure that these files are in the same folder where the working folder locates as basePoS works.

6. Usage

basePoS is a command line program currently working in windows 32/64 and linux-64 platforms. The usage is

basePoS -l [chinese|english] [input] -o [output]

These are some usage samples:

basePoS.x86.win32 -l chinese input.gb.txt -o output.txt

basePoS.x86.win64 input.txt -o output.txt -l english 

basePoS.x86.linux64 -l chinese -o output.txt input.txt 


ZHAO Hai

haizhao@cityu.edu.hk

2008.06.02

