![[ICO]](/icons/blank.gif){kind=icon}
![[PARENTDIR]](/icons/back.gif){kind=icon}
![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/folder.gif){kind=icon}
| Name | Last modified | Size | Description |
|---|---|---|---|---|
| Parent Directory | - | ||
| README.md | 2024-07-15 11:31 | 751 | |
| seg_ja.sh | 2024-07-15 11:31 | 410 | |
| seg_ko.sh | 2024-07-15 11:31 | 409 | |
| thirdparty/ | 2024-07-15 11:31 | - | |
| tokenize_indic.py | 2024-07-15 11:31 | 750 | |
| tokenize_thai.py | 2024-07-15 11:31 | 336 | |
| tokenize_zh.py | 2024-07-15 11:31 | 323 | |
| tokenizer_ar.sh | 2024-07-15 11:31 | 785 | |