tokenization

from corio import tokenization

Tokenizer training/configuration helpers.

Install: pip install "corio[tokenization]" --upgrade