Tokenizer Examples

Examples of different tokenizer configurations and usage patterns in Forgather.

Projects

• tiny_stories_bpe/ - A BPE tokenizer trained on the TinyStories dataset.
• wikitext/ - A BPE tokenizer trained on the Wikitext dataset.

For CLI usage with tokenizer projects, see docs/guides/working-with-tokenizer-projects.md.