# Tiktokenizer

OpenAI has a Tokenizer web app to encode text to tokens or count them. Many people use it to count tokens for ChatGPT, however the fact is that it only supports older GPT-3 and Codex models. GPT-3.5 and GPT-4 use a completely different tokenizer, cl100k_base, the canonical encoder for which, tiktoken, is implemented in Rust and available for Python as an extension. However, there’s no web app version of it from OpenAI.

David Duong created a convenient web app called Tiktokenizer which you can use instead.

Leave a Reply

Your email address will not be published. Required fields are marked *