How to use OpenAI GPT tokens?

What are OpenAI GPT tokens?

Tokens are the basic unit that OpenAI GPT models (including ChatGPT) use to compute the length of a text. They are groups of characters, which sometimes align with words, but not always. In particular, it depends on the number of characters and includes punctuation signs or emojis. This is why the tokens count is usually different from the word count.

Video tutorial

Video preview

How to count tokens in my text?

In order to know exactly, you need to tokenize your text, which means running a tokenizer algorithm that will count all the tokens. Sounds difficult? It’s actually as easy as copy/pasting. You can use our free tokenizer.

Rule of thumb for word to token ratio

Word counts vary by language because the way text is split into tokens differs from one language to another.

On average (figures from this study):

  • English: 1 word ≈ 1.3 tokens
  • French: 1 word ≈ 2 tokens
  • German: 1 word ≈ 2.1 tokens
  • Spanish: 1 word ≈ 2.1 tokens
  • Chinese: 1 word ≈ 2.5 tokens
  • Russian: 1 word ≈ 3.3 tokens
  • Vietnamese: 1 word ≈ 3.3 tokens
  • Arabic: 1 word ≈ 4 tokens
  • Hindi: 1 word ≈ 6.4 tokens

Refer to the study for other languages.

These figures are presented for estimation purposes only and are not guaranteed.

How many tokens for punctuation marks, special characters and emojis?

Punctuation marks (,:;?!) = 1 token Special characters (∝√∅°¬) = 1 to 3 tokens Emojis (😁🙂🤩) = 2 to 3 tokens

Word and token counts examples

Write a tagline for an ice cream shop A scoop of happiness in every cone! ⮑ 15 words · 19 tokens

Generate 3 subject lines for a cold email about Instagram growth

1. "9 Ways to Boost Your Instagram Reach & Engagement" 2. "Explode Your Instagram Following - Log in to Find Out How" 3. "Maximize Your Instagram Presence - Learn How Now" ⮑ 40 words · 58 tokens

Write a thank you email to my grandmother

Dear Grandma, I hope this letter finds you well and happy. I wanted to thank you for the thoughtful and generous gift that you sent my way. It was so thoughtful of you and I truly appreciate it. Your gift was so lovely and will certainly be used and enjoyed. It was a perfect way to brighten up my day. Thank you for all the love and support you continue to give me.It means more to me than anything else. With love and gratitude, ⮑ 93 words · 107 tokens

Check out more examples

What is the price of a token?

The price of a token depends on the model. The price is per 1000 tokens. Find the pricing table below:

Check out our pricing calculator.

What is the max_tokens parameter?

max_tokens is the maximum of new tokens that should be generated in any request to OpenAI GPT APIs. This applies to requests made through GPT for Sheets and Docs. It should always obey the following constraint: prompt_tokens + max_tokens ≤ model capacity

What to do if the response from GPT is cut?

If your response is cut, it means you didn’t provide a large enough max_tokens value. You should increase the max_tokens parameter.