If you follow the development of artificial intelligence (AI) tools with interest or use them yourself, you certainly stumbled upon the term — AI tokens. And no, we don't talk about tokens used in cryptocurrencies (regarding terms such as AI coins or crypto coins).
In the context of this article, tokens are nothing more than units that make it possible to count the cost of using language models such as ChatGPT. But that's not all; they also help determine the length of the input and output, which are equally important in the final calculation of the usage cost of an AI model.
Does that sound complicated? Don't worry, it's not. If you still need help understanding what AI tokens are all about, then in this article, we explain what they are and how AI tokens influence the usage cost of generative AI.
What are AI tokens?
Tokens are units of text in artificial intelligence technology and tools such as ChatGPT (AI models). They can be as long as one word or consist of a few characters, including spaces. How tokens are counted also depends on the user's language.
As is known, ChatGPT works best in tandem with the English language, so how it counts tokens significantly differs when we want to communicate with it in another language. For example, when we want to see how a given text fragment will be divided into tokens through tools such as Tokenizer by OpenAI, we can observe the following occurrence:
Request in German: Wie schreibt man ein Gedicht?
Request in English: How to write a poem?
The first request in German was divided like this:
- Wie
- _sch (with a space)
- re
- ibt
- _man
- _ein
- _Ged
- icht
- ?
This gives us a total of nine tokens and 29 characters.
In contrast, the same request in English consists of the following tokens:
- how
- _to
- _write
- _a
- _poem
- ?
This sentence consists of six tokens and 20 characters.
As you can see, in this case, we saved three tokens using the same command but in English. It may not seem like a lot, but ChatGPT is usually used to process a lot of text, so such seemingly small differences can quickly translate into high costs.
According to the calculations presented by OpenAI, one token equals four characters in English, and consequently, 100 tokens equal more or less 75 words.
The significance of AI tokens and their influence on the cost
Tokens play a very important role in the usage of AI tools. Ultimately, the cost you will incur during their use is calculated based on them. They're like an AI currency. That's why it's worth taking a closer look at a few elements that influence the number of tokens.
Language
As mentioned above, the language can noticeably affect the number of tokens and how much you will pay to use a chatbot. Interestingly, when we take German characters containing diacritic marks such as ä, ü, and ö, the Tokenizer will count them as one token. However, in the case of a language such as Polish and its letters like ł, ś, ć, and when written in uppercase, they will be treated as special characters and counted as two tokens.
Input and output length
Remember that the final cost doesn't only include the number of tokens contained in input data. This means that the total cost consists of your input: the prompt and context (it can be additional text fragments uploaded into a chatbot or the instructions the bot has to follow) and output — the answer generated by the system.
Moreover, sometimes the situation looks like this: the input is very detailed and consequently long, and the system answer is short. You need to manipulate the input to make the generated answer worth it.
Understanding the dependencies between these three parameters will allow you to manage costs and optimize them efficiently.
Token limits
What's more, text generators such as ChatGPT have a built-in limit of tokens (4069). If you exceed this limit, you will have to truncate your input or delete fragments of a conversation. However, the second solution may cause the bot to lose essential elements of context, and its responses will be of less quality.
Cost management — useful tips
Managing the costs associated with the interaction between the user and generative AI systems requires balancing the length of your input and output. So, what can you do to get the most for the lowest price?
AI tokens — tips and tricks
Be concise and precise.
Short prompts will not only result in a more beneficial number of tokens but also allow you to achieve more satisfactory answers. Therefore, think carefully about the construction of your prompts and decide how much context will be enough to achieve the best effects. Additionally, the fewer tokens you use in your input, the more will be left to generate output.
Also, remember that there are some unique tokens that ChatGPT doesn't include in the cost, for example, "<|endoftext|>" that signals the end of the text or a given fragment, but there are also some like "\n" that are subject to standard counting. What does this all mean? Namely, it's worth experimenting with tokens and looking for helpful advice online.
Pay attention to the language that you use.
The grammar of different languages can differ fundamentally. Whether you decide to create prompts in English, German, or Polish will significantly affect how the system counts tokens. That's why you need to be aware of these differences and choose the option that will benefit you in terms of costs and suit your use case.
Remember about token limits.
It's very easy to get caught up in a long conversation with a bot without realizing it, and before you know it, you will lose all your tokens. There are a few ways that can help you avoid this problem. The first one is to delete previous messages, but you must know you will lose context. A better solution is summarizing the most important points of the conversation and then deleting older messages. You can also take a different approach and use system messages to instruct the bot how to respond. This will enable you to save tokens and time spent correcting the chatbot's output.
Choose the appropriate language model.
The cost of tokens depends on the language model you want to use. OpenAI has already released a few versions of ChatGPT: ChatGPT Legacy, ChatGPT-3.5 Turbo, ChatGPT-4, and ChatGPT-4 Turbo. The choice of the appropriate model depends on the goals you try to achieve. If, for any reason, you don't need the bot responses to be of very high quality, then older versions of the language model may be enough for you. These models differ in the quality of generated content and the price you need to pay for one token.
It's also worth knowing about tools that will help you predict the cost of chatbot use. For ChatGPT, it's the Tokenizer mentioned above and a handy Python library called Tiktoken. Thanks to these tools, you can estimate the cost of your input and decide whether a given conversation should be split into a few segments.
AI tokens. What are AI tokens? Summary
In the context of generative artificial intelligence, AI tokens are a unit of text that determines the cost the user pays while using ChatGPT. Tokens depend on the grammatical structure of the language, so their number will differ accordingly.
Understanding how AI tokens work is essential for the efficient use of language models. The ability to count them is critical for the optimization of costs and the best use of tools like ChatGPT. Skillful management of AI tokens will allow you to control costs and ensure that you can easily fit within the imposed limits.
