this post was submitted on 05 Oct 2023
11 points (78.9% liked)

ChatGPT

8880 readers
2 users here now

Unofficial ChatGPT community to discuss anything ChatGPT

founded 1 year ago
MODERATORS
marcar@lemmy.world
11
Understanding Generative AI: Part One - Tokenizer (scorpil.com)
submitted 1 year ago by Scorpil@discuss.tchncs.de to c/chatgpt@lemmy.world
2 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[โ€“] StereoTrespasser@lemmy.world 2 points 1 year ago (1 children)

This is really helpful, thanks. I followed up until the part about token IDs.

Token text, even in binary, is of variable length, making it hard to work with, but token IDs are just numbers.

How and why is token text converted to an ID? Is the ID for a specific word always the same?