Find out which tokens are used for your words. Uses Hugging Face Tokenizer, locally.
End of word character (</w>) is part of the token to represent a token that is at the end of the word. This is different than if it was at the start of word, or in the middle of a word.
Tokens have alternating background colors to represent how the words can be broken up.