Rendered at 16:29:43 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
jephs 1 days ago [-]
So the thing is, bytes are not actually a more "natural" encoding than tokens. I mean, jesus christ Unicode, right? A lot of what the BPE is doing is just un-doing the wacky arbitrary standards we use to encode characters into bytes, like, say, merging the 3-4 random bytes used to specify the codepoint of some CJK character.
There are so many possible naturalistic encodings. Waveforms, one-hot encoded discrete articulatory features, motor programs for simulated vocal apparatus, IPA, bitmaps of printed characters (handwritten? averaged across a bunch of fonts?), SVG programs to draw characters, decomposing big CJK chars via Ideographic Description Sequence, treating all unicode codepoints equally (kinda halfway between bytes and tokens)...
I don't think a thorough survey exists.
jaen 23 hours ago [-]
Bytes are not used as tokens here and are in some sense an arbitrary choice (but they are a good one).
The article's approach just requires some indivisible base "atoms" to start the building the larger "tokens" from. (see: the DNA example)
It could be raw bits, but that's pointless, since dividing bytes into bits does not give the model any extra expressive power and requires 8× more computation. (yes, there's some structure at the bit level eg. lowercase/uppercase is represented as a bit in ASCII, but that's trivial to learn from just bytes at the scale the model operates)
Bytes are the ~best point on the expressivity/computation curve for this approach (for text).
(something else to ponder - why do bytes have 8 bits?)
jephs 22 hours ago [-]
Unicode bytes are totally bizarro unless you happen to be writing English or some other language with a Latin script. They are absolutely not the best choice.
jadermcs 19 hours ago [-]
Current tokens are very problematic, they are constructed by frequency, but frequency doesn't capture morphology and syntax that well. If the model learn how to tokenize internally it's way better then forcing an incorrect tokenization.
vatsachak 1 days ago [-]
They are though. Bytes are raw data and the architecture chooses the tokens
jephs 22 hours ago [-]
Bytes are not raw data.
Vocalizations, hand writing, and keys strokes are raw data.
Bytes are downstream of both those raw communicative performances, and of arbitrary encoding schemes, and of several further layers abstraction. There are byte sequences representing Unicode codepoints that will display different glyphs depending on whether the user's OS and browser are configured like a typical person from mainland China, Taiwan, or Japan.
A single Latin letter is one byte. A single Cyrillic letter that looks identical displayed on the screen and indicates the same underlying phonological value is two bytes. That is not raw.
augment_me 1 days ago [-]
Its cool, but it goes against the whole optimization that NLP has gone through with BPE and redundancy/repetition in large data in general.
Put this on a coding task or English and you will underperform against tokenizer that exploit multi-char patterns and can decode much faster.
Given where things are going with spec decode (I've seen 32 token spec-decode pop up in the frontier models), you will need a very domain-specific use for this to be useful imo like massive tabular data, but at that point you already have everything structured, and might as well use an LLM with polars/sql/arrow.
yorwba 1 days ago [-]
The whole point of this architecture is that it can also make use of multi-char patterns, but unlike BPE, they aren't hard-coded. And you can still use speculative decoding. Whether it works better or worse is something that you have to actually measure.
nivter 1 days ago [-]
> you will need a very domain-specific use for this to be useful imo
The main premise of this idea is how general and widely applicable it is. They also show examples of how it works on images.
A common tokenizing paradigm would be a huge step towards consolidating how different modalities are preprocessed - something I believe is quite useful for intelligence to scale well across modalities.
1 days ago [-]
kavalg 23 hours ago [-]
Their premises are:
- If you work with the raw bytes in a non-hierarchical (isotropic) model, such as a byte transformer, you get better transfer to downstream tasks, but at a huge computational cost.
- if you use a tokenizer (pretrained or regexp based), you gain a lot of computational efficiency, but loose semantics and the transfer to downstream tasks may degrade
- finally, with H-Nets you get the best of both worlds. The model learns how to chunk the data in a better way than a hardcoded/separately trained tokenizer
Authors also demonstrate that this approach works in other modalities, such as DNA sequences and images, where we don't normally have lots of ready to use tokenizers.
augment_me 18 hours ago [-]
But you lose the semantics you choose to not care about. BPE is equally equipped to not fuse single chars if they have downstream value. Most modern tokenizers are already trained with
the target text corpus in mind.
For example Google's propabilistic Unigram embedding model is trained jointly with the end task, so if you would train it on a DNA set where individual chars matter it probably would not fuse them
So the interesting question is rather how the authors manage to maintain perf on all modalities/tasks with a single corpus or if they are benchmaxxing something.
kavalg 9 hours ago [-]
Good questions. I guess we will never know unless someone with enough money and data tries the approach on a larger scale.
vatsachak 2 days ago [-]
Oh man, that haskell quicksort tokenization has got me feeling great
There are so many possible naturalistic encodings. Waveforms, one-hot encoded discrete articulatory features, motor programs for simulated vocal apparatus, IPA, bitmaps of printed characters (handwritten? averaged across a bunch of fonts?), SVG programs to draw characters, decomposing big CJK chars via Ideographic Description Sequence, treating all unicode codepoints equally (kinda halfway between bytes and tokens)...
I don't think a thorough survey exists.
The article's approach just requires some indivisible base "atoms" to start the building the larger "tokens" from. (see: the DNA example)
It could be raw bits, but that's pointless, since dividing bytes into bits does not give the model any extra expressive power and requires 8× more computation. (yes, there's some structure at the bit level eg. lowercase/uppercase is represented as a bit in ASCII, but that's trivial to learn from just bytes at the scale the model operates)
Bytes are the ~best point on the expressivity/computation curve for this approach (for text).
(something else to ponder - why do bytes have 8 bits?)
Vocalizations, hand writing, and keys strokes are raw data.
Bytes are downstream of both those raw communicative performances, and of arbitrary encoding schemes, and of several further layers abstraction. There are byte sequences representing Unicode codepoints that will display different glyphs depending on whether the user's OS and browser are configured like a typical person from mainland China, Taiwan, or Japan.
A single Latin letter is one byte. A single Cyrillic letter that looks identical displayed on the screen and indicates the same underlying phonological value is two bytes. That is not raw.
Put this on a coding task or English and you will underperform against tokenizer that exploit multi-char patterns and can decode much faster.
Given where things are going with spec decode (I've seen 32 token spec-decode pop up in the frontier models), you will need a very domain-specific use for this to be useful imo like massive tabular data, but at that point you already have everything structured, and might as well use an LLM with polars/sql/arrow.
The main premise of this idea is how general and widely applicable it is. They also show examples of how it works on images.
A common tokenizing paradigm would be a huge step towards consolidating how different modalities are preprocessed - something I believe is quite useful for intelligence to scale well across modalities.
- If you work with the raw bytes in a non-hierarchical (isotropic) model, such as a byte transformer, you get better transfer to downstream tasks, but at a huge computational cost.
- if you use a tokenizer (pretrained or regexp based), you gain a lot of computational efficiency, but loose semantics and the transfer to downstream tasks may degrade
- finally, with H-Nets you get the best of both worlds. The model learns how to chunk the data in a better way than a hardcoded/separately trained tokenizer
Authors also demonstrate that this approach works in other modalities, such as DNA sequences and images, where we don't normally have lots of ready to use tokenizers.
For example Google's propabilistic Unigram embedding model is trained jointly with the end task, so if you would train it on a DNA set where individual chars matter it probably would not fuse them
So the interesting question is rather how the authors manage to maintain perf on all modalities/tasks with a single corpus or if they are benchmaxxing something.