CNM-BERT

A Chinese character is built from parts. 樹 contains 木, the tree radical, and a reader who has never met the character can still guess what it concerns. A tokenizer discards all of that and hands the model an arbitrary index.

The model then has to rediscover the meaning from context alone, and rare characters get the fewest chances to do so. Characters the tokenizer cannot represent at all stay opaque no matter how large the model grows, because the information was destroyed at the input layer.

CNM-BERT restores it. The model parses each character's Ideographic Description Sequence into a tree, encodes that tree with a recursive Tree-MLP, and adds the result to the standard embedding. The transformer, the vocabulary, and the output head stay exactly as they were, so it drops into an existing fine-tuning pipeline unchanged.

The interesting result is the shape of the gain. On familiar text CNM-BERT wins by 1.3 F1. On the long tail it wins by 4.0. On characters outside the vocabulary it wins by 9.8. A gap that widens as the data gets harder means the model has learned something structural about writing, which is exactly what scaling alone cannot buy.