niftycent.com
NiftyCent
US
Login
Marketplace
Discounts

Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)

Article URL: https://ndingwall.github.io/blog/tokenization

Comments URL: https://news.ycombinator.com/item?id=44134290

Points: 3

# Comments: 0

https://ndingwall.github.io/blog/tokenization

Created 2mo | May 30, 2025, 10:30:10 AM


Login to add comment

Other posts in this group

Claude Code introduces specialized sub-agents
Claude Code introduces specialized sub-agents

Article URL: https://docs.anthropic.com/en/docs/claude-code/sub-agents

Comments URL:

Jul 26, 2025, 3:20:12 AM | Hacker news
A Union Pacific-Norfolk Southern combination would redraw the railroad map
A Union Pacific-Norfolk Southern combination would redraw the railroad map

Article URL: https://www.trains.com/trn/news-reviews/news-wi

Jul 26, 2025, 3:20:06 AM | Hacker news
DJI couldn't confirm or deny it disguised this drone to evade a US ban
DJI couldn't confirm or deny it disguised this drone to evade a US ban

Article URL: https://www.theverge.com/report/714103/dji-skyrover-x1-evade-ban-amazon

Comments URL:

Jul 26, 2025, 3:20:04 AM | Hacker news
SRAM Has No Chill: Exploiting Power Domain Separation to Steal On-Chip Secrets
SRAM Has No Chill: Exploiting Power Domain Separation to Steal On-Chip Secrets

Article URL: https://cacm.acm.org/research-highlights/sram-has-no-

Jul 26, 2025, 12:50:08 AM | Hacker news
How to draw lambda diagrams (2020)
How to draw lambda diagrams (2020)

Article URL: https://risingentropy.com/how-to-draw-lambda-diagrams/

Comments URL:

Jul 25, 2025, 10:30:14 PM | Hacker news
Google in 1999: Search engines escape the portal matrix
Google in 1999: Search engines escape the portal matrix

Article URL: https://cybercultural.com/p/google-1999/

Comments URL: https://ne

Jul 25, 2025, 10:30:13 PM | Hacker news
Experimental surgery performed by AI-driven surgical robot
Experimental surgery performed by AI-driven surgical robot

Article URL: https://arstechnica.com/science/2025/07/experimental-surgery-performed-by

Jul 25, 2025, 10:30:11 PM | Hacker news
Techie
Techie