Sam said yesterday that chatgpt handles ~700M weekly users. Meanwhile, I can't even run a single GPT-4-class model locally without insane VRAM or painfully slow speeds.
Sure, they have huge GPU clusters, but there must be more going on - model optimizations, sharding, custom hardware, clever load balancing, etc.
What engineering tricks make this possible at such massive scale while keeping latency low?
Curious to hear insights from people who've built large-scale ML systems.
Comments URL: https://news.ycombinator.com/item?id=44840728
Points: 51
# Comments: 36
Autentifică-te pentru a adăuga comentarii
Alte posturi din acest grup
hanaco Weather is a minimalist social network where users post short, emotional weather thoughts and connect with others experiencing the same weather.
No likes or followers (for now) — just gen
Article URL: https://arthi-chaud.github.io/posts/type-machine/
I have recently installed this extension on FF: https://addons.mozilla.org/en-US/firefox/addon/port-authorit
Article URL: https://www.mooreds.com/wordpress/archives/3530
