System prompt and history: how does the model "see" a conversation, and why do long chats degrade?
Essence
A chat with a model looks like a conversation, but the model doesn't "remember" previous messages. On every request the whole text is sent again: the system prompt, the entire dialogue history and your new message. The model reads that document in full and writes a continuation. Everything you perceive as "memory" is just a growing document.
Two practical consequences follow: every new message in a long chat costs more, and the longer the document, the worse the model holds on to what matters in it.
The rest of this answer, with code and interview follow-ups, is in the DevPrep app.
Read the full answer in DevPrep