Multimodality: images, screenshots and audio — what can you give a model?
Essence
A multimodal model accepts not only text but other kinds of data as input: images, PDFs, audio, video. Internally it all comes down to the same tokens: an image is cut into small square patches, each patch becomes a vector, and the transformer processes them together with text tokens. That's why images also consume the context window and cost money.
The rest of this answer, with code and interview follow-ups, is in the DevPrep app.
Read the full answer in DevPrep