Diffusion models learn to remove noise step by step; generation starts from random noise and denoises iteratively, guided by a text embedding for text-to-image.
Multimodal LLMs encode images, audio or video into embeddings (for example with a vision encoder) and feed them into the language model alongside text tokens, so one model can reason across modalities.
Typical challenges: aligning modalities, higher compute and token costs for images and video, and hallucinations about visual details.
Loading…