Graph rebuild on every token?

[tlemo]

I noticed that llama_context::decode() appears to rebuild the graph for every token:

https://github.com/ggml-org/llama.cpp/blob/master/src/llama-context.cpp#L955-L971

Why isn't the graph, including the splitting across backends, reused across tokens?

[ggerganov]

See #13816 (reply in thread)