Add downsampled images as context instead of screenshot.png?

[klxu03]

Would it be advantageous to keep a collage of downsampled previous images, maybe to 160px x 90px and just stack them in a line left to right, one after another, and constantly pass this image as additional context for each action like here is a timeline of previous states the model has traversed through?

FYI: I would be happy to draft and code this feature out!

[shubhexists]

Out of curiosity, Doesn't the Vision model has any sort of "Token Limit" like it has in GPT4?

[michaelhhogue]

@klxu03 This is a good idea. I've been trying to improve the vision prompt by telling GPT to reference its previous action. But also giving it a stack of previous images would allow it to reference even farther back. You would need to experiment with the downsampling and size of the previous image stack to not hit token limits as @shubhexists mentioned. Feel free to open a PR/draft demonstrating this change!

[klxu03]

Got it. I'll go draft up a PR for the previous image stacks! I'm planning on just making it an immediate previous message in pseudo_messages.

Also, regarding the token limit, correct me if I'm wrong, is the GPT4V context window up to 128k tokens?
If so, would there be a real concern about hitting the token limit? With some calculations, where x = one dimension of maximum image size you get x = 14044, meaning you can pass up to a 14044px x 14044px image. To hit the token limit, you would need to pass a lot of images to hit 2 * 10^8 total pixels.

@michaelhhogue @shubhexists

[michaelhhogue]

@klxu03 Can this be closed since #57 was merged?

[joshbickett]

closing since old and less relevant now