Click always off

[DavidKotyk]

Can a coordinate grid be added to every screenshot so the clicks are not so off?

[DavidKotyk]

But the different screen resolutions can be scaled in code by pulling the resolution. I feel like not allowing the bot to truly click will limit the projects that can be done in the future.

Like using Xcode or some other developer application will need precise clicks

[DavidKotyk]

I feel like if we keep requesting gpt 4 vision to have this ability, OpenAI will train it too.

Thanks for the information and insight!

[Kreijstal]

why don't you do image segmenetation, or add a grid of coordinates, no need to hardcode coordinates

[joshbickett]

@Kreijstal @DavidKotyk I recommend checking out this PR

It is an interesting "segmentation approach" that looks promising.

We also created a discord channel on our discord page if you want to go there to discuss more ideas.

Thanks for the interest. I'm going to close this ticket for now. Let me know if you want to discuss more on discord!