This is fantastic. I was wondering when something like this will come up. Thank you for your work.
I have a question for a friend who might be interested, but doesn't speak English.

Normally, GPT4 can provide quite decent descriptions in other languages too, but customizing the prompt would of course be necessary, to tell it in that language.

Does the addon support customizing the default prompt, to allow this use case?
If not, would that be something you would consider doing?

Thanks.

This is fantastic!
I was planning on building something just like this during my winter break, but I guess I no longer have to. neat!

A couple questions:
1. Does the Optimize images for size checkbox set the detail parameter when calling the Vision API to low?
2. have you considered allowing users to reply to the image descriptions? That would open up many more use cases, I think.
3. Could you possibly make the prompt field a multiline field? Properly formatted text especially if you're using Markdown, really does improve the quality of responses.
Anyway, that's it from me for now. Thanks again!
Anyway, thanks for building this!

@4 Thank you for the answer. I'll let you know how it goes, and wouldn't mind helping to provide a translated English prompt that can be used as a preset, when the time comes.

@8: That won't really work.
Say it picks up on text in Spanish. If we specifically tell it to *only* respond in English, we might experience weird results, like it deciding to translate the text from Spanish to English on its own. Which would be unfortunate because then that translation service would be translating all that text back into Spanish, and the more times you translate text, the more likely you are to introduce inaccuracies.

Anyway, the OpenAPI docs do talk about it struggling with other languages, specifically, non-latin alphabets.

My guess is this is specifically referring to it struggling to OCR non\-latin alphabets, not that it can't describe images in those languages.

I have an open ai account with an api key, but do I need to own the pro version to use this?

@11: are you referring to ChatGPT plus?
No. They are completely different services and are billed separately.

@10: Your argument doesn't make much sense to me.
Yes, image description pairs were likely done in English, but given that OpenAI is an American company, it's probably safe to assume that RLHF for this model was also done in English.
If your argument were true, GPT-4 would struggle to communicate in languages that weren't English, and it does not.

Anyway, it doesn't matter what language you're using. Accuracy will vary regardless. People should know that before using this.

Is there a monthly subscription for this? Do you have unlimited usage of the add on?

It's not a subscription. You need to add card details and buy credits with it. Once you have credits, it will slowly be used up by each request you make. You can buy more any time, and when it runs out you will probably be unable to use the addon. It's kind of like prepaid for your cell phone where you buy air time or data and then it's there until you use it all up.

@16: Of course it won't perform equally well in all languages. That's not what I'm trying to say here.
Its competence with any given language is primarily determined by the language's representation within the model's training corpus.
Common languages which are strongly represented in the corpus will perform better, while those with sparser representations will perform worse.
More concerning, though, is the fact that tokenizing non-English languages tends to be more expensive, because more tokens are required to represent each word.
All that being said, it's going to do really well with the most commonly spoken languages. I talk to it in Spanish all the time and have never noticed a degradation in performance.

Anyway, I don't disagree with you here. This is a given.
Where I disagree with you is the argument that RLHF and other fine-tuning being done in English is the reason English is the best performing language.
Models like this are really, really good at transfer learning. Given knowledge of how to converse in one language, and also given that they are basically a walking talking encyclopedia in a good chunk of languages out there, they can figure out how to talk and answer questions. Can they handle complex domain-specific stuff in said languages? Probably not, but that's not a fine-tuning problem. That's a "hey, nobody wrote two billion math textbooks in Swahili problem so I don't know how to predict the next token!" problem.
Unless OpenAI open-sourced the base GPT-4 model tomorrow, which wouldn't surprised me given the insanity and drama going on over there, there's no way for us to test this either way.

Something that we can absolutely test, though,  is how well GPT4-V handles requests in other languages.
I, for my part, have had conversations about images in Spanish already, and it does great!
I can't test domain-specific stuff because my Spanish isn't good enough for me to be able to follow complex technical things, but I am curious to know how native Spanish speakers find it.

Anyway, I guess what I'm trying to say in this rambly passage of text is that I understand what you're saying, but unless you can cite several sources that indicate that GPT-4-v doesn't work in *all* other languages other than English, we can't assume that it won't.

People assumed they understood what LLMs were capable of way back in 2022 (pre-ChatGPT) then this paper practically shook the AI world to its foundations by demonstrating that adding "let's think step by step" to prompts massively improved performance on many tasks.
LLMs are not a solved problem. There is still a lot to discover, but we can't discover new things by immediately making the assumption that something isn't gonna work.

@19:
Sorry, but your anecdotes are moot. Show me benchmarks or I'm not interested.
I'm sorry to be so harsh, but if it's the difference between a correct description and a hallucinated one, then that difference is enough.

@21, why exactly are you so against it. Have you actually used it in another language? People that actually used it with other languages tell you it works fine. And deepl? That's not supported by all languages that gpt 4 supports.