@Exodus
Yeah that's fair. I used to use an image downloader for this, but it can be a huge pain sorting out all the crazy image link names and what's actually useful. Ironically, an AI might be able to help with that. LOL

Version on GitHub now does support gemini and the three claude 3 models. So I guess you can test out if and how much it is better.

Just not an official release yet. Clone the repository itself and ccopy the files in the addon directory, then update the manifest if required. Or just wait a bit, author will probably release it soonish.

Hello everyone,

Sorry for the silence on this. A new ask from a client of mine meant that my hands have been pretty tied over the last two weeks, but born out of necessity came face detection, modelled after VFO's recently introduced face in view feature.

I have published a pre-release for anyone who is kind or curious enough to test. Here is the changelog.

New in this version:

• Added face detection, which will tell you whether you are clearly centered into the frame of your camera. The hotkey NVDA+shift+j is bound by default, and options to select a different device or release the camera so that other apps can use it may be found in the AI content describer context menu.

• Added an option and script to take a picture from the onboard webcam.

• Added Google's Gemini model.

• Added the three major Claude 3 models (Haiku, Sonnet, and Opus).

• Added support for Llama.cpp.

• Rewrote the AI content describer section of the NVDA settings dialog, making many of preferences model specific. On installation of the new version your settings should automatically be ported over.

• Added a model selection submenu to the AI content description context menu.

• Changed some default options to make them more logical.

• Now, when you trigger a description action, the model in use will be spoken.

I am aware that the way I've designed the model selection means that the prompt field is unnecessarily difficult to get to, a noteworthy regression. Next to bugs, that will be the first thing I'll resolve in the upcoming version, after which will hopefully come multi-turn conversation.

Anyway, that's enough talking. Here's the link to the add-on file directly. AIContentDescriber-2024.03.29.nvda-addon.

I'm looking forward to getting everyone's feedback!

Yeeeees, you are a godsend. Thank you so much for this! I'm extremely excited to try out Claude 3 Haiku. <3

Totally, you are actually spot on with the implementation I have planned. Except there will also be a list of saved prompts. I've noticed that most people seem to cycle through instruction sets depending on the task at hand i.e. recording a professional quality video will demand different feedback from attempting to deduce what an inaccessible button does, so the goal is to make it as straightforward as possible to toggle between them.

Another thing I've noticed is that while some of us really enjoy prompt engineering for the sake of seeing what we can squeeze out of the different models, the vast majority of this addons users just want something that works. I think there would be immense value in a repository of good prompts for different tasks broken down by model and maintained by contributions from the community. Along with demonstrating what is all possible it would be good to document what works and where, so people aren't expected to pay for a model that may not be optimal for their primary case. I have a couple describe selfie prompts that work spectacularly in Be My AI and Claud, but Gemini completely chokes on the one I use for comics. They're interesting like that.

Has any of this been updated on? i've been trying to keep tabs on it, because I have the chatPGT4 addon installed and the key. Also, can you chose to use this with Claud, Gemeni, or chatPGT? I'm probably going to be switching to Claud because of it's higher skill in some other aspects.

Here's a question. Is there a way, or could a way be implemented so you can get information about an overlay text dialogue? For example. when I use the Look Up Anything mod in Stardew Valley, I then use AICD to read the screen and it describes the scene in game. Then it says there is an overlay  and it reads what is in the overlay. Then it goes back to describing the scene. So, there are times when I would just want it to tell me the info from the mod and that's it. If you want, I can post an example of this so you can see what I mean.

I thought I might have heard a bit back that you could ask a question from the result you get? Or, is that still being worked on?

would love the option to enter a prompt when you hit the hotkey to do one of the actions, and the ability to ask things about the picture after submitting it and stuff, also hwo do i get as much details as jaws picture smart ai

Yeah Having a series of saved preset prompts that can be accessed with the 1 through 0 keys for instance, either in combination with the NVDA key and other modifiers or in a layer setup would be incredible!
I feel bad asking for so much without giving anything in return though. A paypal.me or ko-fi link would be nice.