2019-09-10 22:27:38 (edited by Ethin 2019-09-10 22:37:26)

Hey all,
I have some (hopefully) excellent news regarding the accessibility of computers running UEFI-based firmware. On August 11 I joined the EDK2 RFC list, which is a list where people can propose changes to the EFI Development Kit 2 (EDK2). EDK2 is pretty much the reference implementation of UEFI, and is fully open-source. That same day I submitted an email to open discussions on implementing accessibility features as a mandate in the specification. Today (at 11:25 AM CDT) I got an email back from Leif Lindholm from Linaro describing exactly what EDK2 can and can't do regarding the UEFI specification. Additionally, he CC'd the response (and I have CC'd those same people) as well to get suggestions and ideas from them; those other people include Rafael Machado (who did a PhD on firmware accessibility, including a prototype implementing an audio driver for EDK2), Andrew Fish from Apple, Laszlo Ersek from RedHat, and Michael Kinney and Cetola Stephano from Intel. In summary, the eventual goal is to provide a reference implementation of an accessibility subsystem in EDK2, which would take "a pretty spectacularly incompetent product owner not to incorporate it in their products", according to Leif.
If all goes to plan, we'll be taking it by stages and not implementing everything at once. (That includes the fact that this first iteration will not include support for braille displays.) In the first stage, we hope to:

  • Implement protocols for communicating with audio outputs;

  • Implement protocols/hooks/events that can be intercepted in the Human Interaction Interface subsystem (HII) by a foreground/background UEFI driver/application; and

  • Implement some kind of method to allow the accessibility protocols to read what is drawn to the text when navigating around.

The third item on this stages agenda will prevent the accessibility subsystem from doing a "say all" every time something new pops up, which would get annoying. I would like your guys's opinions on this progress and what you would like to see in the future. Do remember that this is not an operating system. This is a preboot environment that is only used when your computer is starting and not while an operating system is active. Therefore, please keep your suggestions to "reasonable" ones; i.e.: we don't need speech recog or natural language processing in UEFI.
You can find the full discussion as is at https://edk2.groups.io/g/rfc/topic/32841464

"On two occasions I have been asked [by members of Parliament!]: 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out ?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."    — Charles Babbage.
My Github

2019-09-10 23:20:02

Impressive work man, getting all these people on the same page by having all these connections is exactly what we need if we are ever to get this done.
Have you considered also roping in some motherboard manufacturers, people from Microsoft, or media sources that may be interested in such a thing?

2019-09-10 23:58:10

@2, I have not. I reached out to the EDK2 developers mailing list after I was referred to it, and so haven't been able to yet. I do need to include someone from MS to balance out the chain I have already established though. I never considered taking this to the windows logo program... I thought that was discontinued...

"On two occasions I have been asked [by members of Parliament!]: 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out ?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."    — Charles Babbage.
My Github

2019-09-11 01:06:32 (edited by Chris 2019-09-11 01:10:36)

It's nice someone is finally trying to solve this issue. Here are my thoughts in no particular order. Also keep in mind I'm not a developer, so I don't know how hard some of this would be, and I don't understand most of the jargon thrown around.

I assume this is only for new systems released in the future? Would existing manufacturers have to release updates if they used UEFI in their machines? All options should be spoken and the navigation to those items should always be possible using the keyboard. As far as speech goes, I don't care what is used, because any speech in those screens would be better than nothing at all. Would eSpeak work?

Regarding audio, perhaps including the generic USB drivers would help? Even if sound couldn't be heard from the built-in card, it should be possible to connect a USB headset or other audio device and get audio. Then again, wouldn't the manufacturer be required to properly support audio anyway? I always thought the firmware was specific to its machine, so this shouldn't be an issue unless there's something I'm missing.

That's about all I can say on the subject. Good luck. This has the potential to open more employment opportunities for blind IT professionals. Oh, please get Microsoft involved. If a company like Microsoft is mandating something like this, maybe people will take it seriously. I don't know who you would contact specifically, but maybe try @msftenable on Twitter.

edit

I don't know if this is the right place for something like this, but perhaps it might help if you also posted to the Microsoft Accessibility Uservoice forum. https://microsoftaccessibility.uservoic … y-feedback

Grab my Adventure at C: stages Right here.

2019-09-11 01:13:23

This is some exciting stuff, Ethin. I shared this discussion over Twitter and sent it to Microsoft's accessibility team to look over.

2019-09-11 01:45:08

@4, in order:
1) Probably yes. Updating existing machines would require a firmware update, and updating firmware is a nontrivial (and unsafe) task. It is also unlikely that this would be standardized in UEFI 2.9 or 3.0. 2.9 is definitely unlikely, 3.0 is possible but still unlikely (it depends on how fast things go).
2) ESpeak would be hard to port to that environment. The major hurtle is getting it to compile under non-VS 2005/2008 environments, which is what its windows binary targets. The UEFI specification requires that UEFI applications/drivers use the PE executable format, but we'd need an up-to-date VS implementation to build for it (VS 2017 and beyond is required, I think). I don't think VS2015 supports UEFI.
3) Yes, every firmware implementation is unique to the machine/manufacturer. One manufacturer may have a master implementation that they use as a "generic" implementation, however they would need to modify that implementation whenever they make a new motherboard.
@jack, thanks for submitting that to MS -- I have no idea who to go to on an issue like this, but getting them on board and making their bootloader accessible would be a massive leap forward. If we have to, then we'll take one component at a time.

"On two occasions I have been asked [by members of Parliament!]: 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out ?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."    — Charles Babbage.
My Github

2019-09-11 01:51:37

Update: so I've just given those guys over at the EDK2 RFC mailing list a link to this topic. If they choose to join then we'll have some excellent individuals joining this discussion before long.

"On two occasions I have been asked [by members of Parliament!]: 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out ?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."    — Charles Babbage.
My Github

2019-09-11 03:25:28

@Ethin No problem.
@Chris: UEFI is entirely controlled by keyboard actually, so that's already taken care of. It's just the speech that's going to be hard. Especially when there's no serial port equivalent that is simply analog, i.e usb is not one of those, so having a hardware synth option isn't a good solution notwithstanding the act hardly anyone still has those (I still have my Dectalk Express) but with soundcards no longer having built-in speech processors, we now have to get a speech system to compile and run under UEFI.
@Ethin: Somewhat related to this is a fellow that I know on blind-bst with a talking preboot environment that he uses for his custom build machines. I can ask him if he'd be willing to make contributions.

2019-09-11 03:28:58

@8, The more the better. I'm curious how he managed that.

"On two occasions I have been asked [by members of Parliament!]: 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out ?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."    — Charles Babbage.
My Github

2019-09-11 03:38:24

This is fabulous stuff, Ethin. Thank you much. Will be keeping tabs on this thread and the links you've posted.

The Beast continued its studies with renewed Focus, building great Reference works and contemplating new Realities. The Beast brought forth its followers and acolytes to create a renewed smaller form of itself and, through Mischievous means, sent it out across the world.
from The Book of Mozilla, 6:27

2020-05-18 08:51:22

Any update?

Pics or it didn’t happen

2020-05-18 09:59:08

Exciting stuff.

2020-05-18 11:27:07

Yo Ethin! Anything new regarding the thing? It's very interesting!

If you want to contact me, do not use the forum PM. I respond once a year or two, when I need to write a PM myself. I apologize for the inconvenience.
Telegram: Nuno69a
E-Mail: nuno69a (at) gmail (dot) com

2020-05-19 03:46:29

Unfortunately not; I've asked them about it but things have been extremely quiet regarding that. I'm hoping these go through -- I may need to remind them of this.

"On two occasions I have been asked [by members of Parliament!]: 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out ?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."    — Charles Babbage.
My Github

2021-03-31 01:37:28 (edited by Ethin 2021-03-31 01:37:44)

Well, guys, bringing this topic back from the dead because progress has been made. As a part of Google Summer of Code 2021 I will beginning the first stage of this process by implementing an audio device output protocol. At present the plane is to tackle VirtIO sound devices, which are devices that are only emulated ones and cna be emulated by things like QEMU through KVM. Once that's been accomplished (irrespective of whether I'm accepted at GSoC 2021) we'll (hopefully) start to tackle USB audio devices, and then Intel HDA. The reason Intel HDA is last is primarily because I am not very familiar with the HDA specification, and so my plan is to get a successful sound driver working in my hobby OS kernel (along with testing the hell out of it to make sure it doesn't crackle and all that) and then to rewrite that driver in C for EDK2. That might not be the most optimal solution, but I'd rather dive in to the firmware development process with some experience under my belt instead of trying to learn it on top of the requirements of GSoC and the timeline they give me. Due to the COVID situation, I only have 10 weeks to do this once the coding process starts, and so I don't have time to mess around with the HDA specification and all that just to get it working. So VirtIO is the simplest option (and the path of least resistance).
Here's how its going to work: I'm going to implement a small audio output protocol (in UEFI specification lingo, it'll be called UEFI_AUDIO_OUTPUT_PROTOCOL). Once I have a basic working prototype, the developers want me to make a UEFI shell application to test it. (I'm mentally debating whether I should make a separate app to test it or whether I should just add a new command to the UEFI shell -- something like Grubs play command). Whichever I choose, that application/command will be working proof that I can use to prove that my implementation works (as well as being a debugging tool), as I won't have an operating system. So yeah, working on this will be interesting.
This process has been an incredibly slow one, and discussions about it are on and off, and I hope none of you guys are bothered by it. Things like this typically do take a long time to happen, so this kind of slowness is nothing compared to other kinds of major slowness in the industry (you know the works: businesses waiting absurdly long times to adopt new technologies and all). The good news is once I get this going, It should, in theory, work in OVMF, which means that anything you can load OVMF into will have this protocol in it. So we've taken our first step down this road -- we'll see where things go!

"On two occasions I have been asked [by members of Parliament!]: 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out ?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."    — Charles Babbage.
My Github

2021-03-31 03:08:40 (edited by defender 2021-03-31 03:09:28)

Even if you don't end up achieving much with the limited time and help you have, this could bring attention to the effort and possibly help to expand it later.
No matter how it turns out, thanks for being willing to tackle this!
Good luck!

2021-03-31 03:50:30

@16, thanks for the encouragement! Here's to hoping that I get far with it!

"On two occasions I have been asked [by members of Parliament!]: 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out ?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."    — Charles Babbage.
My Github

2021-03-31 05:21:03

I might be able to reach people at MS, I know a guy who knows the guy who maintains UEFI. No idea where that would lead, but I always wanted an accessible BIOS, same with bitlocker.

Reading is one form of escape. Running for your life is another. ― Lemony Snicket

2021-03-31 05:58:36

How would this work though.If the UEFI specifications got modified to allow audio output, wouldn't laptop OEMs need to update their bios files?

A learning experience is one of those things that say, "You know that thing you just did? Don't do that."

2021-03-31 06:06:17

@19, yeah, but the idea is that new computers in the future will have it by default.

"On two occasions I have been asked [by members of Parliament!]: 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out ?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."    — Charles Babbage.
My Github

2021-03-31 08:53:04 (edited by bgt lover 2021-03-31 08:54:23)

I know this is early dev and all that, but as flite is mainly built within the ancy C(c 99) specifications, won't it support being compiled in a uefi environment. I mean, the voice can sometimes be awffle, well, every damn time, but it's certainly more than we have now. Plus, it doesn't support communicating to a soundcard directly, rather it gives back buffers with samples. That's why I tryed compiling it to the nes, sending the generated samples through the dpcm(sample) channel of the soundchip. Thing is, it worked, even though slowly and only on emulators, real hardware ran out of memory or whatever, so I scrapped the code for that particular project.
Thing is, I managed to, even though with real and obvious pane, compile that for the nes, even though I had to disable some systems cause I couldn't get it working, so it would probably fit with uefi kinda good, won't you think?
edit: Can you share the uefi binary with the play command after it's done so we could run it under qemu?

2021-03-31 08:53:30

Hi Ethin,
Good luck on your way to being our VI hero! I am extremely happy you do this and I admire people like you.
Good luck again.

If you want to contact me, do not use the forum PM. I respond once a year or two, when I need to write a PM myself. I apologize for the inconvenience.
Telegram: Nuno69a
E-Mail: nuno69a (at) gmail (dot) com

2021-03-31 20:03:22

@21, I was actually thinking of Espeak. Porting it to UEFI (hopefully shouldn't) be too hard. And yeah, of course I'd share the EFI application to prove it works.

"On two occasions I have been asked [by members of Parliament!]: 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out ?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."    — Charles Babbage.
My Github

2021-03-31 21:08:07

You probably want to use flite or something, not Espeak.  The GPL-ness of Espeak is likely to be an issue in this context.

My Blog
Twitter: @ajhicks1992

2021-03-31 21:29:18

can i just say i dont know much of these dev terms, but  this is fucking cool. i cant waite until/if thisis mainstream and i can just be like yep lets just change the boot device of this thing to have the harddrive last so i can easaly install shit

i am a system, i have headmates, and that is my life, and my discord is rings2006wilson#8609