51

RE: savestates, it's more or less a 65k RAM dump, but that's still simple enough and thoroughly documented. The refresh rate can be adjusted, but once every second is the default. So it can also identify what screen the game is on, and in theory wat menu item is selected, though I don't remember adding that with how simple the menus are. hmm
The level layout is in there, too, but it's an awful lot of confusion for a game with so little layout variation, so I didn't bother.

Some of my games
Keep up to date by following @Jeqofire on twitter!
Ear Ninja?

Thumbs up

52 (edited by Tensoon the Kandra 2017-10-09 10:02:11)

This idea seems very interesting, but I want to make sure that my understanding of how this works is correct.

From what I understand this software will analyze changes in memory addresses (not sure if that jargon is right) and will provide audio or text feedback depending on the changes.  The specific addresses and accompanying text/sounds will be added by users in config files.  The addresses and changes in them will be determined by something similar to cheat engine where you constantly keep track of values and update the scanner to isolate an address.  So if you apply that to old school games which have been thoroughly cracked wide open like the old pokemon games or final fantasy, one might be able to isolate the address for health values or a specific wild pokemon appearing and then add a sound or text notification when that specific value appears or reaches a certain point such as low health or whatever.  Basically my understanding revolves around changes in hex values. 

If my understanding is right this may be able to make a lot of games more playable. 

Even if my understanding is wrong, even if it stops with fighting games, this will be amazing.  This will make us able to start on equal footing with simpler fighting games like the upcoming Fantasy Strike.  That game will be a fantastic starting point for folks that are not familiar with what is required to truly excel at modern fighting games.  After losing my sight I have just not had the patients to learn combos with all their ridiculous animation cancelling and the like.

Thumbs up +1

53

@CAE_Jones - Ah, got it. Clever :-)

@Tensoon - Yeah, that's pretty much the size of it. We locate values of interest (i.e. health, ammo etc.) through software like Cheat Engine, then find pointer chains that will consistently locate the memory addresses of those values of interest and add those chains to a config file along with some audio sample(s) or speech we want output when those values hit a specific value or pass a threshold. Although memory addresses are commonly specified in hex, the values we read usually aren't - they're typically just standard primitives like integers (whole numbers), floats (numbers with decimal points) etc. We can also search and compare against strings if we want to.

And absolutely, if there are older games where the memory layout is already documented then we could simply look up the offset that will take us to a given value. It'll take a little bit more work on my part to get SoniFight to play nicely with emulated systems, but I think it'll turn out to be very do-able.

While this is the first I've heard of Fantasy Strike, in terms of learning to play fighting games at a high level Killer Instinct is a game that I know has had a lot of time and effort put into the audio design from an accessibility standpoint, and you can simpify the combo system via a combo assist feature.

Cheers!

Thumbs up

54

This may work very well with older top down styled rpg games.  Games like Final Fantasy VI and RPG maker games that have tiled maps and block by block movement.  I have a lot of ideas for how this software might be implemented in games like that, but I will hold back until it is actually released. 

I am very excited to see how it works.  I have very little programming knowledge, but this seems like something that would make it worth expanding my knowledge a bit.

Thumbs up

55 (edited by magurp244 2017-10-11 05:31:34)

You can find the technical details about Peter Meijers vOICe method [here], i've played around with it quite a bit and ported some of the source over to Python, which you can find on my repository [here].

The method takes a 2D array (typically an image) and converts it into a singular sound, with tones in the sound representing pixels, pitch representing Y position and time representing X position of said pixels, with volume representing brightness. Generally its not great for fine detail, but is functional for getting a rough overall view of regions and areas.

There are some examples of using the vOICe for 3D gaming [here] and [here], though i've found inverted depth maps work better for navigating 3D spaces, I have a basic minecraft example using a simulated depthmap over [here] demonstrating the concept.

-BrushTone v1.3.0: Accessible Paint Tool
-AudiMesh3D v1.0.0: Accessible 3D Model Viewer

Thumbs up +1

56

@magurp244 - Thanks for the links and discussion, that's really useful. I was already thinking about sonifying the depth buffer and have been looking into RenderDoc as a way of accessing it, but at least now I know I don't have to re-invent the wheel and can adapt from other work. I'll take a look at the vOICe code and your examples this week, as I think the technique would be fantastic to incorporate (althought the functionality likely won't be there on initial release).

Thumbs up

57

r3dux wrote:

@kool_turk - that's a good question, I'm not quite sure what kind of audio feedback scheme would work best for side-scrolling games. It's something that's going to have to be experimented with. I'd be interested in any feedback CAE_Jones might have on the feedback scheme used by their save-state based Streets of Rage accessibility program. Also, be aware that support for emulated games has not been implemented at present but is theoretically possible.

@hadi.gsf - Finding the pointer trails is definitely the most troublesome part of the process, and while there are techniques / strategies that can be used I'm not entirely sure to what extent users with visual impairment will be able to perform this process. My gut feeling is that it's do-able, but it might be a bit of a pain in the neck.

I'm currently looking into some alternative techniques rather than the present 'brute force' approach where you find a value, generate a list of pointers that point to the address of that value, quit and restart the game, find the value again, and finally filter the pointer list to only keep those pointer chains that now point to the new location of the value. This 'whittling down' process is repeated until you either have just one, or only a small number of chains - one of which should be able to consistently locate the value of interest.

As a source of pointer chains I've also started thinking a little about trainers - not the shoes, but the ones that give you infinite health or ammo and so on ;-) My present understanding is that for something like an infinite health trainer, the address of the instruction that modifies the health value is located via an array of bytes search, and then the actual code that decreases the value is replaced with an instruction which doesn't such as a no-op. So if a trainer contains the location of the code that alters the value, then the code that performs that action must know where the value to be changed is located. Unfortunately my assembly language is pretty weak at the moment, but I'm hopeful that something along these lines could be used to extract the locations of values from trainers based on Cheat Engine cheat tables.

I'm not quite sure if that will work out or not, but it seems reasonable to me that it could - so I'll certainly look into it.

Thank you for your explanation!

Do you think that developers of video games would give out such information to us if we ask them too? e.g.  memory addresses for parts of the game, so then we could make them accessible. From a developer point of view, Does it make sense to provide such information to users?
A lot of developers have twitter accounts that we can ask them for this. We even have EA's accessibility manager and developer active in the  forums. Could Fifa developers, for example, give us what we need? or  security wise  it wouldn't be a good idea.

John Petrucci Fan all the time.
twitter: @hadirezae3
skype ID: hadi.gsf7

Thumbs up

58

Game modders would also be a likely source of information.

Access to information is one of the reasons I said that RPG maker games might work well with this.  With some of the pokemon fan games for example, the whole project file is available to the person that downloads it, so you can actually create your own little map with the same assets the developer used ideal for determining memory values.  This might actually make trial and error a viable option since you can be sure that whatever you are testing is being tested in isolation away from a thousand other possible things happening. 

Also, since RPG maker is such a commonly used tool, it will also be easier to find information on how various things work in the games and the engine which may also make isolating values easier.

Thumbs up

59

I'm not sure how much developers are aware of memory addresses and the like, these days, since those are usually very obfuscated outside of embedded systems. But eh, I could be way out of the loop regarding what's mainstream or what's typical for Unity etc development.

Some of my games
Keep up to date by following @Jeqofire on twitter!
Ear Ninja?

Thumbs up

60

@hadi.gsf - I don't think developers can really help us out with pointer chains because they (or the companies they work for)  likely don't want to make life particularly easy for modders unless modding is built into the game. And even if they were allowed to tell us, they'd have to go and dig into the code to locate the pointer chains themselves because they likely have no idea what they are (so that's work for them).

For the second one you could think of it this way - you likely have a phone with lots of contacts in it, but you probably don't know every single phone number off by heart. All you care about is that you have a contact for a person, and that the contact has a phone number through which you can reach them - but you aren't especially bothered about what that phone number actually is. It's a little similar with programming. You care that the value is stored, and you know that it's stored somewhere, but usually you don't really care where it's stored, just that it is stored and that you can access it on demand.

@Tensoon - Yeah, absolutely. Game modders may be able to help because they've gone to the effort of finding chains to monitor or modify game aspects for their own reasons. And if something is made in RPG maker then it's possible this may make finding the chains a little easier.

@CAE_Jones - Yup, some games deliberately obfuscate addresses of values of interest. A good example of this is that I was trying to find the health value in Doom (2016) the other day. I can find it, no problem - but then as soon as the value changes the health is stored in a different memory address! You can get around this by attaching a debugger to the process, identifying the code that actually modifies the value and using the register values that THAT code uses to locate the value's memory address, but I'm not certain about how to get at that data in a consistently reproducable manner without attaching a debugger. I'm hoping that it's possible and will keep looking into it, but I just don't know for sure yet.

An alternate explanation of why I'm having difficulty tracking down a pointer chain to Doom's health value is that I'm actually not as good as I'd like to be at doing it, and it could well be that people with greater skill in game hacking or memory forensics/analysis would be able to get it done with minimal hassle. I would actually prefer this explanation because it means that the ability exists, so then it's just a matter of knowing and applying a technique to get the job done big_smile

Thumbs up

61 (edited by r3dux 2017-10-11 12:44:14)

If you're interested, I just uploaded a brief (30 second) example of using SoniFight with the game Street Fighter 4. You can find the video here: https://www.youtube.com/watch?v=q82vj1eGu90

In the video, Ryu is on the left (played badly by me on the keyboard, female voice) and Guile is on the right (played by the CPU on easy, male voice).

Thumbs up +2

62

Ok, I saw the video and the results are awesome!
Definitedly you can do a lot of things, specially with games that have their memori chains knowed. I want to see this type of results with some  emulator games tongue

But continue with your job! is awesome!

Thumbs up +1

63

[wow]. Thanks for the short video.

Best regards SLJ.
If you like the post, then please give it a thumps up.
Feel free to contact me privately if you have something in mind. If you do so, then please send me a mail instead of using the private message on the forum, since I don't check those very often.
Happy gaming... :D

Thumbs up

64

Hi.
[wow], this is, awesome!
One question remains for me though.
Do you need to be a super programmer to enable those functions for game? I suppose you need a lot of programming experience, but how difficult is this actually?
To me, it sounds like you have to modify a lot of the games code to enable those features.
Greetings Moritz.

Hömma, willze watt von mir oder wie, weil wenn nich, dann lass dir mal sagen, laber mir kein Kottlett anne Wange und hömma, wo wir gerade dabei sind, dann iss hier hängen im Schacht, sonns klapp ich dir hier die Fingernägel auf links, datt kannze mir mal glaubn.

Thumbs up

65

The thing to keep in mind is that you should be able to share configs. So once someone's made a config for a game, no one else has to do anything but download it to the right directory. And for complex configs that would become an ever-updating community project, it can probably be made into an open-source project via the likes of Githhub.
At least, that's how it sounds so far.

Some of my games
Keep up to date by following @Jeqofire on twitter!
Ear Ninja?

Thumbs up +1

66

god damn it, that sounds awesome! Nice win there btw, ken I think? Been a while since i've played, should get back into that.

I am the blind jedi, I use the force to see. I am the only blind jedi.

Thumbs up +1

67

I watched the video and while it wasn't bad, I would report the health a little different.

For those who know how NVDA reports the progress status you'll know what I'm talking about.

Have a sequence of beeps that get lower in pitch as your health decreases.

The only thing I didn't like in that video clip was the TTS was too loud.

It was drowning out the audio in the game.

Apart from that, not bad.

Thumbs up

68

@67
As you might not have clocked, two things come to mind:
1. You'll be able to, from what I understand, set up all your own health values to look for to do just what you're suggesting all by yourself.
2. The TTS was meant to be loud in this case from what I'd guess, to demonstrate how the system works.  If it's this loud when you get it, you can just change the volume of the samples as well.

Regards,
Sightless Kombat.
***If you wish to refer to me in @replies, use Sightless***

Thumbs up +1

69

r3dux wrote:

@hadi.gsf - I don't think developers can really help us out with pointer chains because they (or the companies they work for)  likely don't want to make life particularly easy for modders unless modding is built into the game. And even if they were allowed to tell us, they'd have to go and dig into the code to locate the pointer chains themselves because they likely have no idea what they are (so that's work for them).

For the second one you could think of it this way - you likely have a phone with lots of contacts in it, but you probably don't know every single phone number off by heart. All you care about is that you have a contact for a person, and that the contact has a phone number through which you can reach them - but you aren't especially bothered about what that phone number actually is. It's a little similar with programming. You care that the value is stored, and you know that it's stored somewhere, but usually you don't really care where it's stored, just that it is stored and that you can access it on demand.

@Tensoon - Yeah, absolutely. Game modders may be able to help because they've gone to the effort of finding chains to monitor or modify game aspects for their own reasons. And if something is made in RPG maker then it's possible this may make finding the chains a little easier.

@CAE_Jones - Yup, some games deliberately obfuscate addresses of values of interest. A good example of this is that I was trying to find the health value in Doom (2016) the other day. I can find it, no problem - but then as soon as the value changes the health is stored in a different memory address! You can get around this by attaching a debugger to the process, identifying the code that actually modifies the value and using the register values that THAT code uses to locate the value's memory address, but I'm not certain about how to get at that data in a consistently reproducable manner without attaching a debugger. I'm hoping that it's possible and will keep looking into it, but I just don't know for sure yet.

An alternate explanation of why I'm having difficulty tracking down a pointer chain to Doom's health value is that I'm actually not as good as I'd like to be at doing it, and it could well be that people with greater skill in game hacking or memory forensics/analysis would be able to get it done with minimal hassle. I would actually prefer this explanation because it means that the ability exists, so then it's just a matter of knowing and applying a technique to get the job done big_smile

Thank you for explaining these stuff. I am really, really excited for this. I can't wait for the release of the software! Even though i probably wouldn't be able to get things going at first.

John Petrucci Fan all the time.
twitter: @hadirezae3
skype ID: hadi.gsf7

Thumbs up

70 (edited by r3dux 2017-10-13 23:34:22)

@simba - I'm glad that you like it. You don't have to modify any of the games code at all - in fact, we don't really have any access to the games code, apart from the binary version of it that actually allows us to play the game of course (grin). So no, you don't have to do any coding. To use the software with a config someone else has made you just launch SoniFight, pick your config, click the run button then run your game - the software will do everything else to provide additional audio cues as specified in the config for that particular game.

If you decide that you want to write a config for a particular game that you want additional audio cues for, then that's where it gets a little more complex as you have to find pointer chains to the values of interest (in the software this is called a 'watch'). And then you have to provide a criteria for the value to match and a sample to play (which in the software is called a 'trigger').

There's an in-depth user manual that comes with it, so you'll be able to have a look through that to understand how it all fits together when the software's released.

@CAE_Jones - Absolutely. My hope is that people will get together to 'crowd-source' the creation of configs for the games they feel would benefit from it. There could even be things like polls for which games or aspects to tackle next that you could vote on. I don't know what's going to happen when the software's released, but I'm excited to find out!

@kool_turk - The audio samples in that demonstration are file based, and each trigger (i.e. each sample that plays) has its own volume control, so you can set the volume you'd like it to be played. The range is between 0.0 and 1.0. Also, you're free to change any sample from what's provided to any kind of tone or beep you'd like. In addition, each sample has a playback speed property so if you wanted a series of tones that descended in pitch, then you could just have a single 'beep' sample and adjust the playback speed to change the pitch. If I'd used a series of tones or chimes in the video, then I'd also have had to explain what each one means - so I just wanted it to be clear what condition is being reacted to in this instance.

However, prior to this morning, if you wanted all the samples to play quieter or louder you'd have to go through each trigger and set the volume for each one individually. As this wasn't ideal I've just added master volume controls for normal and continuous triggers to the main game config so that all trigger volumes can be adjusted by changing just a single number. So thanks for your feedback, you've just helped improve the software big_smile

@Sightless - Yuppers, exactly!

@hadi.gsf - You're welcome! And I don't think you'll have any trouble using the software with existing configs, but as mentioned creating your own configs is where it gets a little bit trickier.

Thumbs up

71

@70, I'd love it if you'd release the user manual that you have so far so all of us can see what's available so far, but hey, I'll be fine if I have to wait! big_smile One good game I'd love to set up with this is Quake 3. Now that would be awesome. And playing with it to see what I can accomplish with it would be a great long term project too.

"On two occasions I have been asked [by members of Parliament!]: 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out ?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."    — Charles Babbage.

Thumbs up

72 (edited by hadi.gsf 2017-10-16 07:49:49)

Hey,
Once again i have a dumb question about game menus, sorry lol.

For the game UIs that are heavily keyboard/controller dependent, There is always a UI element that is selected by the controller. If this is the case, Are there possibilities of a single variable  or  pointer that would hold the value of the selected menu item?
Shouldn't  there be a single entity that holds such information, (For example the main menu item called "settings"), so when i focus on settings and  press enter or select, How does the game know to go to settings and not "online" for example?

Though now i'm thinking, even though that can be true for some cases, They could store the values as numbers or some weird strings instead of the actual UI item that is on the screen.

John Petrucci Fan all the time.
twitter: @hadirezae3
skype ID: hadi.gsf7

Thumbs up

73

The game has to decide what to highlight/display/etc, so it's not undetectable. But this would be a bit complicated, since it's not unlikely that the selected item is internally identified with an int. So determining which item, numerically, is selected shouldn't be quite as troublesome as identifying said item.
And then there's the case where the int is a property of each menu, if the menus are separate objects. So, yeah, possible, but would take a few extra steps.
If you could intercept the bits which display text, that might help, but most games, so far as I know, display as much of the menu as can fit on screen as possible. Some games do have menu items rotate out, which would be more annoying because they'd almost definitely be 3d objects and you'd be better off just using a menu guide to match numbers to items. Even in the general case, where the selected item is indicated by a pointer, border, highlight bar, color change, etc, after Unity3d came along, even these started being handled as 3d objects. Now, I don't know if that means that the text is not stored as strings anywhere in memory, but that would be kinda weird for a modern system, so you probably can hunt down the text if you can figure out the format of menu items. But this will most likely be far harder to generalize than a few lines of text should be.

Some of my games
Keep up to date by following @Jeqofire on twitter!
Ear Ninja?

Thumbs up

74

@Ethin - I can release the draft of the manual, sure - here's the link: SoniFight User Guide DRAFT in PDF format .

Unfortunately I've decided that there's no way I'll get the beacons and vOICe integration for use in FPS games ready by the end of October, so that's going to have to be pushed to the next project. In better news, that means that I'm scrapping another project that I was going to do and will instead be focussing on updating SoniFight for another two to three months.

@hadi.gsf and CAE_Jones - The currently selected menu item is typically stored as an integer (i.e. just a whole number), and the text itself for the option (if it actually is text rather than a picture with words on it) doesn't change when its highlighted, so it makes more sense to find the integer that's tracked when the currently selected menu item changes. Also, just to make life interesting, different menus usually have different memory addresses for the highlighted option - and as menus can be quite involved this may mean needing a handful of different pointer chains to keep track of navigation of the game menus as a whole.

Thumbs up

75

Reading through the manual draft has me pretty excited for this project. Everything seems very straightforward. We'll be on our way to enhancing accessibility in mainstream games in no time. smile

Go to Heaven for the climate, Hell for the company. - Mark Twain

Thumbs up +1