1 (edited by r3dux 2017-10-04 22:08:38)

Hi all,

I'm a university lecturer and Ph.D student working on I/O technologies and I love video games.

Some time ago I saw a video of a player who was completely blind competing against a fully sighted opponent in a fighting game and winning convincingly - and it blew me away! At first I couldn't understand how it was even possible, so I decided to look deeper into sonification and learn about how audio cues in games are designed and can be used. While some games are designed and implemented with accessibility in mind, many fall short in this regard - which got me thinking: would it be possible to provide additional audio cues to existing games that might help players with visual impairment?

Cut to the present day (after twenty months of development) - and I think the answer is yes!

I've written a software application called SoniFight which provides sonification cues based on changes in the game state. In essence, if the game keeps track of something then it must be in the process memory, and if it’s in the process memory then we can access it and respond to any changes. For example, this can mean providing audio feedback to help navigate menus, or providing audio updates based on in-game events such as player or opponent health being low, or the clock getting near to zero in a fighting game, or the player running low on ammo in a FPS.

SoniFight isn’t quite 100% yet, but I should be able to release it by the end of October 2017 – at which point it’ll be made freely available as a binary release with a full suite of user documentation for players, along with the source code for anyone who may want to modify or improve it.

I should also mention that the software is massively configurable and works with config files for any given game, so in theory you could create a configuration for your game of choice that provides sonification events for any manner of changes to the game state. In practice, it’s a little bit more complicated to enact because to find a value of interest in a game you need to find what’s called a pointer chain to that value, and that can be a bit of a bother. But there are ways and means of doing so which are documented and demonstrated in the user docs, and once you’ve found a working chain it’ll work forever for that version of the game.

Oh, and sonification events can be a file-based samples like “Health-Low.mp3” or you can opt to use your screen reader, in which case you might put something like “Health low” (which would say exactly that), or perhaps “Health {}” (which would say the word ‘health’ followed by your exact health value).

If I’ve piqued your interest or if you have any thoughts, comments or questions about SoniFight then please don’t hesitate to ask me about it – I’d be happy to discuss any and all aspects of the software.

Cheers!

Thumbs up +9

2

Hi,
[wow], that sounds awesome, and I am looking forward to this.

Thumbs up +1

3

[wow], that would be amazing! Would this work for games which run in emulation software?

Thumbs up

4 (edited by r3dux 2017-10-04 16:58:34)

Interesting that you should ask that, because I think it will.

Also, because older consoles (think SNES or Genesis) have such limited memory, and the fact that they won't randomise the value layout - it should be an absolute doddle to find values of interest for such games.

That's just cheered me up massively - I'd forgotten that older systems were simpler, lol!

Full disclosure: It won't work with emulated systems on initial release, it's not even released yet! But yes, it absolutely can.

All that we need is the starting address, which can be found by scanning the emulator software's process memory with an array-of-byte search to locate the start of emulated system memory - and you basically find the start of the BIOS or system firmware and offset from that.

Thumbs up +1

5

OMG YES PLEASE FINALLY SOMEONE DID THIS! big_smile big_smile big_smile

I have been wanting to do something like this for years, and between health and not being much of a hacker, the farthest I got was a program that sonifies Streets of Rage 2 based on savestates. I wanted to do something more generalizable with Genz or ZSNES, but modding emulators is apparently a bit more than my attention span can handle.
A ton of older games have had virtually all of their RAM mapped, and published in wikis for modders/cheaters/hackers/whatever. So with a tool like this, making half a dozen Genesis games accessible could become the work of a weekend. ... maybe two weekends.

Thanks a ton for doing this!

Some of my games
Keep up to date by following @Jeqofire on twitter!
Ear Ninja?

Thumbs up +1

6

Oh [wow]! This is way cool! One could envision this being used to make more mainstream games accessible to us. I'm eagerly looking forward to this release! Thanks for creating and sharing it! Great work! Keep it up!

Thumbs up +1

7 (edited by r3dux 2017-10-04 17:56:19)

CAE_Jones, you say that some older games have their memory layouts already documented?

I had no idea that such information even existed - that's great! Do you have any links or search terms I could use to find such information?

Also, and again in full disclaimer mode, SoniFight doesn't support emulation at the present time. It totally CAN - but my current dev goals are just to make it a little more user friendly and fault tolerant before initial release.

Thumbs up +1

8

Really looking forward to seeing how this all turns out once everyone gets their hands on it.  It'll be interesting to see how quickly the first game using this software as an overlay other than any pre-built configs comes into being as well.

Regards,
Sightless Kombat.
***If you wish to refer to me in @replies, use Sightless***

Thumbs up

9

You can probably find memory documentation most easily in hacking guides, for example at Sonicretro or soronline. The completeness of the available information is usually based on popularity, since more people will be working on it.
I suppose the trick with emulators would be to figure out where they store the emulated RAM and use that as an offset.

Some of my games
Keep up to date by following @Jeqofire on twitter!
Ear Ninja?

Thumbs up +1

10

This has piqued my interest, but I have two questions.
One: Is this a piece of software with a large learning curve? In other words: Could I, as someone who is not very experienced technically, but not totally inept, pick this up easily?
Two: Is this limitted to PC at present? Or could I possibly rig this up with my XBOX?
My thanks to you for your efforts. I look forward to reviews.

"Dragomier is a fire dragon. It's body is covered in smoldering fiery scales, it's wings glow red and it has dark black spines going from it's head to it's torso. It has nine tongues, horns, and fangs."

Thumbs up

11 (edited by r3dux 2017-10-04 22:28:43)

CAE_Jones - if you can provide a link to a memory guide that may be useful I'll click that bad boy and get my reading on.

Dragomier - Answer 1 - To run the software the learning curve is pretty much zero, you just kick it off, pick a game config, click run, and then launch the appropriate game.

To adjust a config it's either easy or hard - if you already have the watch that points to the value of interest then it's very easy, because you can just add or modify a trigger. If you don't have access to the value then a new watch needs to be found and this is where it gets tricky. Full details of finding pointer chains can be found in the user docs, but as they aren't released at present, just take my word that it's a perfectly do-able but somewhat-of-a-dick process.

Dragomier - Answer 2 -  Yes,  this software is limited to PC only. Unless Microsoft wants to sign Cheat Engine and this app so we can plunder the game memory. Lol! (Hint: This will not happen).

Thumbs up +1

12

[wow], sounds really interesting, and the  posibilities are really huge.
So, a first question. The program itself include a memory viewer? to see witch things have a specific game on a specific state, to try see or find something interesting? or at first step we need know what we want to have spocken, and use other methods to find interesting memory points.

The program use scripts / profiles to work with different games? For example, somebody make a script to accesibilice the menus of a x game, can you share it, to import in to your program and have good results?

About the online playing. This program can be detected (falcepositive) as a cheatin program, because is checking all the time the game memory? And how we have to do, if the game use some protection systems as change the memory value of each thing each time that game runs to prevent some hacking tools?

Ham... and, it can read some raw data present in memory values? example, text, numbers, etc? For make a dinamic messages. Like, if your chahealth nges, not say "low health", but "your health is at x%" where X is the value of the memory on that time.
A, a suggestion, make the option to react with keystrokes. For example, instead of have a trigger to speak your health when it is low, give the option to press a keystroke, and with that say "your health is x%"

I'm giving the health only as example, but can works with any interesting value on a game tongue

So. that is all for now. A really iinteresting tool.

¡thanks for all!

Thumbs up +1

13 (edited by r3dux 2017-10-04 22:47:03)

Woo! Good questions! Happy to reply in turn:

1 - The program itself is not a memory viewer, it merely depends on pointer chains which are applied relative to the process base address. So no, it doesn't view memory - but it does read it.

2 - The program does not use scripts, but does use configs. Each config is a single directory inside the SoniFight\Configs folder and contains a single config.xml file and none or more audio samples. (yes, none or more - output can be purely screen reader driven).

3 - Online playing should not be an issue because SoniFight never writes anything to memory, it merely responds to changes in game state. In the future, assholes may take my code and modify it to write to memory, but there is absolutely nothing that I can do about that because haters gonna hate.

4 - Keystroke activation would be possible if demand was there for it

Thumbs up +1

14 (edited by r3dux 2017-10-04 23:14:00)

It does have the option to say "health X", you just put a pair of curly braces in the sample filename field. You can also say "health {9}" to pull the value from watch 9 etc.

This is all documented (and this aspect is mentioned in the very first post), but you haven't seen the documentation because it's not released yet. Oh, how we laughed! ;-)

Thumbs up +1

15

Here are a few memory guides for some Genesis/Megadrive / NES games. The first two are geared toward ROM hacking, but contain RAM info as well.

Here's a Streets of Rage 2 hacking guide: http://woc.brawlersavenue.net/sor2_hacking.htm
It gives addresses for savestates, but there's a link for how to convert to par codes. I'm assuming this savestate format is effectively a memory dump with some emulator-specific info prefixed, in which case there's just an offset to be accounted for.
This page has links to guides for several Sonic games. Focus is on editing ROM content, but there are RAM guides as well: http://info.sonicretro.org/SCHG:Sonic_C … e-Specific

Super Mario Bros: https://datacrystal.romhacking.net/wiki … s.:RAM_map

Super Mario Bros 3: https://datacrystal.romhacking.net/wiki … _3:RAM_map

The Legend of Zelda: https://datacrystal.romhacking.net/wiki … da:RAM_map

Megaman: https://datacrystal.romhacking.net/wiki … an:RAM_map

... and basically anything from datacrystal.
Hth

Some of my games
Keep up to date by following @Jeqofire on twitter!
Ear Ninja?

Thumbs up +1

16

OK r3, thanks for your answers! So, first, you have to fing the memory chains with other engine to know witch triggers are interesting for a program tongue

So, thanks for all. the idea  is awesome!

Thumbs up

17

Hi,
@r3dux, no questions so far, but super excited and looking forward to it!
This could finally make it so much easier to play all those mainstream games we weren't able to before,or the games that required memorising menu,gameplay and what was where and what not. so a huge cheers for this.
Grryf

Of all sad words of tongue or pen, the saddest are these, ‘It might have been.
Follow me on twitter

Thumbs up +1

18

Ah, one question regarding sounds. Namely, are on-the-fly effects such as volume, pan/balance, or speed/pitch adjustment possible? Stereo would make a huge difference. I'd settle for left/right channels with adjustable volume, but the 2d triad, as I just now decided to refer to volume/pan/pitch, would be nice.
I seem to remember something about scrolling being a low-level part of the Genesis' video system, though I don't fully understand the details. If there are actual x/y offsets being updated when a Genesis game scrolls, being able to detect that would be helpful for many games which might not be so well documented. Ex, if there are no sounds to indicate movement or bumping into an obstacle, hearing when the plames A and B are scrolling might provide that information.
Actually, I also remember hearing that the SNES has a more thorough sprite system built in. I don't remember if this comes with the whole bounding box, or just sizes, but if it's the former, that's potentially an even better bandaid for games without a customized config.

Some of my games
Keep up to date by following @Jeqofire on twitter!
Ear Ninja?

Thumbs up +1

19 (edited by r3dux 2017-10-05 04:37:38)

Another great question.

At present, all output is in mono. Yes, I know, that sucks. But these are really early days, and the iirKlang audio library used by SoniFight is capable of both panning and 2D/3D positional audio, so this isn't a limitation of the audio library, just a limitation of my software (as it currently exists) to utilise the library. And honestly, I don't think it would take that much of an overhaul to introduce either panned or positional audio feedback.

In terms of audio output at present, there are 3 different types of what I call triggers, where a trigger is a criteria which must be met in order for the software to play a sound.

There are "normal" triggers which work as you expect - clock 50? Say "clock 50", value of super gauge more than or equal to 200? Say "Super One" (because you'd have one 'stick of butter').

Then there are "continuous" triggers. These are audio samples (not screen reader compatible) that constantly play in a loop. The playback speed and/or pitch of the playback of this continuous audio is modified based on the difference between two values. So for example, if you put the player 1 and player 2 horizontal location watches as the input, and specify a maximum range, then the continuous trigger's speed or volume can be set to change based on the distance between the players. So as a concrete example that I've experimented with, it  might mean that a blowing wind sound plays during the game, and when I'm far away from the opponent it's very quiet but when I'm right up in the opponents face it plays quite loud (and all variations in-between). Or you could invert that so that it's loud when far apart and quiet when close up. Or change the pitch on distance, both ascending and descending.

I've also got thoughts about using continuous triggers in driving games as indications of left/right and faster/slower - but I have lots of thoughts, most of which should be ignored, lol.

The final trigger type is what I call a modifier, and this is used to significantly adjust the playback state of a continuous trigger. For example, if I had a continuous trigger that got louder as the players got closer together, but I wanted to know when the opponent was crouching so I could give them a delicious overhead attack, then I could set the modifier to do something like halve the continuous triggers playback speed or volume on opponent crouch (the state of the opponent would have to be known - for SF4 this is already done).

Finally, thanks for those links CAE_Jones - I'll check them out tomorrow!

Cheers!

Thumbs up +1

20

Actually, just out of curiosity I took a quick skim of those mario memory maps.

That's perfect! You don't even need a pointer chain, just a single offset and there you go! It'll likely be the same for stacks of nes/snes/master system/genesis etc games.

But, this doesn't mean that everything will come easy. Lets say you wanted to play the very, very first mario game. The one where you either jump on top of turtles to smash 'em, or jump up into the pipes to knock turtles onto their backs then walk into them and it's all single screen. How could that game state be represented as audio? You could do proximity for 'enemies', or a different alert for 'enemies' that might be above you. But what about navigation? That is, jumping from the floor onto a platform, then from that platform up the one above and so on?

I'm not pointing any fingers or anything, I'm just saying that even if you had access to all the information that the game was aware of, there can still be difficulties in presenting that information to the user through audio alone.

Anyways, it's food for thought big_smile

Thumbs up +1

21

this sounds interesting. I've been thinking of  a lot that could be done with this already today. and yeah I'm sure this thing could make those snes and genesis games more playable. not sure how this would work for newer games though. so would it be possible for this thing to tell you if you are about to fall off a platform and die and things like that? or would it only be good for health or possibly ammo. either way it sounds great, and I can't wait to see where this goes.

Thumbs up

22

Hi.
[wow]... This sounds totally amazing and unbelievable... smile
A few minutes after having read the first pos, I already started to imagine a tons of games which would become much more playable...
i have a few questions:
1. Can this tool help by informations when navigating a huge map? Maybe directions, play a sound if there is a pit in front of you, scan for platforms around you in a platforming game etc.? There are so many games which would be playable if the navigation was more playable.
2. Can it read text in menus etc.? I think this got mentioned somewhere in the topic, but I didn't fully get how that works.
3. Can it help by aiming in a shooting game? I'm not thinking about FPS shooters, but games which have a few shooting parts where you are forced to aim and shoot a few times.
I'm looking very much forward to see this... I just hope the tool is easy to use for most people, or the possibility to share game configurations in profiles or something simular. I could imagine a small team of people who are working together to make a game fully playable with this tool, and then they wanna share it in a profile kind of configuration file to make it easier for people to use.
Keep up the fantastic job. I look so much forward to hear more about this...

Best regards SLJ.
If you like the post, then please give it a thumps up.
Feel free to contact me privately if you have something in mind. If you do so, then please send me a mail instead of using the private message on the forum, since I don't check those very often.
Happy gaming... :D

Thumbs up

23

I can't wait to get my hands on the source code of this thing! The possibilities of this application -- if it does come out, no offense -- are limitless!

"On two occasions I have been asked [by members of Parliament!]: 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out ?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."    — Charles Babbage.

Thumbs up

24

[wow], I'm super excited to see just what this could turn into. We live in such an exciting time.

James

Thumbs up

25

This sounds absolutely amazing.
I don't have a lot of mainstream games, but the couple I do have would be so much more fun with this.

Thumbs up