26

Re: navigation. That is a long and dynamic topic. The original Mario wouldn't be too hard with this system, since the layout is always the same excepting the POW block. The conditions would be a wee bit more complicated, but if you can use distance between players, it shouldn't be too hard to handle distance from one of the four gaps in the platforms. Something less static would be more challenging (I've asked people at Sonic Retro about how level layouts are stored and didn't get a clear response). In the case of emulators, if the ROM is fully loaded into its own block of memory, things might get easier, but IDK.
The way I imagined this initially focused on the ability to pole player position and level number, so as to allow people to create something of an audio map of each level. I suppose this might look like subconfigs for each map, which does seem kinda cumbersome. And, well, it'd require people to go through and create loads of maps, unless it becomes possible to scan wherever layout data might be found in a vOICe-like fashion. (The vOICe itself is not so great for this sort of thing, but having the ability to scan the most important information directly rather than through a visual filter would clean things up considerably, IME.)
So, yeah, it's complicated and confusing and the closest thing to a standard for these things is Bokurano Daibouken 3 (side-scrolling) and maybe Paladin of the Sky (top-down) or Swamp (first person). And those aren't really standards so much as among the most successful audio games in each category. Something as complex and time-sensitive as Sonic would be a whole other can of worms.

Some of my games
Keep up to date by following @Jeqofire on twitter!
Ear Ninja?

Thumbs up +1

27

[wow]. This could revolutionize main stream gaming for the blind.

Thumbs up

28

[wow]! Just...[wow]. This sounds absolutely fantastic. Can the software be used to match games like Yu-gi-oh? It would be nice to add triggers, pointers or whatever to the cards, so you know what you're playing. This would make a lot of oldschool rpgs playable to say the least. A huge thanks for bringing the concept.
Best regards, Haramir.

The true blind is the one who refuses to see.

Thumbs up

29

Will there be any games released for this app in basic?

Raduvay se, raduvay
Raduvay se domaki ne
Kolko liste po gorach
Tolko zdrave na taz kyshcha

Thumbs up

30

[wow], sounds realy fantastic. Do I understand that right that this software can make mainstream games fully accessible? And if yes, were are the limits of it?

Thumbs up

31

I hope we'll be able to play Chrono Cross with this.

Thumbs up

32 (edited by r3dux 2017-10-06 07:02:18)

SLJ wrote:

Hi.
[wow]... This sounds totally amazing and unbelievable... smile
A few minutes after having read the first pos, I already started to imagine a tons of games which would become much more playable...
i have a few questions:
1. Can this tool help by informations when navigating a huge map? Maybe directions, play a sound if there is a pit in front of you, scan for platforms around you in a platforming game etc.? There are so many games which would be playable if the navigation was more playable.
2. Can it read text in menus etc.? I think this got mentioned somewhere in the topic, but I didn't fully get how that works.
3. Can it help by aiming in a shooting game? I'm not thinking about FPS shooters, but games which have a few shooting parts where you are forced to aim and shoot a few times.
I'm looking very much forward to see this... I just hope the tool is easy to use for most people, or the possibility to share game configurations in profiles or something simular. I could imagine a small team of people who are working together to make a game fully playable with this tool, and then they wanna share it in a profile kind of configuration file to make it easier for people to use.
Keep up the fantastic job. I look so much forward to hear more about this...

Hi SLJ,

To answer your questions:

1. Can it help to navigate a large map? Um, maybe? This is a bit of a tricky one because while we can access any elements of the game state, what's the best way to represent this information as audio? I think a feedback scheme could me made that reads the tiles infront of the player and may be able to give feedback on if there's a hole or pit (for example, a series of notes that rise in pitch as you get closer to it), but I think there's a lot of trial and error ahead finding out what kind of audio feedback schemes work and how effective they can be.

2. Yes, it can read out menus, but perhaps not how you might think. Some menus may have actual text, but others may be graphics with the words written on them. However, because the game needs to know what item in the menu is highlighted, we can find where the game keeps track of this (typically as an integer going from 0 number-of-menu-items minus 1) and then when that value is 4 we might say "Options", or when its 5 we might say "Exit" or such.

3. Can this help aim in a fps? No, that's not really what's it's designed for. Although I guess you could use a separate aim-bot if you had one available and wanted to use it for a given game. Also, I'd be reluctant to add any facilities to the software that might genuinely be construed as 'cheating', because games may then look out for the software and perhaps decide to close down rather than continue with cheat software running.

Sharing configs is deliberately very easy as each config has its own folder with a config.xml and any samples it uses. If I wanted to share a config with you I could just zip up the folder, send it to you or give you a link, then you extract it into the Configs folder of SoniFight then either launch SoniFight or click the refresh button to pick up the new directory, then you can select and run it. Easy as that! :-)

And absolutely, finding pointer chains is something that's easily broken up amongst people so you can colloborate on finding chains for a few different aspects and then combine them into a single config.

Cheers!

Thumbs up +1

33

CAE_Jones wrote:

Re: navigation. That is a long and dynamic topic. The original Mario wouldn't be too hard with this system, since the layout is always the same excepting the POW block. The conditions would be a wee bit more complicated, but if you can use distance between players, it shouldn't be too hard to handle distance from one of the four gaps in the platforms. Something less static would be more challenging (I've asked people at Sonic Retro about how level layouts are stored and didn't get a clear response). In the case of emulators, if the ROM is fully loaded into its own block of memory, things might get easier, but IDK.
The way I imagined this initially focused on the ability to pole player position and level number, so as to allow people to create something of an audio map of each level. I suppose this might look like subconfigs for each map, which does seem kinda cumbersome. And, well, it'd require people to go through and create loads of maps, unless it becomes possible to scan wherever layout data might be found in a vOICe-like fashion. (The vOICe itself is not so great for this sort of thing, but having the ability to scan the most important information directly rather than through a visual filter would clean things up considerably, IME.)
So, yeah, it's complicated and confusing and the closest thing to a standard for these things is Bokurano Daibouken 3 (side-scrolling) and maybe Paladin of the Sky (top-down) or Swamp (first person). And those aren't really standards so much as among the most successful audio games in each category. Something as complex and time-sensitive as Sonic would be a whole other can of worms.

Yeah, it's a tricky one. For single screen games a single audio-map type thing might be okay, but for scrolling games that would quickly turn into a lot of work. Or even games which had a large number of single screens (bubble bobble, for example).

I hadn't heard of vOICe, so I'll take a look into that shortly, as well as the games you mentioned as best-in-class for accessibility.

Thanks for your feedback and ideas!

Thumbs up

34

Niklas wrote:

[wow], sounds realy fantastic. Do I understand that right that this software can make mainstream games fully accessible? And if yes, were are the limits of it?

Fully playable would be a stretch. It may be able to make them more playable / accessible - but that might still be a long way off from fully playable, based on the type of game and the configuration.

It's pretty early days in terms of figuring out what kind of feedback schemes are useful / effective at the moment.

The software was originally designed to make fighting games more accessible, and I'm confident it'll be able to do a pretty good job of this. But if you think of a first person shooter, how would you repsent the enviroment as audio? We can know where the player is located, and their health and ammo or such, but how would could the environment be conveyed to the user as audio? It's a really difficult question to answer.

Some games have navigational aides (I think Dead Space does, and Bioshock or Bioshock 2?), where you can bring up a line on the floor that guides you to your next objective. It may be possible to provide sonification cues for that to help quide the player, but I can imagine that being a fair bit of work to implement.

So no, it's not a magic bullet that can instantly make mainstream games fully playable, but it's hopefully a step in the right direction.

Cheers!

Thumbs up

35 (edited by r3dux 2017-10-06 01:30:45)

devinprater wrote:

I hope we'll be able to play Chrono Cross with this.

Hmm, maybe. There'd need to be a way to indicate branches in the paths as you navigate. The combat system would probably be do-able as its basically a fancy menu where you choose options, and the text could likely be found and sent to a screen reader.

It would probably take some work, but you could definitely sonify a fair swag of it. Whether the config could be made comprehensive enough to enable you to play the entire game without any sighted assistance I'm not sure, as I've personally never played it and just took a quick skim of footage videos to be able to reply.

So the answer's a firm 'maybe' ;-)

Thumbs up

36

nuno69 wrote:

Will there be any games released for this app in basic?

I think there's a slight misunderstanding here. SoniFight doesn't run games in basic or any other language, it merely connects to existing games and plays sounds based on changes in the game-state of whatever game it's connected to.

Thumbs up

37

Haramir wrote:

[wow]! Just...[wow]. This sounds absolutely fantastic. Can the software be used to match games like Yu-gi-oh? It would be nice to add triggers, pointers or whatever to the cards, so you know what you're playing. This would make a lot of oldschool rpgs playable to say the least. A huge thanks for bringing the concept.
Best regards, Haramir.

Yeah, I think a config could be made for something like Yu-Gi-Oh. However, I've never played it myself and only just took a quick skim of gameplay footage on YouTube so there may be some other aspects of the game that might not lend themselves as easily to sonification. But yeah, I reckon that a config that announces the cards and/or their properties would be do-able.

Thumbs up

38

@r3dux, in case you didn't know, there's a rule about not posting multiple posts in a row. It's not always obeyed, but I wouldn't recommend pushing your luck -- the mods might get you big_smile. The best way to minimize is to reply to everyone in the same post. (There's an edit link below every post you write.)

"On two occasions I have been asked [by members of Parliament!]: 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out ?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."    — Charles Babbage.

Thumbs up +1

39

Ah, thanks for the heads up. I'll combine replies as you suggested in the future.

Thumbs up

40 (edited by hadi.gsf 2017-10-06 04:50:14)

Hey, Amazing stuff!

Do you think that  it'd be possible to make certain aspects of EA's  Fifa games accessible? e. game statistic being read to you after  each half or match, or the tactic screen being read to you?
Not sure how hard is to access that sorta  things in  newer games.

John Petrucci Fan all the time.
twitter: @hadirezae3
skype ID: hadi.gsf7

Thumbs up

41 (edited by r3dux 2017-10-06 06:14:34)

hadi.gsf wrote:

Do you think that  it'd be possible to make certain aspects of EA's  Fifa games accessible? e. game statistic being read to you after  each half or match, or the tactic screen being read to you?

While I can't give you an unequivocal yes or not, I'd say that there's a very good chance that yes, that would be possible.

You might end up with a separate pointer chain to each statistic, but its very likely that these values will all be very close together in memory so when you have a chain to the first stat, you may be able to simply adjust the final offset to get to the next, and the next and so on :-)

Update: Actually, after thinking about it more - even if you can't actually highlight each given stat to have it read out, then it would still be possible if a condition could be found that indicates we've moved to the stats screen, and then the screen reader output could be a string like this:

"Possession %: {10}, Shots on goal {11}, Free-kicks taken {12}"

and so on. Where the numbers within the curly braces specify the ID of the watch that points to that particular statistic.

Thumbs up +1

42

@r3dux: Thanks so much for all your replies.
Regarding navigation on maps, there are a lot of ways to make this more playable.
1. It would help very much if the software can help by making a compass which tells the player in which direction the player is phasing. Just a voice saying north, east, south and west. I have a more advanced compass in mind which would be interesting to give a try, depending on what's possible.
2. Audio will help a lot when navigating maps, jumping over pits etc. An other way to give a better idea of a map is by having a camera, virtual cursor or what you would like to call it, which let you navigate across the visible map. Maybe pause the game, switch to the software, open up the virtual map and then you can navigate using the arrows or other commands to get an idea of what's around you. Maybe a sound which means free space on the tile the cursor is on, and an other sound which means an obstacle such as a wall or anything else. I think that would help very much regarding to navigating maps.

Best regards SLJ.
If you like the post, then please give it a thumps up.
Feel free to contact me privately if you have something in mind. If you do so, then please send me a mail instead of using the private message on the forum, since I don't check those very often.
Happy gaming... :D

Thumbs up

43

@SLJ - you're welcome!

1.) I hadn't thought of the compass way of doing things. If you think of a typical first person shooter, then it's quite possible that the player's location and their pitch (i.e. looking angle up or down) are stored as two 'vec3's - that is, x/y/z for the location and another three values stored as either degrees or radians indicating rotation (i.e. pitch and yaw).

A trigger could be made that then says North, North-East, East or such as the player rotated into each 45 degree segment. This still might not be all that much use for navigating the game the world though as you still might not know about ledges, drops, lifts and the such - but it might a good start. It could also be that the yaw/pitch/roll values are stored as quaternions, which would may possibly make things a little trickier, but likely nothing that couldn't be resolves with a few cups of coffee and a maths text book.

2.) The software's not really geared towards visualising game worlds, but in later versions of the software it may be possible to place audio 'beacons' into the game world. The irrKlang audio library allows you to create a 3D sound beacon at a given location, and update your location relative to it, then the sound will play in a spatialised manner relative to your location, and you can specify the sound drop-off distance parameters and such. It may be possible to doppler shift the playback rate based on your speed in relation to the beacon if I'm understanding the irrKlang docs correctly.

To do something like this you'd need a workflow such as:
a.) Find a pointer chain to your players position,
b.) Go to some in-game feature such as a lift or a button or pressure-pad or such,
c.) Write down the coordinates of the feature and what it is.
d.) Create a beacon trigger set at the feature coordinates and with the sound, which would have to be looping, you want played (note: beacon triggers don't exist yet, but if I was going to add this facility that's likely what I'd call 'em).
e.) Play the game with the beacon active and it'll provide audio feedback based on the relative positions of the player and the beacon.

As far as I'm aware nothing I've suggested sounds outside the realms of possibility of what could be achieved, and your ideas are really good ones, but that functionality isn't available at present and would definitely have to be a feature added in a later release. As the software will be open source it might not even be something I end up implementing - many hands make light work and all that - but if people do modify and improve the software they can always issue a pull request to merge their functionality back into my version, or they could keep it in their own fork of the software, it's entirely up to them.

Thumbs up +1

44

Just wondering how this thing would work with sega games like double dragon, the streets of rage trilogy, the golden axe trilogy, moonwalker, Kenseiden, alix kid.

I think most of those are side scrollers or platformers.

There are many more sega games in my collection, these are just a few that come to mind.

If I list the ones I have, we'd be here for quite some time.

I don't think I quite have the entire master system collection, but it is a fair bit.

There was a time when I was into playing some of these with an emulator, but I gave up trying to find emulators that would play nice with a screen reader.

Trying to configure them to work with a joicestick can be tedious.

Thumbs up

45

Genz and Kega Fusion always worked fine for me. Fusion had better sound last time I checked. My SoR2 access program is based around Genz/Kfusion savestates, and it seems like R3dux's program can do the same thing, but more cleanly, since it can access memory directly.

Some of my games
Keep up to date by following @Jeqofire on twitter!
Ear Ninja?

Thumbs up

46

r3dux wrote:
hadi.gsf wrote:

Do you think that  it'd be possible to make certain aspects of EA's  Fifa games accessible? e. game statistic being read to you after  each half or match, or the tactic screen being read to you?

While I can't give you an unequivocal yes or not, I'd say that there's a very good chance that yes, that would be possible.

You might end up with a separate pointer chain to each statistic, but its very likely that these values will all be very close together in memory so when you have a chain to the first stat, you may be able to simply adjust the final offset to get to the next, and the next and so on :-)

Update: Actually, after thinking about it more - even if you can't actually highlight each given stat to have it read out, then it would still be possible if a condition could be found that indicates we've moved to the stats screen, and then the screen reader output could be a string like this:

"Possession %: {10}, Shots on goal {11}, Free-kicks taken {12}"

and so on. Where the numbers within the curly braces specify the ID of the watch that points to that particular statistic.

This is very very exciting for some of us. (possibly all who already replied to this topic). I play games like Mortal Kombat, Street Fighter, Fifa, NBA. So this software can potentially make all of those games more accessible, even by small amounts.

I have a question though. How hard would be for non-technical users to track down those variables in  the process memory, then make the screen reader announce them?
Things like the mentioned fifa game stats, character meters such as health and stamina,  character mouse movement (such as up and down and left and right). Do i need to learn how to script for  this?
cheers.

John Petrucci Fan all the time.
twitter: @hadirezae3
skype ID: hadi.gsf7

Thumbs up

47

@kool_turk - that's a good question, I'm not quite sure what kind of audio feedback scheme would work best for side-scrolling games. It's something that's going to have to be experimented with. I'd be interested in any feedback CAE_Jones might have on the feedback scheme used by their save-state based Streets of Rage accessibility program. Also, be aware that support for emulated games has not been implemented at present but is theoretically possible.

@hadi.gsf - Finding the pointer trails is definitely the most troublesome part of the process, and while there are techniques / strategies that can be used I'm not entirely sure to what extent users with visual impairment will be able to perform this process. My gut feeling is that it's do-able, but it might be a bit of a pain in the neck.

I'm currently looking into some alternative techniques rather than the present 'brute force' approach where you find a value, generate a list of pointers that point to the address of that value, quit and restart the game, find the value again, and finally filter the pointer list to only keep those pointer chains that now point to the new location of the value. This 'whittling down' process is repeated until you either have just one, or only a small number of chains - one of which should be able to consistently locate the value of interest.

As a source of pointer chains I've also started thinking a little about trainers - not the shoes, but the ones that give you infinite health or ammo and so on ;-) My present understanding is that for something like an infinite health trainer, the address of the instruction that modifies the health value is located via an array of bytes search, and then the actual code that decreases the value is replaced with an instruction which doesn't such as a no-op. So if a trainer contains the location of the code that alters the value, then the code that performs that action must know where the value to be changed is located. Unfortunately my assembly language is pretty weak at the moment, but I'm hopeful that something along these lines could be used to extract the locations of values from trainers based on Cheat Engine cheat tables.

I'm not quite sure if that will work out or not, but it seems reasonable to me that it could - so I'll certainly look into it.

Thumbs up

48

One of the ideas I had for identifying position/orientation variables is to pay attention to user input. So, if there is a value that consistently changes either liniearly, or with constant acceleration, when a directional button is held, it's a strong candidate for position, angle, or velocity. If it's the sort of game where it's practical to look at the screen directly, it'd be possible (if horribly inefficient) to do some kind of motion-detection, and notice when regions of motion tend to correspond with changes in pairs or triplets that could be coordinates or velocity, and eventually refine this with more data until there's a usable model of where object positions are found.
With modern stuff, you'll run into some common or very similar physics engines, some of which will neatly track objects such that finding the necessary chains would be fairly easy. Or, at least, I should hope so. If it's not C/C++/assembly, memory allocation gets "fun". So the more information on what was used to develop any particular game, the more techniques for hunting down the most relevant information can be refined.

SoR2 is extremely low in complex layouts (at most, there are 3 levels with diagonel sections), so my program just tracks the player, enemies, items, and a few other variables like screen/settings/time. Item detection is kinda unreliable, as it's bad at distinguishing active items from uninitialized junk, but it has no trouble finding enemies. The biggest weakness is probably the refresh rate. It offers two ways to get information: examine the enemies via tts, and just get names/numbers directly, or loop a sound at their position (iirc, individual enemies can be muted if the noise outweighs the usability). The sound files are easily replaced, and there's a default for any enemies or items that don't have associated files. It uses pitch-bending for y position, but that's because I prefer clarity to realism tongue most people seem to prefer the opposite.
I've tried accessifying Mario and Sonic, also, but couldn't really use a single method for both. And in both cases, I wound up trying to rebuild the games from scratch (there was a Sonic 3 savestate reader, but it was way less developed than the SoR2 one). Mario is simple enough, with enemies playing sounds as they move, and sweeping ahead every time Mario moves to a new x position (measured in blocks, because anything else would be overkill). The sweep could be set to find blocks above, and drops by counting the distance to blocks below. It only looks ahead... 5 blocks, I think? It also identifys glowing blocks, pipes, and coins with sounds during the sweeps. (my coin sound is kinda terrible and I should have just pitch-bent the normal coin sound sad ). In addition, there's a cursor which the player can move around block-by-block, to hear what's there, and it has a keystroke to scan the column it's in to more quickly find platforms/drops. The cursor does speak the name of any enemies or powerups it passes over. Moving platforms play a distinct sound to indicate which direction they're moving. And that more or less covers it.
Sonic was harder. I tried breaking up the track into linear segments, straightening out any curves, and have a sound play to indicate the angle of a section when Sonic gets close. There was also a way to have Tails fly along the track, to give a more continuous idea of its shape. There was also an object list to give raw information to screen readers, pits all played 16-bit noise on a loop, enemies and rings made sound in accordance with their animation, and pitch-bending for y was a togglable feature. This did not work past the second act of Green Hills Zone.
Bokurano Daibouken 3 uses something like an improved version of what I did with Mario, and includes a cursor, an object viewer for raw info, echoes player footsteps from the nearest ledge, and can scan ahead every second or so for the nearest blocks ahead and above.
So anything which can be neatly divided into rectangular grids almost has a standard, which can be adapted for 3d. More complex layouts are harder and I keep coming back to a vOICe-like model of tonal scans. I don't see this program supporting that sort of thing very easily, but more discrete scanning like BD3 seems manageable.

Swamp has a very cane-like scanning, which can be set to the direction of interest as needed (ex, if you're looking for an opening in a wall to your left, it can just scan a few rows to your left). I feel like something like this would make a ton of 3d games far more playable, if not completely so, much like a cane makes the real world more accessible. To the extent that someone could almost certainly slap together a generalizable version for Unity in an hour or 3. But I'm not familiar with Unity's approach to physics, so I could be horribly mistaken.

This brings us back to the question of what this software can do smile. I expect the ability to scan for things wouldn't be too difficult once one has a good map of the way the game stores layouts, so stereo pan and the ability to put this on a timer would cover most of it. Going full vOICe would require rather different features than what this is designed around, I think.

Some of my games
Keep up to date by following @Jeqofire on twitter!
Ear Ninja?

Thumbs up

49

Wondering if this can be done for steam games like yugioh legacy of the duelist? Would give it a perfect read instead of 90 percent, so just wondering. I mean I don't think I'd complain if not, but I figured it might be possible just because it's all pretty constant/pretty menu based.

Silly Gohan, animals don't eat people. People eat animals.

Thumbs up

50

@CAE_Jones - Fantastic reply and lots of great ideas! I've got this gut feeling that while tracking game movement by tracking input might work at first, it would end up drifting out of sync in pretty short order due to changes in framerate. Also, there might be issues of feedback delay from mice/keyboards/joysticks and O/S versions that would make it a bit of a nightmare.

I was thinking about how your save-monitoring sonification worked, because I don't know for sure. My best guess would be that the save-state only has the level, the players, current score, health and any enemies saved, and maybe some additional housekeeping -  so it's a relatively small amount of information to parse and the file format for saves has likely already been documented? Then you just trigger game saves maybe once a second in a background thread and parse the save data to provide sonification? if so then that's definitely thinking outside the box and I whole-heartedly approve! big_smile

That cane-like scanning idea is superb - as soon as I read it I was thinking about different ways it could be implemented. I may well follow up on this, but right now I have to get SoniFight ready to ship. I thought it was going to be ready two months ago, and then I completely re-worked the audio logic and added features. Definitely in feature freeze now and aiming for end of October!

I still haven't looked at vOICe, but I assure you it's on my to-do list.

@death - I've never played Yu Gi Oh Legacy of the Dualist, and you don't mention what aspects are and aren't a good 'read'. So, while I really don't want to over-promise and under-deliver, from looking at a quick skim of gameplay footage I'd say it's quite likely that SoniFight could help with sonification of this. But please do be aware that to provide sonification for any given game a suitable config needs to be made.

Thumbs up