2018-11-14 08:48:54

Hi.
@JLove: It works with Blazblue, Street fighter, Killer Instenkt, Mortal kombat and some other games which I haven't tried.
@r3dux: Thanks so much man. I'll check it out as soon as possible. Thanks for all you're doing.

Best regards SLJ.
Feel free to contact me privately if you have something in mind. If you do so, then please send me a mail instead of using the private message on the forum, since I don't check those very often.
Facebook: https://facebook.com/sorenjensen1988
Twitter: https://twitter.com/soerenjensen

2018-11-14 08:57:34 (edited by JLove 2018-11-14 09:03:50)

So according to the read me, it says this works with killer instinct, but it says, "(64-bit proof of concept only, Windows Store version).  Proof of concept?  Does this mean that the config is incomplete?  The problem that I see with us making our own configs if we're blind is that, if I remember correctly, cheat engine was not accessible last time I looked at it.  In other words, I couldn't tell what memory address went to what.  On a side note, purely theoretical, couldn't this same thing be applied to other mainstream games?  For example, create a config that could read the menus or story lines from a mainstream game, or could tell you what certain items were in games that weren't fighting?

2018-11-14 09:26:33 (edited by r3dux 2018-11-14 09:28:12)

@JLove - The KI config is proof of concept, which means it's very incomplete.

KI is a 64-bit process, and originally SoniFight could only work with 32-bit processes, so I worked on it to the extent that it could work with 64-bit processes and started a KI config, but I only got as far as some of the menus because KI uses internal memory obfuscation to avoid hacking. In essence, the memory address for health and guages etc doesn't stay in the same location - they're constantly on the move.

While it's likely possible to track the pointers that point to the memory addresses of health etc, I don't have the technical skills to do so. In fact, I've seen cheat engine "cheat tables" that track such things, but they attach a debugger to the killer instinct process, and that's something I really don't want to do.

You're right when you say Cheat Engine is not very accessible for VI users. The only way I can see that being remedied would be if I forked CE, added the screen reader hints and made a pull request. This would be a significant undertaking in terms of time and effort. And even then, blind/VI people would need the time and inclination to find pointer chains for games.

In answer to your final point regarding SoniFight being applied to other games - absolutely. That's why I wrote it to be config based. There are already configs for the point-and-click adventure games Day of the Tentacle (Steam version) and Beneath a Steel Sky (Good Old Games version) that come with it.

SoniFight is meant to be configurable to assist with most games. It may not make something that was never designed to be playable without sight 100% playable, but it can certainly help. Unfortunately, the people who would benefit most from configs are the people who have such a difficult time creating them. And to be honest, even if you were fully sighted but not in any way technical it would be difficult.

So, does SoniFight work in theory? Yes.

Does it work in practice? Yes.

Is it easy to create configs for games? No.

And that last one's the kicker.

2018-11-14 10:03:05 (edited by JLove 2018-11-14 10:17:24)

So, I just tested KI with Sonifight.  I have Win 10, running the MS store version.  I ran the KI config, and it's not even reading main menus. I just double checked under "edit config," and I see triggers for the main menu, but my screen reader (tried both JFW and NVDA) doesn't read them.  I found the following text window.  From what I can see, it looks like it should work, but it's not.  Text might not be spelled correctly due to OCR, but it is overall readable, and the jist can be understood:
irrK1ang sound library version 1.5. e
Loaded plugin: ikpf1ac_64. dll
Loaded plugin: ikpmp3_64. dll
Using DirectSound8 driver
irrK1ang sound library version 1.5. e
Loaded plugin: ikpf1ac_64. dll
Loaded plugin: ikpmp3_64. dll
Using DirectSound8 driver
The current culture is:
The current UI culture is: en-US
Returned dir: Killer Instinct (Windows Store 64-bit)
irrK1ang sound library version 1.5. e
Loaded plugin: ikpf1ac_64. dll
Loaded plugin: ikpmp3_64. dll
Using DirectSound8 driver
irrK1ang sound library version 1.5. e
Loaded plugin: ikpf1ac_64. dll
Loaded plugin: ikpmp3_64. dll
Using DirectSound8 driver
irrK1ang sound library version 1.5. e
Loaded plugin: ikpf1ac_64. dll
Loaded plugin: ikpmp3_64. dll
Using DirectSound8 driver
About to read config: .\Configs\Ki11er Instinct (Windows Store 64-bit)\config.xml
Attempting to connect to process: KillerInstinct_UWPx64_r
. Tolk: The active screen reader driver is: JAWS
Tolk: This screen reader driver supports speech.
Tolk: This screen reader driver supports braille.
... Found process base address at hex address:
7.ff74475 eeee
So, it seems like it found everything it should have, but is not reading.  Just thought you should know.

2018-11-14 10:52:55

@JLove - it's likely the KI executable has been updated since the config was created around mid-2017 and the addresses have changed. I'll look into it and get back to you.

Also, thanks for the log-output - at least we now know it's connecting to the process, finding the screen reader and nothing is going wrong with the sound output libs.

2018-11-14 12:55:43

@JLove - I can replicate and confirm the KI lack of sonification. I've found the updated chains, but the new Single Player time-limit is infinite - it's not 99 seconds anymore, so there's no easy way to tell when you're in game or in the menus, which leads to some innapropriate sonification events.

Anyway, rather than getting you to change files I've pushed SoniFight v1.1.4 with the updated KI menu and clock chains.

Link: https://github.com/FedUni/SoniFight/rel … v1.1.4.zip

2018-11-15 05:03:27 (edited by JLove 2018-11-15 06:28:21)

So, I deleted the folder where I had V1.1.3, and downloaded and extracted 1.1.4.  Tried again, and this time the main menu will read, but not menus within menus.  Were the secondary menus within menus completed?  OCR half works here.  Also, why do I only see one config?  The only config I see in the config folder is for KI.  I thought that Sonifight had other configs that came with it automatically?  On a side note, you know, for OCR to be natively built into Windows now, it sure does have difficulty reading shit correctly.  Ugh.

2018-11-15 07:52:10

@JLove - The KI config is very bare bones as we've previously discussed - it's just the main menu and clock / round-timer alerts at present.

The reason there's only one config displaying is that KI is a 64-bit process - and for SoniFight to be able to connect to it SoniFight itself needs to be 64-bit - which is why the release ships with two versions: 32-bit (x86) and 64-bit (x64). Only matching "bitness" processes can communicate.

There is only one 64-bit config and that's the KI one. If you launch the 32-bit version of SoniFight then you'll see the rest of the configs for Street Fighter 4, Mortal Kombat 9, BlazBlue etc.

All configs are stored on a per-folder basis in the "Configs" folder right next to each SoniFight executable (it's just a config.xml file along with, optionally, some audio samples).

2018-11-15 08:49:55 (edited by JLove 2018-11-15 09:00:19)

Ok, some questions about memory addresses.
1.  When a game is launched, are all menus and associated choices preloaded into memory, or does this happen on the fly, i.e., the secondary menu under single player only loads into memory if that option is chosen at the main menu?
2.  Does an entire menu reside in a single memory address, or is each choice within a menu stored in a different address, i.e., menu choice "single player" at memory address pillows, menu choice "multiplayer" at memory address billows, and menu choice "options" at memory address willows?
3.  Does the active memory address change as you highlight different options in a menu, or is a separate memory address checked when you highlight different things?
4.  If a choice is made in a main menu, is the memory address/addresses  that hold the choices of the main menu now updated with the choices in the subsequent menu (i.e., if memory address pillows contained the "single player" choice, and you chose that, is memory address pillows then overwritten with the new menu choice), or were the subsequent menus given unique addresses when game loaded?
5.  Are menu selections within memory addresses stored as strings, i.e., "multiplayer," or are they stored as some sort of numerical code that represents those characters?
6.  When searching for memory addresses with something like cheat engine, is it possible to search for strings of text, like "single player," etc.?  If not, then how do you find memory addresses that store pieces of text, so that you actually know you're reading choices from a menu?

2018-11-15 09:27:37

Some answers about memory addresses:

1 - Are menu addresses preloaded or loaded on the fly

Every menu option must have a number, so for example, the very first item in any menu is typically identified as option 0, then below would be option 1, then below that would be option 2 etc. The menu options don't need to be "together" because this single value will track the currently selected menu option. However - submenus will commonly use a different memory location to track their submenu currently selected option.

Because there's typically only a few levels of menu these menu tracking values are created on launch and kept throughout the lifetime of the process.

2 - Does an entire menu reside at a single memory address (pillow/billow/willow)

Haha, pillows/billows/willows. You're getting the hang of it. No, the currently selected menu option is a single value. Submenus are also single values but may be stored at a different memory address. In fact, this is the entire reason behind the idea of "dependent triggers" in SoniFight - because if the same memory address is used to track the currently selected option of a main menu AND a sub-menu then the sub-menu would sound like the main menu. By using dependent triggers we're trying to find the "level" of menu that we're on (0 for main, 1 for pillow, 2 for billow etc, haha).

Dependent triggers are discussed further in the SoniFight user guide.

3 - Does the active memory address change as you highlight different menu options.

Nope - only the the value of the currently selected menu option changes, the memory address remains constant (thank heavens).

4 - What happens when we go from a main menu to a sub-menu (paraphrased)

When you select a menu option something happens, but precisely what that thing is depends on the option selected. If we're going into a submenu because you selected "Options" then that's what happens, if you select "Arcade mode" then you'll go into the character select screen etc. There is generally no pattern to this - so the same currently selected option memory address may be used if we're going into a submenu, or it may not.

5 - Are menu selections strings

Well, they can be - but typically they're numerical values as discussed above. However, the Street Fighter 4 config DOES use string matches for menus in a way because each menu item has a "subtext" which explains what that item is. For example "Arcade mode" might have the subtext "Play against the CPU in a series of battles". This is actually non-desirable because users may not be playing the game in English, or may have a different default "culture" - that is, the native language and keyboard layout.

When I was doing the SF4 config I was relatively new to the game and I couldn't find the menu values numerically so I was forced to use the subtext to identify options, although if I tried again I would increase the pointer hop max spacing from 2K to 4K in Cheat Engine and I'm sure they'd turn up.

If you're unsure about what I mean by culture, this Microsoft article will explain all:
Performing Culture Insensitive String Comparisons

6 - Can you search for strings in Cheat Engine

Hells yes. Further, you can search for UTF-8 and UTF-16 strings - and SoniFight also supports UTF-8 and UTF-16 encoded strings in triggers.

7 - There is no number seven, I just wanted to say thank you to JLove for asking the most informed and detailed questions about SoniFight that I've ever fielded. You absolutely get it, and I am delighted that you do. Well done, sir.

2018-11-15 10:05:42 (edited by JLove 2018-11-15 11:05:23)

Ok, I want to make sure that I understand, because chances are I don't.  I'll get it, though, just bear with me.  We're going to keep pillows, willows, and billows, hehahaha.  And, we're going to use KI for this example.  So, just to make sure I understand, main menu is stored at mem address pillows.  "Shadow lords," the first option, is stored at mem address pillows.0, single player at pillows.1, multiplayer at pillows.2, store at pillows.3, options at pillows.4, and exit at pillows.5, right?  If I then choose "single player," which, if I understand correctly, is pillows.1, then I am now taken to a new menu, located at mem address billows.  Options in here would be billows.0, billows.1, billows.2, etc., and pillows.0, 1, 2, 3, and 4 still hold main menu values.    So, Does that mean that we have to set up triggers for each one of those, or do we set up a trigger that only watches pillows and billows?  And how exactly do we find the individual options that are stored within a single menu in cheat engine?  In other words, will cheat engine just show us pillows, or will it show us pillows.0, 1, 2, 3, and 4?  Or, now that I am writing this, and I am thinking about it, I think that maybe I went about looking at it the wrong way, from a programming perspective, assigning each option sort of like a property would be.  Now that I think about it, that's most likely wrong.  Instead, it probably works like this.  The main menu is stored at memory address pillows.  The value at address pillows is currently 0, because "shadow lords" is highlighted.  However, highlighting "single player" now changes the value stored at mem address pillows to 1.  Highlighting "multiplayer" changes the stored value at mem address pillows to 2, highlighting "store" would make that stored value inside of mem address pillows equal 3, etc.  That would mean that a trigger could be set to monitor the memory address pillows, and when pillows changed value, it would read the associated highlighted option.  If that's how it works, and it's not the first way I thought, then how do we determine which addresses contain a menu, since the value stored in them is always going to be 0 unless you're actively inside of that particular menu?  In other words, all menus are initially assigned to addresses pillows, willows, and billows, but until you're inside of them, actually highlighting options, all of the values will be 0.  So how do you figure out which addresses were assigned to be menus?  And is there a way to tell the trigger to output whatever text is highlighted, no matter what it is, when that value changes.  In other words, if I didn't know that a value of 1 inside of memory address pillows was "single player," then how would I know what to tell a screen reader to say?  The numerical value of 1 stored at mem address pillows does not tell me what the text on the screen is.  It just tells me that the second option is highlighted.  So is there a way to tell a trigger, and I am paraphrasing here, "retrieve value of mem address pillows.  Whatever the highlighted text is on the screen, send it to my screen reader."  Or do I have to know specifically what that text says?  That's if I am understanding this right.  For all I know, I am way, way off course.

2018-11-15 11:11:56

@JLove - Yup! The latter half of your post nails it. To elucidate:

Main menu is stored at memory address pillows. Shadow Lords is at pillows and it has value 0. Single player is at pillows with value 1. Multiplayer is at pillows with value 2 etcetera.

Jumping into a submenu will commonly mean that the currently selected menu option is now at billows, but it works in the same way in that options at that address change but the address stays the same.

With regard to whether we need triggers for each menu item - yes, we do. That's why I added the clone current trigger and clone current watch buttons to SoniFight - because once you've found a menu you have to check if it's 0, or is it 1, or is it 2, or is it 3 etc. I do the first trigger, then clone that trigger and change the value to 1 and change output text, clone and change to 2 and update output text, clone and change to 3 and update output text etc.

The way I find menus is I launch the game and Cheat Engine, then I punt down the menu to, say, the 3rd option, so I'm guessing we're looking for the value 2 (i.e. 0, 1, 2). Search the process memory for the value 2, then move up to the first option (so 2, 1, 0) and then search for the value 0. Chances are that will narrow it down from tens of thousands or millions to just a couple. Wash / Rinse / Repeat until you have only 1 or two memory addresses. Pointer scan for that address. Kill the game, start it up again, re-attach Cheat Engine, check if the pointer chain identified works. If initial process left lots of pointers you re-find the address of interest and then filter the pointer chain list to only include those that point to the new address. That's it. That's the process.

If you look at the BlazBlue or SF4 configs there are over 200 triggers, most of which are just menu options. It's exactly as fun as it sounds to create, haha.

I've previously put together a few videos demonstrating the process live which you can find here:

Cheers!

2018-11-15 11:28:51 (edited by JLove 2018-11-15 11:48:05)

So, is there a way to retrieve what the actual text is?  I'm blind.  I can't see  a menu screen to know that option 1 is shadow lords, option 2 is single player, option 3 is whatever it is.  Is there a way to actually figure out what text is stored at a given address certain value?  If so, this could be used for any game, any menu.  In other words, for the text to actually be printed on your screen, doesn't that have to be stored somewhere in memory?  is there a  way to pull that address and find out what characters are stored there?  In other words, Yes, highlighting it will change the value of address pillows, but when you first launch a menu, all of the options are presented to you, and usually option 1 is highlighted, but you can still read the other choices without highlighting them, so they have to be in memory somewhere.  In that way, we could figure out what text to output.  Or is there a way to give trigger wild cards, so that whatever the text is that corresponds to each value, it will output it?  Being blind, I wouldn't be able to write out specific text for triggers if I don't already know what the options are, like if I were in a game I'd never played before, but it seems to me that those words on the screen have to be somewhere in memory, which theoretically means that they should be retrievable.

2018-11-15 11:47:38 (edited by r3dux 2018-11-15 11:48:39)

If the text cannot be read by a screen reader or OCR'd then the only way to "retrieve what the actual text is" would be for a sighted person to convey that information to you.

You can't read shadow lords, single player, multiplayer etc but you can make an educated guess that they'll be values 0, 1 and 2 in the menu as they are the first/second/third options available.

While the text for menus may possibly be stored in memory it doesn't have to be - it could be a picture of the word (like a jpeg or gif or something).

If the text is rendered as text (not a picture) then it will exist in memory.

There is a way to set trigger "wild-cards" by using the "Changed" criteria. This is used in the Day of the Tentacle and the Beneath a Steel Sky configs. In essence, the item that is highlighted shows the text of that object - so a watch points to the memory address of that changing text, and every time it changes it sends the new value to a screen reader.

Here's a video of the Day of the Tentacle config using the "Changed" criteria in action:
https://www.youtube.com/watch?v=6p62y1-BLtQ

Anything the game knows we can know. It doesn't mean that finding it will be easy, but it does mean that it's possible.

2018-11-15 12:04:04 (edited by JLove 2018-11-15 12:06:54)

So, let's say that I've got a game that I've never seen before.  Let's say that I figure out that the main menu is stored at memory address billows, but I have no idea what the menu items actually say, because I am blind, and my screen reader doesn't read them.  Now, let's say that main menu options are:  Option 1, buy a fluffy pillow.  Option 2, buy a flat pillow.  Option 3, create a custom pillow.  Aren't the actual words "buy a fluffy pillow," "buy a flat pillow," and "create a custom pillow" actually stored somewhere in memory?  Somewhere the words have to be stored, or else you wouldn't see the text on your screen, right?  And, if I write a trigger that says, watch memory address billows, and when value changes, output.  Will it somehow be able to know that when value 0, option 1 is highlighted,  that equates to the string "buy a fluffy pillow," and send that to my screen reader, and when we highlight the next option, it will send the string of text associated with value 1, "buy a flat pillow," to my screen reader?

2018-11-18 12:05:28

If the game renders the actual text that says "buy a fluffy pillow", "buy a flat pillow" etc using a font, then YES - that text will exist in memory.

However, if the game has a picture of the words "buy a fluffy pillow", "buy a flat pillow" etc. then that text will NOT be stored in memory because the information is being conveyed by an image rather than drawing the text.

While the latter case of image-based information can and does occur, it's far more common for the text to be drawn as text so it will be possible to find that text in memory.