2011-10-02 07:09:58 (edited by CAE_Jones 2011-10-02 08:31:08)

How useful do you think it would be to convert some images into text that would convey some of the shapes, objects, Etc when displayed as braille?
Naturally, this would be limited by the fact that braille displays are generally one line. I was mostly thinking in terms of horribly reduced resolution; not depicting great amounts of detail, just maybe pointing out where objects are located in an image. (I thought of this mainly as an assistive tool for visual games. It seems like it'd be easiest to apply this technique to the partial 3d sidescroller / beatup genre, but I could see it being of use for more platform-oriented games, if people are willing to scroll all over the place).

This would probably need to go through a screenreader, meaning it would probably need to be sent in text form, which would probably make it much more difficult to use all eight dots on displays with eight dot cells.
And I wonder how possible it would be to make such a thing work on what's actively being displayed on the screen. Mostly because of the video driver changes in Windows 8.

This would obviously lack the ability to imply shading in the slightest, but just being able to locate objects might be helpful (at least until Aprone finishes his invention of doom wink )

Would this be worth pursuing? Does it seem possible?

[edit] I made a small test program in java that actually seemed to work pretty well on a test image of contrived yellow squares: the text output wound up shaped in braille just like the squares were positioned in the image.
The problem is that I also threw in some teal squares that didn't show up. sad
[/edit]

看過來!
"If you want utopia but reality gives you Lovecraft, you don't give up, you carve your utopia out of the corpses of dead gods."
MaxAngor wrote:
    George... Don't do that.

2011-10-02 08:41:24

I think it is sounds doable, but I don't know how.

I post sounds I record to freesound. Click here to visit my freesound page
I usually post game recordings to anyaudio. Click here to visit my anyaudio page

2011-10-02 15:40:58

CAE_Jones, was your program having trouble reading the image or was it just not sure how to represent it as braille?  How fast do 1-line braille displays refresh themselves?

- Aprone
Please try out my games and programs:
Aprone's software

2011-10-02 18:36:20

I have no idea how fast displays refresh. Sometimes they seem like they can work rather quickly. I think it depends more on the program that's sending information to them.

I suspect the program was having trouble reading the image. Right now, it converts the image to black and white, then scales it down to a tenth the original size. I imagine the issue is with the size of the squares I used, since the ones that showed up were 10x10 before processing, and the ones that didn't show up weren't quite so easy to handle at a tenth the size (25x25 and such).

看過來!
"If you want utopia but reality gives you Lovecraft, you don't give up, you carve your utopia out of the corpses of dead gods."
MaxAngor wrote:
    George... Don't do that.

2011-10-02 20:04:10

If you ever need a hand, I have plenty of experience writing software that manipulates graphics, and I'm more than willing to help.

- Aprone
Please try out my games and programs:
Aprone's software

2011-10-03 02:23:41 (edited by CAE_Jones 2011-10-03 03:00:47)

I imagine that your help would come in handy, since it wouldn't be as useful with my contrived tests as with more complex images that would be difficult for me to figure out.
My last test was just me arranging yellow squares to form the words "I win" in braille. lol The output was correct, at least.
If nothing else, it might help if I could save test images at varying points of processing, and someone sighted could point out what went wrong.
The method for converting to black and white involves spreading error to surrounding pixels, which I imagine might not scale down so well.

[edit] I tried adding filters and inversion, and apparently didn't do it right, since whenever I try to use them the output is just blank space.
I tried it on a screencap of sonic 3, and the background was too bright to make much sense of the results. So being able to invert it would probably help.
Either this if statement is wrong, somehow:
if (((value < threshold)&&(!invert))||((invert)&&(value>=threshold)))
Or I should probably just remove the "error spreading" (dithering?) entirely.
[/edit]

看過來!
"If you want utopia but reality gives you Lovecraft, you don't give up, you carve your utopia out of the corpses of dead gods."
MaxAngor wrote:
    George... Don't do that.

2011-10-03 03:23:02

Would it be easier for you if a separate program converted the picture into black and white, and scaled it down first?  Then your program could read it in and avoid that part of the process.  Making a program that scales and converts to black and white wouldn't take me any time at all, especially since I can see to quickly debug any problems and to check how well it is converting the images.

I don't want to come across as pushy though.  This is your project and I will totally understand if you want to handle each part yourself.

- Aprone
Please try out my games and programs:
Aprone's software

2011-10-03 04:41:21

I wouldn't mind at all.

看過來!
"If you want utopia but reality gives you Lovecraft, you don't give up, you carve your utopia out of the corpses of dead gods."
MaxAngor wrote:
    George... Don't do that.

2011-10-03 04:50:00

I'm assuming that each pixel will represent 1 dot in a braille cell, so a 2 by 3 pixel image would be displayed using a single braille cell.  If this is the case, what size bitmaps are you looking for?  I can work on some sample images for you to work with, and once you're happy with the scale and everything, I'll send you the converter.

- Aprone
Please try out my games and programs:
Aprone's software

2011-10-03 06:46:24

I would hope that the size and scaling could be varied depending on what people would use it for, but it would generally be good to have the processed image no wider than 40px (or 80 at most). On height, or the size of the original image, I have no idea.

看過來!
"If you want utopia but reality gives you Lovecraft, you don't give up, you carve your utopia out of the corpses of dead gods."
MaxAngor wrote:
    George... Don't do that.

2011-10-03 12:34:59

Hello,

If by any chance this project will succeed, then another matter should be dealt with: How could be possible that a player would manage both playing the game and inspecting the graphics that is being output by their braille display at the same time? This, in my opinion, could be very challenging, especially while playing a game that needs very swift and rapid reaction. Just a thought. wink

Standing by the window, eyes upon the moon,
Hoping that the memory will leave her spirit soon.

2011-10-03 13:16:05

Sounds really interesting. However, I think the best tactile graphical view we can get is from a Braille printer. This would be very expensive though.
Refreshable Braille displays are able to change extremely quickly. I don't know how fast per second.
I'm wandering how all this could be useful in games, but I'm very interested to follow the project.

Best regards SLJ.
Feel free to contact me privately if you have something in mind. If you do so, then please send me a mail instead of using the private message on the forum, since I don't check those very often.
Facebook: https://facebook.com/sorenjensen1988
Twitter: https://twitter.com/soerenjensen

2011-10-03 18:33:20

at SplendidFault: I'm still wondering that myself. For now I'm worried about getting something that works before tackling the realtime feedback aspect.

看過來!
"If you want utopia but reality gives you Lovecraft, you don't give up, you carve your utopia out of the corpses of dead gods."
MaxAngor wrote:
    George... Don't do that.

2011-10-07 00:46:10

Ah, I'm terribly sorry CAE, I got side tracked and totally forgot about this thread.  I haven't coded anything yet.  Oops.

- Aprone
Please try out my games and programs:
Aprone's software

2011-10-07 10:25:48

Heh, no problem. With all the productivity that you churn out, I'm happy enough that you offered. smile

看過來!
"If you want utopia but reality gives you Lovecraft, you don't give up, you carve your utopia out of the corpses of dead gods."
MaxAngor wrote:
    George... Don't do that.

2011-10-07 10:56:01

Do you want it to focus mostly on edge detection or shading in the entire object?  I'm assuming you want edge detection but I want to be sure.

- Aprone
Please try out my games and programs:
Aprone's software

2011-10-07 21:27:02

It'd probably be best to focus on edge detection (so long as it scales down properly, of course).

看過來!
"If you want utopia but reality gives you Lovecraft, you don't give up, you carve your utopia out of the corpses of dead gods."
MaxAngor wrote:
    George... Don't do that.

2011-10-07 22:41:43

What sources will the images be coming from, or will they be specifically created for use in this program of yours?  For example, is this thing attempting to grab images of the screen to hopefully allow someone to feel something that is on it?  Is it grabbing images of a mainstream game?  The problem I'm having is that you really can't fit much visual information into a 40x? sized black or white image.  You would never be able to represent a computer screen or frame from a mainstream game this way.

At the moment I have it load the desired image, marks the edges, and then shrinks it down into a black and white image that's 40 pixels wide and the appropriate height to maintain the original image's width to height ratio.  I'll need to add in some code to clean up the edges a bit, but in almost all cases it loses too much information to be much good.  I think the key might be to especially create graphics for this braile display, rather than attempting to shrink down existing images.  For example, mario from one of the older mainstream mario brothers games can fit into a 40x40 pixel image.  When converted to only black and white it can even still be possible to see make out his identity, but it is considerably more difficult.  If you started out with a picture of the entire mario brother's scene, mario would end up being only the size of a single pixel in that 40x40 image.  Actually he might even be smaller than a pixel meaning he wouldn't even show up.  So you'd be looking at a best case situation of a single raised dot to represent mario in the scene.  With that kind of information loss, you would have no way of distinguishing mario from an enemy, a wall, a pipe, or just about anything else.  They would all just be a single raised dot.

If you especially designed tiny black and white graphics for some braille output game, it would be very easy to represent what was going on.  As an example, I built an exact copy of the mainstream pacman game that fits into a 95x95 pixel window.  It was done as part of a project to see how small the game could be made before it was impossible to know what was going on visually.  To compare, a normal pacman sized pacman game I've made was 570x570 in size.

I don't intend to be discouraging, I just wanted to fill you in on what I'm thinking.

- Aprone
Please try out my games and programs:
Aprone's software

2011-10-08 01:50:12

Actually, My general thought was that distinguishing the objects wasn't as important as being able to locate them. Back when I had access to a Braille Light, I was in the habbit of making "animation sequences", one of which was the first level of Mario, with the limitation that you mentioned (Mario and the enemies were both one dot, and therefore not distinguishable.).
Though, this is why ideally being able to adjust the scale would be an option.
If being able to filter out colors before converting were possible, that would probably help with distinguishing what all the small, same-shaped objects are supposed to be.

看過來!
"If you want utopia but reality gives you Lovecraft, you don't give up, you carve your utopia out of the corpses of dead gods."
MaxAngor wrote:
    George... Don't do that.

2011-10-08 02:05:50

I can have it split it by colors if you want.  Is this going to need to capture images of the screen or an open game window?  If so, I need to code that.

- Aprone
Please try out my games and programs:
Aprone's software

2011-10-08 02:22:23

Images of the screen would probably be most useful (though, probably the most difficult to properly process. yikes ).

I typed this up in the time it took for the last post to come in, and thought it was worth posting:

I think Mario is actually a good example to work with, partly because it's easy to make braille representations of, and partly because at the scale I'm imagining, a line of 20 cells would probably be wider than the screen. The problem being that everything in Mario is so small (And those random scenery objects in the background might get mixed up with the foreground sometimes).

If I remember correctly, the first bunch of bricks in Mario is such that you can't view the first lone brick and the first pipe at the same time (but it's kinda close).
Let's assume that it's scaled so that something the size of a brick takes up one pixel (which is about the size of small Mario's bounding box, and that of most enemies). I actually have no idea how many bricks can theoretically fit across the screen at once (sure, there are plenty of instances (like in the ice castles) where bricks do span the screen, but they don't have visible borders making it easy to count them. Either that, or they were pretty distinguished, but so thinly that my vision was never good enough to notice, which is definitely possible.).

I would think it would come out something like this, maybe spaced out a bit more:

    a
     '       
a  cca     -
-8----0----=
(I assumed that you didn't kill the first goomba and managed to release the mushroom before it went off screen. tongue )
That'd actually be 12x24, but I imagine I didn't space things quite right (and the top and bottom of the screen get other bits of information in Mario). So at least doubling in either direction seems like it should be sufficient. I say seems, since I'm basically just trying to go off of memory and wouldn't know how well that information would be preserved in the scaling process.
(Heh, I'm sure it would be helpful if I could post that bit with a braille font... Is that possible?)

看過來!
"If you want utopia but reality gives you Lovecraft, you don't give up, you carve your utopia out of the corpses of dead gods."
MaxAngor wrote:
    George... Don't do that.

2011-10-08 02:51:28

I don't know of any braille fonts myself, but this latest mario example has some good possibilities.  I think that rather than making a generic all-purpose program to capture the screen, you should design it for a specific game.  If this program is generic and it processes scenes from mario brothers, you will not be able to distinguish between a pipe or the decorate cloud in the background that has no bearing on the gameplay.  If it is designed especially for mario, it can be designed to recognize bricks, pipes, and enemies so that it displays it in the easiest way to understand.

Of course I'm sure the generic all-purpose program is what is desired, so that anyone can use it on any game they wish to play.  And if we were specifically tailoring it to a particular game, we might as well just code our own version of that game which is already equipped to output the display to the braille without any clunky middle-man software.

It's not an easy choice to make.

- Aprone
Please try out my games and programs:
Aprone's software

2011-10-08 03:07:57

Well, I've already made an audioversion of Mario (I just need to get around to adding enemies to 8-3 and putting in the labyrinth that is 8-4...). Though if BGT ever supports output to braille displays, I'd certainly want to add it to that. ^^.
We could theoretically keep the images larger (100x100, maybe?) and let the user scroll and pan and such as needed. A 40-cell display could handle something that's 80px across without needing to scroll horizontally. If the width weren't an exact multiple of the width of the display, though, horizontal scrolling might have less of a risk for confusing people, since it'd be harder to mix up horizontal and vertical scrolling by mistake if the right side of the image covers less of the display. 100 wide would work for both 20 and 40 cell displays in that regard.
It might even be worth focusing on a section of images rather than the entire image (could be confusing for things like Pacman, but would work reasonably for Mario or Sonic, since the player is usually kept near the center of the screen).

看過來!
"If you want utopia but reality gives you Lovecraft, you don't give up, you carve your utopia out of the corpses of dead gods."
MaxAngor wrote:
    George... Don't do that.

2011-10-08 03:46:40

I've played your accessible mario.  smile
I totally agree that there needs to be an option to zoom in to focus on only a section of the image.  Zooming out to get a better feel for the environment around your player is very helpful, but most of the time people will probably only care about what is actually close to you.  How is the braille output handled that prevents it from being done in BGT?  Is it like using the screen readers?  I've never used one so I'm totally clueless.

- Aprone
Please try out my games and programs:
Aprone's software

2011-10-08 04:22:52

I'm not really sure of all the ways that braille output can be used by programs.
I do know that the messages in castaways get sent to my display if I'm in screenreader mode with NVDA, and that messages to the command prompt in java show up only if there's a java application running (there may be a way to read it with the display using one of the various cursor features that I haven't yet figured out, but the basic review commands won't let the display read them if the program has finished. NVDA will read them, but they won't show up on the display).
In theory, BGT might be able to output to NVDA by changing the window title, though I haven't tried this myself. Actually, I should probably do that now (although, BGT isn't equipped to handle looking at images, so short of analyzing a bitmap directly, I'm not sure if BGT would work as the interface for something like this).


... Ok, so, braille display for NVDA via BGT through the title of the window works just fine. I threw together something small enough that I could paste it here.

So... I think I'm going to go add NVDA + braille display support to Mario, now. : oops :

看過來!
"If you want utopia but reality gives you Lovecraft, you don't give up, you carve your utopia out of the corpses of dead gods."
MaxAngor wrote:
    George... Don't do that.