hi,
I have a webcam

lol, so far all I've been able to do is give myself a headache!  I have a few project ideas involving the webcam, some more realistic than others.  I'll keep the crazier ideas to myself for now, just so people don't laugh at me and think I'm crazy, but one of the more realistic ones is a method for better immersing you into a first person shooter game.  Imagine this.  While traveling down the hallway you are passing doorways on either side of you.  As you'd expect, you hear sounds to your right and left as you pass the rooms, but since all of the sound is simply to one side, it is hard to imagine where those sounds are positioned within the room unless you turn yourself to actually enter the room.  What if you could simply turn your head to the left and right to "look" into each room as you passed it?  I'm trying to get the webcam to reliably! understand that you're turning your head so the left/right audio output of the game changes to make you feel like you're really turning your head within the game.  Lets say you notice something in the room worth investigating, or perhaps an enemy.  You're already "looking" in that direction, so rather than fumbling around with rotating your guy using the arrow keys, some single key would quickly rotate your person to match the direction his head is facing.  If I can get it to work properly, you could continue traveling in 1 direction, while turning your head to check out other directions.  Then, if you wanted, you could quickly change your direction to the new one you were checking out.  I hope it would add a new level of realism to the game.

Yup me too. They're pretty common nowadays.

This is good.  The more people that have them, the more people I can test with as I develop stuff which uses the web cam.  Yay! 

Do you ever find yourself lost in your own little world, with your mind wrapped around some puzzle for which you know there Must! be a solution?  I've found myself doing that a lot today, and it honestly bothers me that I haven't figured this thing out that I'm working on.  I wish things were like when your a young kid in school.  You could be stuck, trying to understand something, but there was always the teacher nearby who could give you hints if you asked.  It was always nice knowing someone right there already knew the answer, it reassured you that this problem could be solved, and no matter what you would understand it yourself soon.  Either you would figure it out for yourself, or the person with the answer would lead you to the answer.

Complicated problems in real life aren't so nice.  All my poor wife hears about, day in, and day out, are my ramblings about engineering or programming problems I'm trying to sort out.  I don't have someone standing next to me with the answers, so I'm left talking it over with myself.  I'm always convinced the answer is just out of reach, and I only have to look at the problem in a different way.  Trying to convince yourself there is a solution, when  you don't know if that's true, isn't as reassuring as the good old days of school.  I guess I'm just rambling because I just threw out another plan, and I'm starting up a new one.

Arg!  I feel like I'm going crazy haha!  This reminds me so much of when I was trying to build the object identifier program.  I'm writing up pages upon pages of code, realizing it doesn't work and isn't going anywhere, selecting it all, pressing delete, and starting over.  I've tried to get myself to stop working on this crazy idea so I can concentrate on the easier ones, but I keep coming back to this with new ideas I want to try.

Sheesh, I might as well explain what I've been working on, on the off chance someone here will have an idea of how to help.  I'm sure you'll all think I'm crazy, but oh well, I suppose people like me for the games I make and not because I'm sane, ROFL!

I'm trying to build software that would allow you to see through your web cam, using sound.  I know, sounds crazy, but let me explain first.  Short of having some kind of major birth defect, all of our brains are hard wired to handle certain types of data.  Sound, sight, smell, touch, taste, and things that are not solely tied to one of the senses, like the sense of motion, spacial awareness, balance (probably should be lumped in with motion).  There's other examples, but hopefully you get the idea.  For each of these concepts our brains are hard wired to understand, the data comes in its own type of format, or organization.  It's a crude example, but your computer can read music files.  If those 1s and 0s are arranged in the proper type of pattern, it is a music file, and as long as the computer knows how to use the different hardware, it doesn't care if those 1s and 0s were read from a magnetic drive, a laser on a cd surface, wirelessly beamed through the air, or sent through a fiber optic cable using light.  In the end, all that really matters is what kind of pattern those 1s and 0s end up in.  That is how your computer sees it as a music file.  I know, the example has some technical holes, but again, hopefully you see the point I'm trying to make.

Now with the computer example, obviously some hardware forms are better at sending the data than others.  For example, reading light signals through an optical cable will always be faster than reading from an old magnetic 3.5 diskette.  In the same way, ears can bring in more data, and faster, than your nose.  That seems like a strange comparison, because ears and a nose are meant for completely different things, but it's actually fair to compare them.  When it all boils down, your ears and nose are just reading data from the world around us, and turning it into electrical signals for our brain.  This is no different than a CD drive reading 1s and 0s with a laser, or an old floppy drive reading a disk with a magnet.  We are talking about hardware equipment, that is all.

Our senses are not completely exclusive.  Sometimes 2 things will produce the same type of data, such as how your nose and tongue can both help you taste food.  If you plug your nose while eating, some of the flavor will go away.  Now I'm not talking about the smell of the food, I'm literally talking about taste!  This is also why you can smell something and usually have some idea of what it would taste like.  Your nose is less efficient at producing taste data, but still, it is capable of doing it to some degree.  Spacial awareness is the same way between the eyes and ears.

Now that all of that background info has been said, what I'm attempting to do is use your ears to relay sight information to your brain.  As long as the formatting of the information is a close enough match, your brain should be able to understand it as visual information.  That is my theory at least.  This process would require a bit of training, but I'm also trying to make that process both automatic and quick.  Let me step to the side for a moment and give an example of what I mean.  When you were very young, you learned to understand language through sound.  You'd hear people speak, and your brain associated those sound patterns to mean different things.  The only thing separating the word "Hello" from any other random sound, is the fact that your brain associated that pattern with something.  So at that time, it seemed that the only way to communicate that message was to hear it.  Of course, if you were sighted at that time, you learned to read with your eyes.  Your brain learned that particular patterns of visual data represented letters, then, you were taught how to "sound out" each letter or each letter combination.  You probably didn't think of it this way at the time, but you were learning how to convert between visual data to letters, and then letters to sounds so that you could then link it in to the language patters you already knew!  From that point on, you could get sound data or visual data, in the correct formats, and it would follow the chain in your brain that would lead you to language information.  Of course, for all of you, you then took it even a step further by linking those same connections to touch data.  As you learned to read braille, the same process linked back to the audio patterns of language that you already knew.  In theory, you should be able to learn to communicate words using smell or taste!  Now I've never received any words of wisdom while eating alphabits cereal, but I have had my brain shout "Run away" when people sent smells in my direction!  ROFL!  Sorry, I got side tracked there.

With those examples, it sounds like even if I was successful, learning to use the equipment would take a long time.  After all, learning to read took a while!  To speed up the process, you need to give you brain near instant feedback on your progress, and I plan to do that using the web cam itself.  Let me give another example.  If you were deaf, imagine how long it would take for you to learn how to speak my name.  I write to you "Say Jeremy" and I wait for you to try, only then to shake my head no or yes depending on if you were correct.  Such a process would take forever!  Now imagine you, being able to hear, trying to say my name for the first time.  If I spoke out loud to you, "Say Jeremy", it is unlikely it would take you more than 1 or 2 tries because you could instantly hear your own speech and your brain would be making thousands of comparisons to the audio data example I had provided you with.  In the same way, I can use the web cam to instantly give your brain feedback on what is going on.  I think, if I can get this working, it would speed up the learning process.  Who knows, it could be a few hours of listening to gibberish sounds and then suddenly your brain would figure it out.  There's really no way of knowing yet, and this really is all assuming I can get anything to come of this.

Now we've already said that different hardware is better, or worse, at producing types of data.  Unless your taste buds are damaged somehow, they will always run circles around your nose when it comes to producing taste information.  No matter what I do, I don't believe it will ever be possible to use your ears to match what the eyes can do.  After all, the eyes were built for the job they do!  My goal is to give you enough visual resolution to get a general idea of what's going on around you, but anything more is probably unrealistic.  I would be thrilled if you could turn your camera out your window and be able to "see" that a blue car and a white van just drove by, and that a person is sitting on the porch next door.  Even if you can't tell who the person is, or what body type the vehicles were, this would have to be a useful tool nonetheless!

The first thing I did was try to figure out the smallest resolution I should be trying to recreate.  I decided to go with 64 by 48.  For those who have never seen, the most basic comparison I can give is to imagine a grid of 64 by 48 where each cell contains a single point of color.  I've given up on any idea that would relay all 3072 cells to you as sound, that's just insane, especially when you understand that each cell would change several times each second!  My next idea was to focus only on changing values, since technically that's how the eyeball physically works, although as a sighted person I'd never be aware of that under normal circumstances.  This was tough too, because it almost forced me to dismiss any color data from the scene.  I hate doing that because having color information is especially helpful when you're working with such low resolution.  If we have to go gray scale, I'd be very tempted to increase the resolution, and thereby cause a whole new problem.  Even when focusing only on the changing cells, there was far too much going on to handle it with sound.

The idea I finally threw out yesterday was to have the computer break the image down to basic shapes, like circles, triangles, and rectangles, to the best of its ability.  Each shape, and its location, would be portrayed using sound so that we only had maybe a dozen or so things to track at a time.  This seemed to be the most promising, except it seemed to have strayed too much from being a visual format.  as things move around and you see them from different angles, completely different shapes are formed.  That's all fine and good, except they don't suddenly change from 1 shape to another as this would be forced to do.  Changes are gradual, and even if they weren't, a sighted person can process that change in a way the blind person can't.  Visual data of a triangle suddenly becoming a square breaks down into smaller elements that let the brain follow the change.  Going from 1 sound representing a triangle, to another sound representing a square, does not work the same way.  I liked the general idea, but it has serious flaws.

The latest idea is a variation on the shapes.  What I want to do, is get away from specific shapes, and assign sounds that tell you relative measurements of each shape instead.  For example, the old way would have been confusing as you switched between a flag oval, to a circle, to a tall oval.  If, instead, the sound of the shape gradually changed as a function of the shape's width and height, you'd hear a smooth changing sound as the shape squished into a circle, and then stretched its height into another oval.  This is the kind of changing we need, since it allows the brain to break down what is actually happening instead of just a sudden sound change which is supposed to represent some new shape.  There are just so many things to consider, it is driving me crazy.  For the moment, I think the most important attributes each tracked shape needs are, shape as a function of width and height, overall size on the scene, color as a function of red, green, and blue content, and position on the field of view.  If I can represent all of those things using sound, and manage to track a dozen or so such objects without it overwhelming your ears, this just might work!  Any comments, feedback, or ideas are more than welcome.  As I've said many times already, this is weighing heavily on me and it is quite overwhelming.

Do you mean something like this?

[edit] Ah, now that I paid more attention to your post, I think you meant something more object-oriented.

Well, the fact that there's facial recognition software (weak though it may be) is reason enough to expect _something_ to be possible.

My thoughts would be to look for regions of continuous color, but that'd only work on a rather flat image without shading, which wouldn't be practical at all...

CAE_Jones, wow I didn't know that existed.  Your link is pretty much what I started out by making, except I threw it away thinking I was going in the wrong direction!  lol silly me, apparently this proves my initial idea could have worked in some way.  Since it already does exist, I guess it doesn't hurt for me to go in the direction I'm headed in now.  Actually I've done a pretty good job of getting color groups to work even when there is shading.  I think I can pull that off from the software side of things.

Dark, your method for head tracking is probably the way I'll have to go.  I've used that technique in the past, I just wondered how practical it would be to ask people to find colored objects to stick to their head lol!

The problem with limiting the camera to too small of a field of view is that you can't understand the whole scene.  As a sighted individual, I can demonstrate this easily by trying to look through binoculars.  Many people put only 1 eye in so the other can continue to see the whole picture, since through the smaller view it is so easy to get lost.  Of course, once the person has lined themselves up with what they wish to see, they can completely rely on the binoculars.  I just don't think it is realistic for someone to point a camera at a tree unless the view is large enough.  If your view was small, and the tree was 50 feet away, you only have a few degrees of leeway before you're looking past it.  That's far too small to require of someone who doesn't even know if the tree is there.  I'm still quite convinced the only way to make this truly useful is to find out how to make it work on a standard web cam sized view.  A web cam already captures a very small view of the world compared to actual eyes, so it is possible we're already on the same page but just describing it differently.

CAE_Jones, the shape stuff is pretty much what I'm coding now.  I've done a fair amount of work with complex shape detection so I'm pretty sure I can get that working on the computer's side of things.  Of course, relaying that information through sound is another story.  I'm thinking that exact shapes aren't necessary though, or at least that's my current attempt at making this work.  I'm really hoping something as generic as an object's general width verse height will be enough to help you understand what you're looking at.  I don't think a single object would help you, but a tall red object on top of a tall blue object could be how you identify your brother who is wearing a red shirt and blue jeans today.  Each day you see him he would change in appearance, but I don't think that's too much of a problem.  The point would be that if you already identified him by voice, or whatever, and then you walked around a store, you should be able to spot him again from a distance based on the colors and general shapes.  If there is another guy wearing the same colors you might mix the two up, but that happens to sighted people as well, just not as often.

The object identifier program I tried to make, back when I made the color identifier, had a lot of this shape stuff already coded.  I'll probably go grab some of that code to save me some time.  I'm still upset that I never got that thing to work!

Oh, I wanted to ask, have any of you guys tried that seeingwithsound.com product?  I'd be curious to know how well it actually works.

I've spent the last little while playing around with the seeingwithsound.com program, and I was pretty much lost.  Clearly this would take a lot of getting used to, as you've said.  I'll admit, the moment I saw that seeingwithsound.com offered their program up for free, I was disappointed.  The more free stuff out there, the better, but it made me think why even bother making my own if there is already something like this for free?  I guess this just means my goal isn't to just to provide sight with sound, it isn't only to provide something like this for free, it'll have to be to provide users with a completely different approach to seeing with sound that will hopefully prove to be more useful.  There was so much background noise with the seeingwithsound.com program, it made it nearly impossible for me to know what I was looking at.  Even though my planned program would hide a lot of detail, by hiding detail it might actually make the whole thing less confusing.  I'd much rather know for certain I was seeing a large green object, than to listen carefully to static trying to decide if this is even an object I'm looking at or just background noise.

Hmm, I hadn't considered reverb, but I do agree it could get tricky to use.  I'll write it down though, so I don't forget.

(Hm, my last post makes so many points that I felt like this warranted a new one...)

Maybe, instead of finding properties of the sound to apply to width, height, vertical position and so on, you could give a couple reference points for the bounds of the shape? Like the top left and bottom right corners? Then pitch could suggest vertical position, stereo panning for horizontal, and something with the timbre for color.
That way, instead of a lot of noise with a lot of aspects, you have a small number of points that fit together.
The difficulty would be objects colliding, or giving a good sense that these corners are on an object and not just tiny objects of their own floating in space.
I imagine reverb might be useful for that, if the colors of the corners could be projected between them. If so, that'd be the sort of thing each user would probably want to have control over (too much could be annoying, too little could mess up the effect; could vary with how many objects are around, etc).

Just an idea. I might try to mix some sounds together to create an example of what I'm thinking of.

[edit] Hmm, you could even set the reference points to trace the outline of the shape, adjust the number of them and where they're located, Etc. Hmm. [/edit]

Hmm, well, I tried to make something real quickly. I don't know that what I came up with is particularly interpretable...
http://dl.dropbox.com/u/16520690/broad_ … square.mp3
I used six tracks total--two corners for each object, and one centered echo that combines both pitches.
I'm not sure that it's a good example. I can tell that the strings are broader horizontally than the square, but they're too musical-sounding to easily distinguish further. It'd probably be easier to tell things apart with repeating sounds rather than sustained tones. (One of the reasons I changed the sounds with my Towering Tones game from sine waves to a variety of distinct sounds).