Here's my second try. No reverb this time:
http://dl.dropbox.com/u/16520690/photophone2.mp3
I think the fact that the different pitched sounds are so far out of sync with one another makes it difficult to link them to a single object.

(spoiler):  There are supposed to be three objects. One on the mid left, a slightly lower one toward the middle, and a tall skinny one on the right.

I tried a couple more (but I only saved one...).

The one I didn't save used continuous sounds like the first one. I actually tried to have more than just those two, with one of them tracing while the others played constantly. What I found was three objects was kind of understandable, but once I added a fourth it got much harder to tell what was going on.

The one I did save, I tried to combine the two approaches. I picked three instruments with fading sound (a bass, a harp and a vibrophone; I picked those three because I tend to associate those sounds with blue / red / green respectively).
Aside from the instrument choices, I decided that each object should be separated from the others by a small amount of time, so it'd be easier to tell them apart. I was pretty satisfied with what I could tell about three (one of each color), so I decided to add a fourth that blended two colors (it's supposed to be yellow, but that isn't as easy to visualize based on the sound as the basic three).
http://dl.dropbox.com/u/16520690/photophone3.mp3
The recording replays three times, each with a different tempo.

I think the continuous sounds of the first example work ok for simpler things (like identifying a tree or a person's clothing). I like this third approach for more complicated images, though.

By separating the sounds with the small gap, even I was able to finally distinguish them rofl!  Are you coding stuff to produce these examples, or are you playing them with a music synth?  I just wondered.  This one is also quite pleasant to listen to, which is a good thing.

Kacijako, you'll have to read the posts to find that out.  Thanks.

I've started taking my code in a slightly different direction.  So far, I've been trying to get the program to just split the screen into the different shapes so that the user can hear the sounds and try to make sense of the view.  More and more that was bothering me while I was at work.  The time I was spending with the seeingwithsound.com program really showed me how tricky it would be to train your brain to sort out all of the noises to understand what was important and what was simply background nonsense.  Even when my program filters out background noise, we're left with a similar problem.  Lets say the scene gets divided into 12 shapes by the computer, currently it would toss the sounds at you for all 12, even though half are probably pretty worthless to you.  A handful of those shapes are probably just the background wall your friend is standing in front of, so by having to hear those it may distract you away from the subject you're trying to focus on.

I'm trying to make the software smart enough to do some of that work for you.  Rather than just dividing up the scene into shapes, it creates a list of all of the different ways it Could! divide up the scene.  Those potential sections are then graded based on how similar they are to neighboring sections.  The more contrast there is with the others, means this object would normally stand out to a sighted person viewing the scene.  A single red ball sitting in the snow will immediately draw the attention of anyone who glances in that direction, so this software should help you do the same with sound.  Once the sections are graded, it will toss out all except the highest ranked ones.  If it works the way I hope it will, this should help eliminate worthless background information and will focus only on the most contrasting aspects of the image.

There could be drawbacks to this approach, I won't know until I get it working, but that's my current idea.

CAE_Jones, actually that's what I'm working on now.  In the process of breaking up the scene, it would have the background as a series of individual shapes, but I'm in the process of making it realize they are all a part of the same larger thing.  Once I get that working, it would then reconnect those background pieces.  Letting the user adjust contrast is a good idea, but I don't think I explained what I'm doing well enough.  What I'm attempting to do isn't the same as the edge enhancement, or even contrast settings, on the seeingwithsound program.  On both of those examples, the seeingwithsound program simply adjusts the contrast of the image or brightens any lines it believes are edge lines.  In the end, however, it is still up to the user to identify the scene.  That extra contrast, or brighter edges, is hopefully a help to the user, but it is still just trying to help them find the important objects themselves.

What I'm trying to do is make the software smart enough to do that whole step for the person.  This example might help.  If I turn myself to look at my sofa, the fabric has its own pattern to it.  There is wood trim around the front arm rests, a white wall behind it, and a piece of paper resting on one of the cushions.  If the program were only to separate shapes, it would have several shapes just on the sofa alone, probably 1 for the paper, a few for the wood trim, and 1 or 2 for the wall.  Lets just assume we have 12 different shapes in the scene, where only 1 is the paper, and most are the sofa itself.  Normal contrast adjustments or edge detection would only help me identify all 12 shapes, that's all it can do, so I'm still left wondering which are important to me.  If I can get it to work as I plan, my program would first connect the few wall pieces since it understands they are actually the same thing.  Out of the several sofa pieces, some could connect, but most won't because the pattern appears different depending on where you're looking and there's no easy way for a computer to realize they are all the same object.  Instead, my program realizes that the patterned sofa pieces contrast with each other, but not as much as they contrast with the wall, wooden trim, or the piece of white paper on the cushion.  Because of that, those pieces are given a lower score.  The highest score goes to the piece of paper, since white paper on a generally dark blue sofa really contrasts.  The wall gets a good score as well, since it clearly contrasts well with the sofa, but it also loses some points because the wall was composed of 2 pieces, and the more similar pieces exist, the lower the score.  This is why the sofa gets an even lower score.  If the wall differs greatly from the sofa, you'd expect the sofa to differ just as greatly from the wall, but the sofa is composed of more "like" pieces and gets a lower score.  The wood trim stands out a little, but brown on dark blue isn't super different.  In the end, the program throws out many of those original 12 pieces.  You'd be left with 5 pieces ranked in order of score.  As the sounds are quickly cycled through, first you'd be presented with the paper, second you'd hear the wall, then the wood trim, and finally 1 or 2 pieces that were both in fact the sofa.  Odds are, those 2 sofa pieces would represent each half, or perhaps they'd represent the top of the sofa verses the bottom.  Either way, you'd be left with very few things to hear, and they would already be ordered from most important, to least important.  There would be some problems, like your intention could have been to focus on the wood trim which seemed less important to the computer.  Even when that happens, the trim is still there, it just isn't what should jump out at you when you first turn to look at the sofa.  Just like with a sighted person, the very first thing that should grab your attention is the white piece of paper on the dark blue sofa.  Most things in this world are designed so that sighted people will notice the important things fast and easily.  By attempting to recreate that order of importance, I'm hoping it will make this program perform better in real life.

I assumed you were using a synth, but I wasn't sure.  I started to wonder if you were coding up your own seeing with sound type program, and I was going to say I was Very! impressed!  You seemed to be making huge amounts of progress in almost no time, haha!  The synth is a great way to test sound combination, and its good knowing I'm not the only person trying to figure out which sounds will work best.

Hi,
This all seems rather incredible. This would actually be really interesting, but the one that really has got me excited is the thing you mentioned before mentioning the seeing with sound prospect: the webcam being used for audiogaming.
Anyone who has been around here for a while may remember a rather old game called demor.
I never got to play it properly, due to you requiring gps and all that.
But they made a version that let you use a joystick, and it was ok.
But this seems even cooler, as you can put yourself in the game right in your chair, by looking around you, tracking an enemy, and then firing!
I'd love to do this, so much so, I just hope this gets done sometime.
I'm quite surprised you decided to work on the seeing with sound prospect first, unless you're working on them both at the same time? You could start off with a space invaders style of game with head tracking, I know, I know guys, we have way too many space invader games, but then what if it could track your head going up and down as well? or even your hand?
If it could track your hand then you could have a sword or something.

Aaron, I suppose I started on the seeing with sound idea first, for a few reasons.  For starters, it is a much more complicated project, and I don't feel like there are as many other developers around here who could do it.  I don't mean to make that sound bad, but only a sighted programmer is going to be able to work this program out with the web cam.  Adding head turning support to a first person shooter seems super complex as well, but I can honestly imagine quite a few developers here giving that a try.  Tracking a moving face is a hundred times easier than tracking everything and interpreting the data.  In a way, I suppose it is like when you see someone post a question on the forum.  If the question is rather odd, and you happen to know the answer, you're almost compelled to take a few minutes to answer it.  If you ignore the post, thinking someone else will take the time to help, there might not be anyone else around who knows the answer.  I'm kind of taking that approach to this seeing with sound program.  I may fail at it, but at least attempting it seems like the right thing to do.

My other reason for doing this one first, is that it could become an even cooler gaming tool than the head tracker!  Lets just assume I pull this off, and we end up with an easy to use tool for seeing with sound.  If you can walk around seeing the world through your web cam, then it stands to reason you should be able to walk around within a game world using the same technique.  I would just design a first person game that outputted all of the data in the same format as the seeing with sound program.  You'd be "seeing" the game world the same way you saw your own house as you walked around with the camera.  Hopefully that makes sense.

It is funny that you asked if I'm working on both at the same time, because really early this morning I started planning out a game that will use the head tracking idea.  Keep in mind, I don't know yet if I can get the head tracker to work properly, but I'm getting a little farther in developing it.  As Dark said earlier, the easiest solution would be for people to wear colored indicators on a head band, but I'm trying desperately to avoid having people do that.  It seems to be too inconvenient for me to demand people locate, and wear, a colored target on their head as they play.  I have stuff here at my house that would work wonderfully, but I need to find a solution that doesn't involve sending things to everyone in the mail Haha!

lol, that's true about the construction paper, although that doesn't always work as well as you'd expect.  Due to differences in lighting, most colored objects will give the camera some trouble.  In my own projects, I use glare resistant foam sheets which are usually intended for children's craft projects.  They are super cheap, where I get a good sized pack for only $1, but I can't exactly start mailing sheets of the stuff out around the world.  On normal paper, from certain angles it will appear pure white or black, even if it is actually a vivid color.  This confuses the camera for a moment, and that would result in some annoying glitches during the game.  Years back I built a helmet for my cat to play a multiplayer action video game online against some people I knew.  In principle, it used the same kind of head tracking technique I wish to achieve here, so I already know what kind of obstacles we'll be facing.  Her first helmet used colored paper, and it would cause the game to suddenly jump around because it could lose track of one of the colored indicators and contort the viewing angle.  A second later it would correct its self, but those glitches happened randomly and quite often.  It wasn't enough to affect the quality of my cat's online gaming performance, you know, since it's a cat! ROFL!, but it would affect a human and probably annoy someone to the point they'd stop playing.  In case you're wondering, my cat did best with a flame thrower, she managed to chase down and wound several people, but in the end she had a terrible ratio of kills to deaths.  What else can you expect from a cat?!

I'm still having a nightmare of a time with the head tracking, but I suppose each day I'm getting closer to a reliable solution.  Since I can't count on tracking any type of head band or even eyes, my latest attempts have been to track the nose.  If that doesn't work, and it might not, is it safe for me to assume that everyone here has some form of sun glasses?  Hopefully I'm not making an ignorant assumption, so forgive me if I am.  If I can find just 1 face-worn item, that pretty much everyone would have, I can incorporate that into my design and probably have an easier time solving this.

Hi Aprone.

As I've said before, it's those sort of land mark style navigation methods I use, so I'll appreciate that in the game deffinately.

unfortunately I don't have a web cam, so I'll likely just have to use basic navigation in the game, though fortunately it sounds like that will be pretty clear.

I've been making tremendous progress, with the head tracking, over the past few days.  I'm getting quite excited about it!

devin, I'm pretty sure that wouldn't work, because the computer wouldn't identify the ipod as a standard usb web cam.  It's worth a try, but I would be surprised if it worked.

An android phone would basically be the same situation.  I don't believe it will work unless you are using a traditional web cam, specifically meant for the computer.