magurp244, I downloaded your python audiocraft to take a deep look. It wont fire up on my macbook but perhaps on a pc at work.  Who is this  rastislav kiss?   and where is his minecraft program?

@52
Binary is windows only, 32 bit. You could probably get the source running on a mac if you setup the environment right, but it would likely take a bit of fiddling and adjustment. I scraped it from an open source bare bones minecraft engine example [here] and modified it with some custom sonifier code derived from Peter Meijer's sonifier source over on [seeingwithsound]. It was originally meant more as a concept demonstration of depthmap sonifier navigation rather than a full game, but the foundations there. I have more refined python source around for the sonifier though, if your interested in something like that, though Audiocraft v1.1 isn't bad with the filtering implementation.

The idea, in principle at least, was to use something like 3D positional audio with reverb effects for the ambient environment for a general sense of the world, but to have the sonifier have various filter modes so people could scan for things like specific types of blocks, items, enemies, or interactive objects like switches, containers, doors, etc. not to mention be able to navigate around various obsticals within a 3D space.

I'm not sure what Ratislav Kiss is up to, though as with many a dev motivation and time can be precious resources. If you have any questions on the implementation, or any other source code like my custom python OpenAL wrapper, just ask.

I used to play Minecraft quite a bit back in like 2010-2014. I had better vision back then and a bigger monitor. Some things were hard for me, like building line after line of cobblestone for a structure, basically, the line would blur for me between two blocks and I'd need to rely on the hitboxes. Block types with more distinct patterns like Lapis Lazuli were easier, but I never really built out of that unless I was on creative. I used to play with a lot of mods, and one that did help out when I got hold of it was WAILA (what am I looking at). A buddy of mine who still dabbles in the game let me know the name changed but I forget it now. I would always get the FTB launcher and use the Direwolf20 pack. I used to watch all his videos and I liked the mods he put in his pack. I usually disabled a few like Thaumatergy and so forth, but I kept most of them. I loved IC2, Red Power, Thermal Expansion, Applied Energistics, Buildcraft, etc. The list goes on and on. I used to play on peaceful because I couldn't see the creepers and they'd blow up my stuff and I thought that was an unfair aspect since my vision impairment prevented me from spotting them soon enough to stop them. When peaceful tables came into the game, i used those to get the mob drops which let me play survival mode without using TMI or NEI to cheat items in.

I'd like something like this to become a reality, since I can't play anymore due to my vision getting worse. I just see a long road ahead. I'm not much of a programmer, but I know how in depth the game is and what all you can do in it. As a sighted person, you will know far better about these things than anyone who has been totally blind from birth. If you have hope that something like this - even if it isn't the full game - can become a reality, then I do as well.

I've looked into modding the java based minecraft before, but i'm not that familiar with the code base or use java and/or javascript that much. I did find a few existing minecraft mods for advanced audio like reverb and occlusion [here] and [here] that should work, and given some thought on how to port the depth sonifier over to java, but even so it'll likely take some modification of minecrafts rendering and audio pipelines for it to work properly. Generally speaking you'll need to change the renderer to output solid textures with no fog or gradients to act as a stencil for the depthmap for object isolation, coupled with access to the depth buffer for spacial positioning and range of objects. A compute module like Numjs or Nd4j for handling the heavy matrix calculations when converting the images into a waveform, assuming your using Java or Javascript, and lower level access to the OpenAL stack for creating a custom audio buffer for playback. Generally you'll want to scale down the render area before you do any of this since it can be very computationally expensive, something like 1/10th the current resolution. The general loop goes something like this:

-Draw solid textures to frame buffer
-Grab frame and depth buffer images
-Pass both images to the sonifier class
-Break images down into uint8 RGB 2D arrays
-Multiply the image array with a prepared sine wave array.
-Sum the array along the Y axis into a 1D array
-Load the array into a ctypes array
-Load data into a custom OpenAL buffer
-Load buffer into player for playback
-When finished playing, eject buffer

If you have it on a loop, when loading the data into the OpenAL buffer you grab another set of images and pass them to the sonifier for processing while the sound is playing, then load the new data in when the buffer is ejected from the player, rinse and repeat.

I was out of the scene before it hit Xbox, win 10 store, all that jazz. I basically always used the java version.

Wait, how? You just hold space to keep swimming. If you let go, you'll go to the bottom. You want to look up and hold space.

Just wait until you start building and yer shit gets blown up lol. Creepers are made to blend in somewhat, and people don't notice them all the time. The only thing you can do when you hear that sizzle is do a 180 and run like hell. Don't use weapons or anything like that, because they may blow up before you would have killed them, just run fast and away until the sizzling stops, and hope you escape the blast radius. Once the sizzling stops, you can either take them out at range, or lure them away from your structures and run right at them with a sword. Also, you can get skeletons to shoot at them if you're good at it.

On PC only, I believe. I don't know about java / javascript, yes , they can give you powers like that.

You have a whole slew of mods that will do things like give you tools that not only let you mine quicker, but let you take out swathes of stone at a time. I've seen tools that take a 3 to 5 block horizontal where the block you dig is the center of that span, and vertically, and one that has a radius of like 3x3, 5x5 around it, etc.

Then you have what was buildcraft which had the quarry system, where you built the framework for this thing, then let it run and it would mine to bedrock inside the skeletal structure of the framework. You just had to pipe its output to a chest or something.

If you have every block within a certain distance from the player each making their own sound, that's going to become overwhelming rather quickly. I would suggest something like a detect nearby objects menu that would give you the closest of a type of block, yet be smart enough to not show each and every wood block and leaf block of a tree.

Yeah, the only way to know is to try it out and see.

been slow work. I don't have much time in the day when I can wear headphones so its hard to do the stereo sound parts.. I am working on a replica of the Minecraft terrain for it though. Ironcross what are the top 5 blocks that would be on the surface of the minecraft earth? rock, water, grass, wood, and leaves?

@ goingSound
keep it up man!
I am soooo excited!