Unless you are really programming an high-demanding resource game, the slight speed difference wont play an huge role. The legend of java being very slow is no longer much true nowadays. Now the ratio is near 1.1 or 1.2 compared to C. The more real truth today is that python is now getting slower than java, and as you know there are a few big games developped in python, SoundRTS for example. So from this point of view, java is perfectly viable to run a game.
The biggest problem is to find libraries for audio, speech and such specialized things.
For speech, my library UniversalSpeech is normally compatible with java.
For audio however, you have the choice between two quite bad options: use a library based on JavaSound and it's quite a pain to have something really good sounding, or use a java port of a well known library e.g. joal, BASS native, FMOD, and there the problem is performances because the JVM make frequent calls to C backends.
If interested, a few years ago, I developped a small audio library based on JavaSound. As already said, it's quite limited because JavaSound is itself quite limited, but if you don't need advanced effects or full 3D, it can suffice. It was sufficient for my game MagicBlocks, and more generally I think that for a simple arcade or side scroller game, it would be perfectly OK.
Another important point against java not to be neglected is its JVM, which has necessarily to be installed separately. Java isn't always installed everywhere and many people have issues installing it, even if they juste have to do I accept next next next finish. So it might, wheither or not you want it, limit a little your audience or you will perhaps have to make more support on how to install the JVM and its dependencies. The size of the JVM make also quite impractical for you to bind it together with your application at the end, opposited to other VM-based languages like python, where it is much easier to do it because the size of the interpreter is more reasonnable.
There are 10 kinds of people : those who know binary, and those who don't.