Yes, I agree with poster 2.
But text editors aren't such a problem, you can install an FTP server on your remote machine or more secure SCP, and then edit files from your windows PC using Total commander for FTP and WINSCP for SCP

Hmm, in most cases, you cannot install WIndows nativelly. You have to use wine or similar software

Windows VPS's will be considerably more expensive than a linux VPS. Not because you need higher system requirements, but because of the ridiculous licensing requirements imposed by Microsoft, the cost of which are transferred to customers by the hosting companies. If you insist on having Windows access, then the best way would probably be installing NVDA with NVDA remote on it though that will not work for iOS.

Accessibility of Linux servers is perfect straight out of the box as you'll always be logging in over text-only SSH. Most things you'd ever want to do will have been explained already on-line with a lot of guides. Things like web servers or TeamTalk all have linux binaries and generally work better than they do on Windows. It would be helpful if you could explain what exactly you want to do with it and what makes Windows a requirement, as you should be able to run some basic Windows applications, IE audio game servers, using Wine (This is how the primary RTR server is running as PureBasic's network support is very unstable under Linux)

The Mac Mini would be better if it were the 2012 model, quad-core. But the current Minis are still pretty good if your server load will be predictable. Virtualisation is one thing that probably throws this calculation off though.

Yours is not a bad idea, SLJ, but I have two Mac Minis on my desk and only one of them runs macOS (the other runs Linux), and the Mac can't be remotely accessed unless I'm next to it and can hear the audio. If you ever figure out how to stream audio, great, but I haven't yet. So you end up with a keyboard and pair of headphones in hand, even though the Mac can indeed do many things as a server; mine is used just to serve iOS clients with the caching server and also handles network backups. Linux, of course, is God's gift to geeks with skill in the command line. Windows is fully accessible with Remote Desktop.

Running a VPS is nice, but it still has two fundamental problems: it's somebody else's computer, and latency. The first is a matter of philosophy: why pay over the odds for someone else to do your work for you? But it has advantages of scalability at the lower end and makes it much easier to experiment or add and remove VMs, even though it introduces possible privacy concerns as do all clouds. The other is much more practical: you can't run a NAS from the cloud, because a network storage box has to be on your own LAN, by necessity.

As somebody here has said, it would be nice to know what your exact needs are, so we can advise you on the simplest route. I'd suggest a Linux box on your LAN as well as a Mac Mini, but since you'll want Windows as well I'm thinking the Mac is still the better option as long as you're prepared to lose a little flexibility and remote manageability. Linux in theory provides everything you need, but you would have to do everything without a head, which is certainly possible but challenging for a novice, and realistically it's impossible to do certain things like run proprietary software logged in (such as iTunes or Arq, in my case).

HTH.

If you just want to get started, you might want to go for debian. It's a system where everything is basically setup for you. for example, getting a website online is as simple as typing apt-get install apache2 in the terminal, and it will install apache and set it up with defaults. Regarding wine, just type in apt-get install wine, go to your executable with cd and type wine filename.exe, and you're done.
Sound rts is opensource, so just download the mac/linux version, and type server.py to launch it. Linux can be overwhelming at first, but if you spend a little time tinkering with it, you'll ask your self why you didn't do it sooner.

I would personally recommend Ubuntu server over debian, mostly because Debian has really old and out of date packages. Using either won't be very different at all as they both use apt for packages and when it comes down to it is Linux, so what roel said can apply just as much to Ubuntu as it does to Debian.

To answer your question though, yes absolutely. 98% of linux distributions are completely free so you can just go to their homepage, get an ISO image of an installation disk and either virtualise it or run it as a live CD on a physical computer.
Ubuntu server download page
Debian downloads

In addition, a few VPS providers offer trials of their machines. For example Lynode, which is many blind people's favourites, has one though I think they are overpriced for what they offer. Setting up Linux on a VPS isn't as complicated as setting it up on a VM.Because the hosting companies have all of the various Linux distributions on their server, all you have to do is select which one you want to use and it will be installed for you automatically and you will just be able to log in and start working right away.

I have just looked at Lynode, and wow it seems great. Really useful guides to get started. Thanks a lot for mentioning that. I think this is what I'll end up choosing.
Currently I only have two questions left:
1: If I wanna start out by setting up a Teamtalk server, how do I then get the required files on the server? How do I put my already configured server on the system? Is there an upload feature on the website where I can upload files to the server?
2: If I choose Ubunto and control the server via SSH, will I then automatically avoid the graphical interface? Will the graphical interface even be activated and use up the resources on the server?

Ubuntu Server does not even include the files required for a GUI and I suspect the debian images offered by the VPS providers don't, either. So no, you won't have to worry about GUI processes using up any resources and you'll be exclusively dealing with a CLI.
Installing things will go in either of 2 ways. The first, which you'll be using most often, is a package manager (apt for debian and Ubuntu). If you have ever used Homebrew on your Mac, then this is the exact same concept. If not, think of it like an app store and software update service in one. The most important packages, like Apache (web server), git, the bash shell, SSH server, text editors like vin or nano, various utilities like wget or curl for downloading files, programming languages like Python or perl or even the linux kernel, will be installed and updated with it. So usually it will come down to typing in sudo apt-get install apache or whatever else you'll need, then saying yes to confirm the installation; You will also be able to update every package on your system at once with sudo apt-get upgrade and you'll want to do this once every few weeks to make sure you're running the newest version of everything.

If something isn't already in a repository, you can install it manually. Most Linux applications are compiled manually by downloading their source code from a Git repository and running a make command. This usually is explained in the readme for whatever you'll be installing and these commands are all done on the server itself.
In the case of Teamtalk you don't have to compile anything. Just get the latest linux debian archive from the bearware site (which also works on Ubuntu), extract it and start the server. You can just download files onto the server directly using wget, then extracting them. If you ever need to upload or download a file onto your computer, you can log in over SFTP by entering the exact same address, username and password as for SSH. Every FTP client I'm aware of now supports SFTP connections, IE transmit, Forklift or commander one on Mac, or filezilla or winSCP on Windows.

Yes. That would download just the file. But (at the risk of putting a damper on things) it's a teensy bit more complicated than that ... the shell characters that might be in URLs need to be escaped. The best way is to use single quotes, or apostrophes, to surround your URL; then any character, except the apostrophe itself of course, will pass by OK.

And wget, itself, is quite a beast. Learn it by reading its documentation. There is a man page (type "man wget" to view it, press space to scroll, press q to quit the pager). Type "wget --help" for a quick summary. "info wget" for the full documentation, or better still, just Google it and read the documentation from GNU.

In fact take this as a general recommendation to go forth and experiment and learn and read. You can't go wrong. "man" followed by a command, "info", and online help, as well as directories under /usr/share/doc, contain a whole load of useful information ...

Oh, and feel free to ask your questions.

You don't really need to do that, both Linode and Vultr have trials of some sort. In fact, why not give Vultr a whirl and receive some free credit while you're at it!
The good thing about this is that your trial is not limited by days, you just get some credit that you can use as you will to try it out. Whether you want the hourly rate or the monthly payment, you can use this and any credit you add to your account in the future for things like that.

Alright, cool. It's a referral link, so if you use that you'll get even more free credit. I believe the lowest priced vps on an hourly rate, a fairly decent starter, is $0.007 an hour.