2014-08-13 23:16:08

Hello,
I have been trying to wrap my head around all the different parts of web development for a while now and still am not sure on everything.
I know that when you run a server, (weather it is the little one provided with django, pyWeb or Apache), it creates an IP address or url like http://0.0.0-8000 Then the computer that is hosting the server has a framework that deals with the input from webpages and decides what to do with them. The frameworks are like django or pyweb and the lamp distribution.
Servers create virtual environments for each host.
After that I'm a little lost.


So a host like In Motion Hosting will have their master computer running a ton of virtual machines, each one is given a limited amount of disk space and often several people will be given one machine as their "shared host".
The people may not know each other, but because the machine is virtual, you can log into it through your web browser.
This is a shared hosting plan.
I am not clear what the difference between a Virtual private network and a dedicated server is, but I think the dedicated server gives you a whole computer to yourself where as a VPN just gives you one of the virtual machines.


In a shared network you are able to use something called the SSH script shell to run commands. How extensive is this? Is it like a command line on windows or unix? I was reading on how to install django on a shared server and it sounds like I just have the ability to install packages on the already installed programs. For example, I can't install python because I just have access to the virtual box and not the system. But if that is the case, how do VMs work?
I guess if the server's computer has python installed, you will be able to use python commands on the SSH line. You can install vertualenv and run a completely new environment, essentially creating a VM inside a VM on a master computer.
Inside this vertualenv instance, you pip install django, pyweb or something like that. You then are able to restart your framework and access files that you have uploaded to the master computer through a File transfer protocol (FTP).
The url that django or pyweb create (http://0.0.0-8000) is then tied to the domain that your website is given (In your panel you are shown when you log into your server account you can have many websites and each are assigned an arbitrary domain (like http://inmotionhosting.com/sharedhostin … ldername/) unless you have payed for an unique one).
What I am mainly confused about is:
1. Why can't I install something like twisted and have many people on a tel-net client like they have for Muds sending and receiving data to the server (server being the vertualenv running pyweb or django that is on the vm that is my shared hosting). If they have specified "unlimited calls, and a mud server is not anything unnatural, this should be just fine right?
2. What are these "security risks" that everyone keeps telling me about and that I need to be aware of when running Apache or my own server on a public network? I understand that someone could send a thousand requests to refresh the page on my website, but I don't think there is anything I can do about that other than add in a delay for the page to load or a error checker that counts requests in a second and lock that IP if more than 3 requests have been sent in a second.
Also, if there is a forum, they can submit huge strings to the forum (although I could just add a character limit and timer for each person to submit forum posts).
This does not seem like anything to do with the server, but good coding practices. Why are these so-called "security risks" so bad that I should not consider running my own server unless I am ready to keep everything updated.
What updates would these be other than maybe the encryption software?


In other questions:
both django, and wordpress process graphics without me needing to hard-code imaging. Why should I use one over another?
Also, what is the advantage of using javascript rather than using pyjs or even native django or pyweb script combined with HTML5?


in still another question:
Is Amazon s2 the same as an ssh server like in motion hosting? Is there any way on something like amazon s3 to do checks for forums and what not? Can I install pyweb on a cloud host like amazon S3 or dropbox and use their native python and html mixtures or is that really when pyjs and or javascript comes in handy?

2014-08-14 01:09:46 (edited by camlorn 2014-08-14 01:44:41)

This is long.  I apologize for this, but this is an extremely complicated topic and your questions touch every part of it.
I'm not 100% sure on which parts are questions to be honest, but think I can clear this up at least a bit.  Let's run through a request on the internet, as I think this will tie a lot together.
When you enter something in your web browser's address bar, it does one of a few things, but the end goal is an ip address.  If you enter one, nothing else happens-it just goes ahead and opens a connection.  If it's a "domain name", i.e.google.com, the web browser does a more complicated algorithm to contact a domain name service, which you do not typically control yourself.  To get a domain name, you purchase it and, after some period of time (seconds to hours), it "appears" and starts working.  The domain name is everything before the first /, and the domain name service is like a dictionary that maps them to ip addresses.
This connection is a simple socket, and the fact that you're using http or https or whatever is indicated by the port.  Http is on port 80.  I'm not going to go into details here because it's highly technical and something that I don't fully know, but it is more than possible for multiple clients to connect to the same port.  Each connection is differentiated by the OS for you, and no programmer I know can explain the specific mechanism offhand because it's simply not important how it works.  What you can't do is have multiple servers on the same port, because the OS on the webserver needs to know which program wishes to respond to the connection request.  Everything here is actually informal, i.e. you can make port 80 ftp or a mud, but you'd not be obeying the HTTP protocol at that point.
Third, your web browser sends an HTTP request and HTTp headers.  This includes a bunch of standard stuff plus a bunch of browser-specific stuff.  It is these headers that allow for a virtual host, because the server can see which domain name you originally used.  I.e. google.com, camlorn.net, etc can all be on the same machine.
As for the server, it tries to handle the request.  Basically all servers know how to do standard html files and some Linux distros are even set up for it out of the box.  What you do when you set up something like PHP is tell the server that the URl needs to be processed by something else; this can be either a scripting engine like PHP or a full-on redirect to another webserver (which is what you do for something like Django).  The reason I'm not being too concrete here is that an HTTp connection is exactly the same mechanism that a MUD uses for telnet, just the MUD happens to "speak" http commands.  You can write web servers that just output the HTML for hello world, and the only thing that stops me connecting my web browser and at least getting a log-in screen is that the MUD isn't sending the browser HTTP commands back.
Finally, when it's done, the server closes the connection.  This tells the browser to go ahead and show it.
So now that I've laid that out some, let's clear up a few things.
Domain names point at a static IP address, which means you need a VPS or a high-end internet plan to use them.  There are some services that do redirects to dynamic IP addresses (i.e. changes every hour, like most home internet), but I've never had luck and setting up all the forwarding is more trouble than it's worth.  I'm not even going to get into NAT, because it's complicated and an unfortunate consequence of some bad decisions and underestimation in the 70s and 80s.
You don't want Amazon stuff  It is very very pricy.  Get a 5-dollar VPS and configure the web server or get a shared host and let them configure it all for you.  Amazon is aimed at very, very high-end enterprise people.  You technically can use it like a VPS, but it's not that simple.
A shared host is not a VPS.  The difference is that a shared host doesn't even typically give you a whole computer, and locks you down completely.  A vPS gives you full admin rites.  Ssh is just a connection program that you use, like a mud client.  The actual Linux shell it connects you to is something like 10x as powerful as the windows command prompt.  A VPS is a computer running on a computer, in the same way that something like VMWare works.  A shared host is just a folder you can put files in, sometimes with some additional features like database access and PHP; the advantage of the latter is that a shared host does all the setup to get things working right.  Under no circumstances use Dropbox, even if you are grandfathered into the old public folders.  This is not worth it in the long run, and you'll need to upgrade as soon as you're even marginally popular and cant run PHP or anything but static HTML.  A VPN has no relation to web hosting and is another topic which I can't talk about for lack of ever needing one ever.
Wanting to install something like twisted and your comment about unlimited calls doesn't actually make sense at all.  I'm not sure why you would do this, or what you mean.  At this point, you start connecting technologies that don't usually connect together, so I'm not sure what you're getting at.  Twisted is for making muds and similar applications, not hosting static web sites.  Servers do not themselves contain logic.  The usual flow is that the web server asks the twisted app a question, i.e. how many players are online right now.  Is there something specific you want to set up?
The point of javascript is that it runs on the client.  This allows some nice tricks like updating the web page without needing the user to click stuff or even writing in-browser graphical MMOs with sound (yes, really.  It even includes platforming elements).  javascript gives you client-side storage, graphics, and sound plus the ability to talk to the server.  It is not for the faint of heart, however; you have to support multiple browsers, all with subtle differences, so most people end up bringing in JQuery or similar.  Anything that compiles to Javascript has this same advantage, so Pyjs is probably just as good.
You seem to not be too clear on what Django and WordPress are.  WordPress is for people who don't know much programming: all the editing is done through the web site, and the feature set is tied down to blogging/articles.  Install it, log in, click create blog post.  Django is a programmers' framework: it can do blogging, but you're going to code your blog from scratch.  But it can also do games or whatever.
Don't worry too much about security.  Others disagree with this opinion, but my rule is to put only nonessential things on my public web server.  What people are referring to is the abilities for others to get access to your server to the admin level, pull out sensitive user information, or do things along those lines.  Without patching stuff, this can happen.  But the emphasis is can; someone has to care first.  Web security is an entire separate and huge topic, and there are people with degrees in it.  Doing basic stuff--always keep all the software on the latest versions, backup often--is a good idea, but going beyond that is basically in the realm of good programming practices, being aware, and doing homework on the best practices for your setup.
I can possibly help more if you're trying to achieve something specific.  Are you?  I'm leaving out a lot in the interests of time, and also because it's been a long time since I last looked at this.  My interests are basically digital signal processing, massive parallelism (I wish I had a project), and low-level networking, so I don't remember all the specifics of configuring everything.

My Blog
Twitter: @ajhicks1992

2014-08-14 02:12:13 (edited by frastlin 2014-08-14 02:14:21)

Hello,
VPN and VPS are not the same thing?
I'm not so much into how the http protical works and mor how the server deals with the info after it comes in and then what it sends out. What do you mean that muds don't send out http output? Why do you need to use tel-net for it? What kind of servers do you need to run a mud? Is something like what Swamp's server is the same thing? Twisted allows several threads to modify global variables and whatnot (I believe) so that I can have several people both talking to one another, kill mobs in front of one another and let everyone know who else is connected at the same time.
I am wishing to do 2 things:
1. Find some kind of hosting plan that I can host my current website on that will also allow me to play around in django and or pyweb.
2. Once I am able to understand how databases and servers work, I would like to create a multiplayer game, creating a client and managing a server.



In my opinion, audio games are super underrepresented in the multiplayer area and that is what I would like to head for. Is django something I should learn then? I kind have gotten turned off pyjs as the output is... not... the most... pretty... And I don't like ugly things on webpages.
It is also horribly complex and difficult to work with.
This is why I like keeping pyweb in there as the webpage code it has looks like:


<code>
$from scriptfile import ObjectName

<body>
<h1>This is a test script</h1>
<p>Hello $ObjectName.name, I'm frastlin!</p>
$if ObjectName.name == "fred":
    <p>You have the same name as my brother!</p>
$elif ObjectName.name == "frastlin":
    <p>No way!!! Me too!!! You are a thief!</p>
</body>
</code>


For django it looks a lot worse and I've gotten a little turned off. But if it is something I can use, I will go back into learning it.

2014-08-14 02:55:12

Do you know any good $5 VPS hosts?
I use Amazon S3 and it is far less expensive than anything I could really get anywhere else (It is $3 a month to backup all my computer's data).
So because I have it already, I've just been updating my folders with new programs and making those files "read only public" and giving people the link. This is great for static webpages! Is JQueary really the go-to for static development? What would be the best thing if I wished to eventually create my own graphical MMORPG like Runescape in HTML5?


Just for clarification, pyweb and django both have their own native languages that act like javascript.
They just don't have the same kind of support that javascript has and they need to be run through the framework in order to be rendered.


Almost all muds use port 4000. The port django defaults to is 8000 is port 80 and 8000 the same? (sometimes the second numbers will be the instance number). So is the most amount people can have on a website 1000 before they need to change the server?
When you say that only one server can be connected to a port at a time, that is a little confusing as something like muds all have port 4000 unless it is like 40000000000000412 or something like that.
And you get an IP address on a shared host, it is just... well... shared, although you can buy your own.
When someone runs their own open server through django, the url looks something like http://0.0.0.215-8000  or something like that. Why would I need to pay extra for that?

2014-08-14 02:55:59 (edited by stewie 2014-08-14 03:05:10)

OK so.

A VPS is a virtual private server. Essentially this is a server not hosted by you, but hosted by a particular company with their own server space.

A VPN is a kind of connection where basically you can more easily send private data between users. A VPN is not a server.

The http protocol is numerous, very detailed and has decades of standardisation (you hope). All it does is handles the sending of data and the security (what there is of it, anyway.)

Django is a python program that allows you to write web server code in python. Django programs are going to be more complicated, particularly to a beginner in this field.

Twisted is a networking library for python. I think it could be used, but it relies on a very rigid structure that twisted applications have to comply to. It all comes down to how experienced of a programmer you are.

Coding a web back end for a game will probably require more than one language. PHP, MySql could work. Or you could code parts of the game server in another language, IE you could use django for an interface between the web and python.

Creating a multiplayer game is far far more difficult than a singleplayer. You have to take into account more features that I can really name here. On the simpler side there are browser incompatibilities, rendering, design, text output, processing power, interface between server and host...
For server side you have space, languages, interface between them, methods of storage, optimisation. For larger multiplayer games your going to need more than 1 server probably. Multiplayer games are huge in scope, particularly an MMORPG. Games such as world of warcraft have teams of hundreds of people and years of production put into them, not to mention the budget.

A smaller game could potentially be completed, but a blind user creating a graphical mmorpg might run into problems with browsers. Text and audio could be done if the work was put into them, but I would personally focus on smaller projects first. Google tutorials on http and the preliminaries for internet protocols and implementing them into applications. Design single player rpg's and learn game design, which might aid on the multiplayer front.

Deep in the human unconscious is a pervasive need for a logical universe that makes sense. But the real universe is always one step beyond logic.

2014-08-14 03:16:26

Oh, thanks!
So is what is sent to django not http? :S
Are people able to host websites like this one on VPSes?
I was looking at vpses and digital ocean was $0.007 an hour for a (I think 20 gig) server with 500 MB processing. That is totally big enough with really what ever I wish to do. Why would I go for something like in motion hosting when a vps is so much less expensive, easier to use and scalable?
Is LAMP what most servers have and that plugins like shopping-carts or word-press use?
thanks,

2014-08-14 03:33:33 (edited by stewie 2014-08-14 03:34:31)

Http is a protocol for the tranmission of data. Basically with http, when data is sent, certain pieces of information are sent with said data, such as headers like Camlorn said above.

These programs are picked up by a web server, a program within a vps. A common web server application is apache2, generally the default for a vps.

this program receives the http request and interprets it. If the http request said "give me testfile.html", the apache2 or django or whatever would find the file, send it through any required programs, then send back html code to the request source. It would be the equivalent of saying "Bob, can you give me this file? My name is Stewie, and I am 3 feet to your left." Bob, ignoring this obviously weird approach to conversation would hand me the file.

Programs such as wordpress and shopping carts just integrate with your site. You paste the files, and they just run as any normal html pages would, running php or interfacing with sql, whatever.

Deep in the human unconscious is a pervasive need for a logical universe that makes sense. But the real universe is always one step beyond logic.

2014-08-14 05:10:34

We are speaking entirely different technical languages, so I'm not sure how far I can help.  You are not ready for anything resembling Swamp, if that's what you are thinking.
Start with the VPS.  What you are paying with at Amazon is not the full package you need, and the parts you don't own (because you do in fact purchase it peace meal) are not a good deal for what you want/need.  Put aside anything that has the word cloud in it.  Put aside anything that has the word spawning multiple instances in it.  This is aimed at high-end enterprise people, not you.  Getting and running Linux on Amazon costs by the hour, not the month.  Wait until black Friday, and VPSes with lifetime promotions fall from the sky; alternatively, look around yourself, just be aware that it'll take a bit of digging.  Bhost has some pretty cheep packages, but offers next to nothing in terms of extras like easy backups.  Before jumping on the let's make a game bandwagon, learn how to set up a basic web site on a VPS.  I.e. Lamp is not a piece of software, it's just an acronym for Linux, Apache, MySql, and Php.
I didn't want to go here, and my advice is to start aggressively Googling.  But let's talk about how data moves around.  At this point, you don't know enough about this to even talk about HTTP properly.  If you don't know programming, an internet project is usually not the place to start; if this is the case, go learn *anything* without the internet involved.
At the lowest level of the internet, below everything, is the raw IP packet.  This concept is so low and far down that only OS developers or network researchers deal with it directly.  If you try as a developer, you can't: the internet will actually block a lot of attempts to use things that are not TCP or UDP.  UDP is beyond the scope of this post, but I'll get to TCP.
At this level, you have almost nothing.  Not even ports.  The internet is an ocean and you can put a message in a bottle.  Since it's the internet, however, that bottle either gets lost forever or goes where you want-and no one else but the recipient gets to see it.
Let's detour for a moment and talk about what a protocol means.  If you've played a mud, you are actually participating in a protocol.  The mud has some fixed set of commands and, should you go outside this, it either says "Huh?" or gives you some info on what went wrong ("I don't see that here").  Nothing forces you to obey the protocol of the Mud's commands, but not doing so will never make useful things happen.  This is exactly what protocol means.
TCP is a protocol, specifically the transmission control protocol.  Through various complicated and nefarious means that involve both ends of the connection, it makes sure that data actually arrives, almost always.  Even if you unplug the internet for a second in many cases.  This is where the port is.  A port is simply a number from 1 to whatever, and a socket is an open TCP connection.  A socket is not a port, though it does connect to them.  After you establish the connection, you get a socket out of it and treat it a bit like a file on both ends-write to send, read to receive.
Now, to tell the OS that you want to accept connections today, you "bind" to a port.  More than one program may open a connection to a server on the same port, but a server may only listen on a unique port.  If you try to run two web servers on the same port, one will error.  A server is not hardware in this case, merely a program, so you can have more than one on the same computer.  Two different computers can use the same port at the same time, as it's machine specific.
So we have two things.  A way to say "I wish to listen to connections" and accept them.  A way to make sure that our data actually got to the other end.  Notice that so far there's no meaning to the port number at all.  maybe port 80 is the destroy-the-server program which, upon receiving a connection, deletes itself  Maybe it's your mud.  Run your Mud on any port and it will work fine, no problems-just so long as no one else bound to it first.
And back in the old days, that was it.  We didn't have HTTP yet.
But someone came along and said "Hey, here's an idea."  And then we said port 80 is the default port for HTTP, unless the URL says something different.  So if you enter google.com, your web browser looks it up and opens a socket on port 80, sends "get / http/1.1", a blank line, a list of information about itself in a prescribed format, and two blank lines.  This is called an HTTP request.  It then waits for whatever the server does, interprets it as HTML, and gives it back to you.  That's it.  The server does whatever-read a file, run a script, calculate the first 1000 digits of Pi-and then writes some HTML to the socket, which may or may not have any relation to what the server just did, even if it almost always does.
notice that I have not used the word HTML, Javascript, or anything.  These are not network technologies and only exist on the client.  Html merely says "I am an HTML document, here's how I look".  javascript is a programming language like all others and can run in a ton of places outside the browser.
Here's the deal.  This is still only the surface of the difficulties.  If you want to make a browser-based turn-based game like Kingdom of Loathing, this might be an exceptionally difficult first programming/networking project.  I get the impression you don't know much programming yet, so if this isn't true please feel free to say.  By the time you know enough to know fully what is involved with something like Swamp, you're probably still a couple years away from being experienced enough to say "and maybe I can do it".  The problem is that you aren't far enough along in your understanding to see how epic such a project is. There's three stages: what I've done so far was easy so this must be, wow this is a big project and I don't think I can do it, and wow this is a big project but I can if I decide to devote a very big chunk of time (years).  HTML5 is not the technology for a real-time Audiogame; not enough of us have it and the audio experience you can offer is going to be pretty sucky and require Chrome.
As for Twisted, I'm not even going to try to explain what Twisted is.  Twisted isn't even necessarily related to networking, but is used in that context so often that everyone forgets that it's actually a library built to enable a programming structure instead.  Twisted gives many very experienced programmers trouble, and we can't even talk about it properly until you've learned at least a bit about lambdas/closures/first-class functions, because that's what twisted is all about.  twisted is actually very powerful, if you can wrap your head around it; this is so hard that very many companies and people are turning away from it now, because inferior but easier solutions exist.  I must confess I am one of them.  And twisted does have some gotchas simply because it is almost ancient in the software timescale now.

My Blog
Twitter: @ajhicks1992

2014-08-14 06:17:34

So is TCP what muds use?
I do programming for fun, but still consider myself a novice. I use python and I have dabbled in HTML and Javascript, enough to know what I'm seeing when I read it.
Programming on board applications is completely different than web work, I don't think they are even similar.
I've made some web applications using pyweb and I've been going through the django tutorial and it seems pretty simple, just a little annoying to memorize for fun.
I used Twisted in a python-based package for building muds called Evennia, but other than that I just know that it can be used to create chat-rooms and the like.

But I have never had a good tutorial on how the web works, everyone just assumes that you know the basics already.


So, why can't I get a VPN running a django application excepting some protocol through a socket (probably TCP) that is sent by my onboard application? Something that would use this would probably be score-boards.
Then above that, one goes to the get and receive functions, so that you are then able to do an ai talker inside your onboard application. Then a transfer between 2 computers through the server which would be the send from computer A to server which would use the get function, then the server would send that data to computer b with the post or send function.
Am I understanding this correctly? It is probably very simplistic, but is that the idea?
Does django do both the sending from my onboard application, run the get and send function on the server, then receive the sent data on the onboard application?
It seems as if that is what it does, just along with a ton of other stuff...


On ports, it seems as if a port is server-specific, so I can run myserver.net with port 4000, you can run yourserver.com with port 4000 and people can connect to both our servers as long as you are on a different computer or network than me?
So, then have ports just gravitated to doing different things? 80 dealing with HTTP protocol 4000 being mud protocols and 8000 being django's home-server?
And if I ran 2 servers of django, the second one would need to move to 8001 or something like that because we are on the same computer? (I'm going to test this!)

2014-08-14 15:18:31

You're right on ports.
What kind of game are we talking about?  Are you trying to do Swamp or are you trying to do something like Core Exiles?  In the former case, you don't even use HTTP; HTTP is not really built for swamp-style gameplay.  It's actually worth noting that swamp isn't as online as you think: you are seeing different zombies than everyone else, for example.
And as for all this interconnection, if it's a browser game like Kingdom of Loathing, you only need Django.  Nothing more.  Django does everything necessary for that.
Swamp-like games are a whole different thing.  You typically use a custom desktop client and a hand-written server, not the browser.  Especially for audiogames, given the poor state of Webaudio-only Chrome has a full implementation, but nothing else on windows does.
When we get into games that talk to web sites, there's two or three ways that I've seen.  You can do what you're thinking and write an HTTP web-based API, but this is overkill.  You can have the game write to a database and have the web site read from that database (probably the most common and what I'd do).  You can use something like Websockets to allow the web site to ask the game directly for the info.  Pick 2 or 3, not 1.  If the game is something like KoL, just make the game the web site and don't worry about talking back and forth between 3 apps.  You seem to have a tendency to bring in way too many pieces.
I can talk a bit about how things like swamp work, if that's the kind of info you're looking for.  Aprone can probably say more, but we have different programming philosophies: I'd never, ever settle for the kind of tricks Swamp uses to get around network limits and would much prefer something fully shared, where everyone sees the same thing always.  This is what I am working towards with my blog articles and Libaudioverse (because why do it with less than the optimum audio experience?).  I think that I can, but am probably a year or so out from it.
Muds use TCP to transport data and speak the Telnet protocol on top of it.  There's not much to say about telnet.  It's just another protocol, only this time it's for talking about how wide your screen is and what you're typing at the mud client's input bar.
The tutorials on how the web works don't really exist anymore.  You can find them, but everyone seems to start off by hammering on Apache config files and the like until they work.  Most people do not care about how HTTP "works", and just want it to draw a smiley face or whatever.  Typically, that's picked up a bit later, and developing a mud makes it happen faster (because Muds do not speak HTTP there are not many frameworks that hide it all away like Django).  Keep in mind that the Django tutorials make Django look easy, but that it doesn't stay that way-it would be accurate to say that Django is probably the easiest approach I know, but there's a difference between easiest and simple.  If you really want to know the gritty and very technical details, there are official specifications you can read for all of it (I had occasion to look at the TCP spec exactly once. I would never do that again).  Most helpful, though, is googling "http introduction" or similar-there are some out there if you look explicitly for them.
And finally, keep in mind that the internet is not like normal software.  A bunch of different people worked on it all at the same time and all without close collaboration.  Then, in the 70s and 80s, we started linking all of them together; in the 90s, the web browser became popular and the internet exploded; in the 2000s, we started having a huge explosion of games that, by now, blow World of Warcraft out of the water.  The internet tends to be someone does it, someone else says good idea, so let's standardize it; most of us wish it was someone says this might be a good idea, someone experiments to see if it is, etc.

My Blog
Twitter: @ajhicks1992

2014-08-14 20:29:39

So Django is a http only framework? That doesn't make any sense, because django is just the receiver on the server isn't it? So couldn't I just tell it to except a packet of info that says:

<code>
name = 'frastlin'
score = 92005
</code>

and once those 2 lines of code are received by django on the server, can't I then take those values and post them in a score-board?
I wish to make a score-board first, something like what entombed has.
I don't think I need all the headers that http uses, I just need my custom lines of code. Almost like telnet, but 2 lines rather than 1. I wouldn't even need a protocol then would I? Or if I had one it would be called My custom Game Protocol wouldn't it?
I don't really wish to make muds, but I can still go to Evenea if you think it would really help, but I use VIP Mud which hides what it is doing through my IP address.
Granted, I would learn how to upload code to a vps and get it configured, but I still don't get the knowledge of what it is doing to send or receive code.
So how can I make a score-board like what Entombed uses?

2014-08-14 21:18:29

There isn't one defined way to do that.

Regarding http and Django, it would be more difficult to handle it in a non-http mannor.

Like you said, there are a few options.
1: A custom server side program that can receive incoming connections and sort them into the scoreboard as required.
2: A php file that receives an http request with standard parameters like score.php?name=bob&score=12345.

The main issue here is the security of it. These programs are generally easy to hack if you have the ability to view a program's network traffic, etc.

Deep in the human unconscious is a pervasive need for a logical universe that makes sense. But the real universe is always one step beyond logic.

2014-08-14 23:10:09

So if I wish to do something simple should I take a look at PHP?
I am not too worried about security at the moment, I think that will come after I have a working server.


I googled what an HTTP request looks like and I think I could totaly use that format for sending my requests. For posting on score boards, I could send an incripted file with the info and then decript it on the server.
The request would look something like:

<code>
POST /path/file_with_score.sco HTTP1.1
Host: www.myserver.com
From: my-game-name
Name: username
Key: RegKey
Content-Type: allincoded-score

randomlyincodedtextthatcanbeincodedontheserver
</code>


Yes it could be hacked, but it would be kind of annoying I think... and it would be perfectly easy to create something to incript and decript that into info I could use.
I just need to know how to create both a cliant and a server in python.
I looked at the python library and what do you know, there is a socket module! So perhaps I can create a mud client like VIP Mud, just much more simple!
That way I can start with what I know which is onboard programming and practice sending and receiving info.
Does this sound like a good place to start? Then I can worry about server stuff once I'm comfortable with working cliant-side.

2014-08-14 23:13:12

That could work.

When sending data though your not sending a file as such, your sending raw data. Think of it as a flow of water between you and the recipient. socket wlil usually make sure your data is securely transferred.

Deep in the human unconscious is a pervasive need for a logical universe that makes sense. But the real universe is always one step beyond logic.

2014-08-15 01:35:41

Sockets are not HTTp and Django is not sockets.  Django does one thing and one thing only: HTTP responses.  This includes, as you have no doubt seen, all the other stuff to make an app out of them.  But it will always be HTTp.  Your scoreboard example is indeed fine, in that it is a client using an HTTp API to talk to a server-this is how I would do it too.
Security is a concern, however.  Those HTTP requests-should I know the format-are trivially easy to do.  I can use my telnet client, curl, or any number of other programs.  You would also want to use SSL here, because then I can't trivially use any of a number of command line utilities or GUI programs that others have written specifically to intercept and fake requests like yours.  You debug your app with one of these, so obviously the hackers have it too.
Please specify the end goal here.  You are flailing around blindly among a bunch of technologies for a bunch of different purposes, and have not clearly specified what you want when you are done.  As a result, I am flailing around blindly and providing very broad explanations.  I think more focused discussion aimed at specifically what you are wanting to do would be beneficial at this point.  Making the mud client, for example, is only helpful for a certain type of thing-learning raw socket programming-and won't help with HTTP explicitly.  Can you lay out exactly what you want to end up with at the end?  There are at least 3 directions you can head in, each only tangentially related to the others and each accomplishing slightly different things.

My Blog
Twitter: @ajhicks1992

2014-08-16 19:01:52

I wish to make a score-board for my game. So when people finish, they can click "submit score" and there will be a list of other people's scores that their score will be compared against.
I wish to do onboard programming, developing my app in python and C++, then just use the server for storing data that is submitted to it from the onboard games.
Eventually I would like to have something like a store like Entombed has, and depending on how complex that is, move on to a chat client, and then storing and sending variable data, so everyone is sinked with the server so there can be multiplayer interactions.
But first I just wish to create a score board.

2014-08-16 20:16:16

Everything you've mentioned except full multiplayer is actually not so hard and, for a certain type of game, multiplayre isn't either.  In this case, since it's a client talking to a server, all you need is Django and perhaps the Django Rest Framework.  You can do without that last one, however.  Django will give you scoreboard serving in a day or two of work, though I make no statement as to the difficulty of the client (this varies widely depending on programming language).  I'd suggest looking at requests as well, a python library for calling HTTP-based APIs.  I started learning these technologies, but am much more versed on how you make your own protocols and use the socket APIs, as this is what both of my MMO prototypes do.
keep in mind that knowing exactly how HTTP works, i.e. low-level socket programming, isn't necessary.  You need the basics of the internet, but after that point you can just pretend that it's opaque and magical.  This is what Django lets you do.
The store is also the same sort of thing, only this time you need a request for the purchase and an error response.  The error is required when two players might buy the same item: the request that gets there first wins, and the other player needs to know that that item no longer exists in the store.  If done incorrectly, an online store can lead to item duplication bugs.  The severity of this problem depends on the game.
You can do board and card games through HTTp-based APIs, but you're now getting into places where Django will start to fall over.  Django is all about responding to requests and stopping and not so much on-running processes.  You can model a card game with django, but doing things like placing time limits is possibly a bit harder.  Not impossible, but I'd start looking to something else.  For something like a card game, a custom JSON-based protocol is what I'd consider.  Alternatively, there's HTTp streaming, but neither fits well into Django as far as I'm aware.  This is the point when you want to maybe look at Twisted or Gevent.  It's worth noting that Gevent looks great at first but falls down hard unless you really think hard, as it has a lot of "magic" going on.  The difference between this and the previous two things is that you probably want to keep the connection open, and Django is not at all aimed at that (the end of an HTTp request, unless it's HTTp streams, is the socket being closed by the server).  it is worth mentioning Python 3's asyncio, which appears to combine the best of Twisted and Gevent and has people involved from both of them.  I have not used this framework, but it does look really nice.  If you look at Twisted and go "Wow, that's complex", you are so far from alone that I can't even describe how far from alone you are.
A chat server is the same problem as a card or board game, but easier because you needn't worry about game logic.  In both of these cases, security is a concern-especially if the intent is commercial.
Full realtime multiplayer games are really, really hard algorithmically.  The reason for this is that the network is not instant.  If I do something, it takes anywhere from 20 ms (UDP, which adds a whole new set of complexities, and I'm on a really good high-end internet package) to half a second or more for the server to know about it.  The server then communicates this out to other players, all of which take anywhere from 20 MS to half a second or more.  Syncing the data is easy.  Making the game feel like it's not molasses is really hard.  If you design it poorly, both in coding and gameplay terms, I don't find out about things soon enough.  If you shoot me and that update takes a whole second to get to my client, I have a whole second where the server thinks I'm dead and I don't know I actually am.  If the client has to wait on the server to say okay before letting me take every step, my movement speed depends on network round-trip time; everyone ends up moving at different speeds.  These are surmountable problems and I'd suggest looking up the quake and half-life protocols to understand it better.  Once updates need to be noticed in less than 5 seconds or so, you need to start being creative.
Alternatively, you can cheat and make the server very light-weight, sharing only a bit between them.  In this case, you make everyone basically play their own game, except for some shared elements.  Games in the sighted world that do this include Demons Souls (you can leave messages for others and invade for PK, but it's never more than two players playing together).  I've heard that Swamp does a lot of this kind of thing from a lot of people, but never played it far enough to make it glaringly obvious to me what it does and doesn't share.

My Blog
Twitter: @ajhicks1992

2014-08-16 23:59:17 (edited by Sebby 2014-08-17 00:19:24)

This is all very confused, so I'm just going to take shots at bits of it and hope your question somehow gets answered by this thread.

A web host server can have multiple IP addresses assigned to its interfaces. Further, during the HTTP request, the client indicates which host it wishes to retrieve content for (HTTP 1.1 only). By these means, a hoster can simply discriminate requests, and serve up the relevant files for the relevant hosts. No virtualisation required. There's a control panel, like you noticed, and they'll let you drop stuff into your web space which may include scripts, specially PHP. Lots of people share these machines, unless it's "Dedicated".

In general, atop TCP/IP runs a server program. That program can be Apache or lighttpd or nginx or some other dedicated web server, which knows how to communicate dynamic content with a "Back end" such as CGI, FCGI, SCGI, or reverse HTTP proxying. It can also be a so-called "Web framework", which in effect is a development environment for web programmers that incorporates a built-in web server. Sometimes it's useful to have both; the web server hosts static files, and then proxies requests forward to the web framework for your dynamic content. But PHP, by far the common and garden variety web programming language, can be implemented simply using FCGI, where the web server communicates directly with the PHP interpreter using a private protocol. For Apache, the biggest and well-known of the web servers, it's even easier: you simply build PHP into the web server itself, as a runtime module; now Apache itself is responsible for executing PHP. Similar comments apply to other languages for which a module is available, like perl.

VPS (not VPN) hosters virtualise machines, not just web servers. This is unlike some shared web hosts which give you ssh access, as a convenient way to access unprivileged commands you may need, because there is no isolation and you can't install new software very easily--only web applications, which communicate with the outside world using facilities provided by your web host, such as the mail system, that is shared among all users of the host. Even when using a VPS, computing resources like CPU and memory may not be evenly distributed among the virtual machines, and it sometimes happens that you can't do certain things on your VPS because of the virtualisation technology in use such as running a VPN server, so you need to pay attention to where you get your VPS from and what they promise, but this way you have complete root access and can install software and run it economically. I think VPSs are the way to go, because you do get maximum flexibility--unless you purely want to host static content, in which case the economics of scale are in favour of a content delivery network (CDN) such as Amazon S3+CloudFront. Amazon EC2 isn't really VPS, more like unit cloud computing, but you can sorta pretend that it's a VPS if you squint hard enough; still, for inexperienced people there are better options out there which are less broken out of the box, such as Linode.

To be honest, my advice for network programming in general is: Keep It Simple, Stupid (KISS). smile If you can get away with a single TCP connection to a central server, or just using HTTP, then just do it. Nobody's going to hold it against you if they have fast enough connections to the interwebs. Yes, action games will profit from more advanced networking using UDP and NAT traversal and client-side prediction, but unless you actually need those, don't bother. You can use various languages to write nice, clean, event-driven code, like Python Twisted or, my favourite, Tcl. Or you can use libevent in C/C++, which provides a nice abstraction around an event loop.

Finally, I do need to point out that there are services offering scalable web infrastructures, if that's all you need. Google has AppEngine, Rackspace has Cloud Sites, Amazon has Elastic Bean Stalk (ugh!), etc. You can offshore the entire process of developing a web application to these guys if that is all you need; this includes the server side of any HTTP score-board, or whatever. For AppEngine, this is free up to a certain traffic level, and all you have to do is learn either Python or Java. You mentioned Python, so perhaps you should take a look at that option.

Edit: oh look, now AppEngine supports PHP and custom domains too! Well, will you just look at that? smile

Just myself, as usual.

2014-08-17 02:32:57

Isn't it possible to have django send commands to a client after a set time?
so have something like:
player sends:


player: "fred"
command: "open the door to the dragon"


on the cliant I have a check:
if "open the door to the dragon":
    if not check_for_arg()
        sleep(5)
    if not check_for_arg()
    do_something_bad()
    else:
        do_something_good()


Then that can be running on one thread while the player is getting stuff from their client and other threads.
I don't know if a full on program can be run from a server, running many different threads at once, but I don't see why not...

2014-08-17 04:18:20

No. You do not stay connected.  HTTp request/response pairs end with the connection closing, unless we are talking about HTTP streams.  HTTP streams are a very special case, almost a hack, and Django does not support them in the manner you need.  I'm not aware of anything that does.  Django gets a request and answers the request.  It is bad practice and I believe requires a bunch of extra steps to keep the connection open for a long time because that blocks other requests or otherwise degrades performance.  That is all.  Django is not, in any way, for realtime online games that resemble Muds.  Django will work fine for a scoring system because that is a request/response system-I ask for the scoreboard or say that this was my score today.  Same with a store system, as that's got the same idea behind it-I request the list and say "buy x of y"-note that this doesn't solve allowing me to trivially lie about my money, and you have to completely trust clients to be telling the truth.  By the time we're on to allowing two players to trade directly, we're talking about custom servers being easier by far, and anything beyond that or anything with time limits means it's probably time to pull out the socket module and make our own protocol.  Other things can work, but that's probably the path of least resistance.
You cannot ever make a connection from a server to a client; the client must always initiate.  This is true for 95% of users; the remaining 5% are those who know how to set up port forwarding and whom you have provided the needed info to.  Do not be fooled by a technique known as hole-punching-you can't connect to my computer in any way unless software on my computer connects to your server first.
I hate to say this because it's discouraging.  You may eventually be able to do an online game.  You are far away from it, however.  The amount of information missing from your understanding of how this works is large enough that conveying it here simply isn't going to work.  You need to research on your own, possibly looking for books or tutorials.  I'd start with "how tcp works" at a search engine, followed up by "how http works".  This article about using sockets with Python is how I put my first MMO prototype together.  My blog contains an entire account of my adventures, but I didn't tag them so I'd have to link all 7 or so articles individually (maybe I'll do that and post here again).
My suggestion is this.  Either make a single-player game to develop your programming skill more, develop some sort of simple online software like a blogging engine with something like Django, or both.  You can add a high score system and a chat system to a single-player game, but the things you are suggesting will not work.
Be very scared of the word thread.  Very.  Incredibly.  The t-word is not the friend of programmers.  The t-word is not always a mistake, far from it, but threads cause a very large number of problems-especially if you don't have a good understanding of how to use queues properly.  Greenlet in Python is slightly better, but also pretty evil.  Beware of client-side simulation, because that's so easy to hack it's not funny (I can have whatever just by saying that I do, and you can't check it).  It's also very difficult to get right.  Clients should never identify themselves explicitly in requests: this is also unbelievably simple to abuse.  The trick is to have a login procedure and to keep the connection open; you can identify me by my socket on the server, and then Alice can't pretend to be Bob.
And drop HTTP.  HTTP is a request/response system, that is you get responses from the server to requests from the client.  It seems to me that you have not fully grasped what this means.  To repeat it one more time, the server does nothing at all unless the client asks for something.  You're going to need to learn socket programming.  And I take back everything I said about not worrying about security: the things you are suggesting can be hacked in five minutes or less, most of us have the packages and whatnot installed to do it already, and the hacks can be bundled in nicely distributable scripts.  I thought you were talking about locking down an Apache/Nginx configuration, not security in your code itself; I gave advice accordingly,.
And finally, look at JSON.  Most people do not shove all the information in the header like this-you have the ability to send along arbitrary information with post, so why not serialize entire python dicts?  Django does contain support for this if you can use HTTP, otherwise you can use null-terminated strings over a socket (there's advantages to length-prefix, though.  See the howto, it goes into that a bit).
But really, I think you need to go to Stack Overflow.  They've got people who do a generally good job on well thought-out questions, and you'll be reading it all the time as a programmer anyway.  Start with asking them about where to learn basic networking concepts, or better search stack overflow for such questions (I promise you can find at least 10).  You will also get better results from Googling, as your questions are still very basic-if you aren't finding things, it's not because they don't exist.  Developing Googling skills as a programmer is almost as valuable as the programming itself.  I think there's maybe 10 of us on these forums with any experience at all in this area, and most of them seem to not be showing up on this thread.

My Blog
Twitter: @ajhicks1992

2014-08-17 05:02:51

Thank you for all you have told me! I think I've got enough to start googling as I know there is something called a socket, something called tcp and http and after I read up on those, I can go back through what you've written and hopefully I'll understand more. I'm not particularly keen on writing my score-board today, so when I am ready to do that, I will come back to this. I just wished to know where to start. I will play with the socket module and probably use google's free server for little testing things.
thank you,

2014-08-17 05:40:08

I would personally advise getting really comfortable with networking from both the windows and linux side of things. If you plan on going cross-platform, your going to need to know IP stack, network layers, that kind of thing. If you plan on doing multi-player online games, making sure everyone can have as lag free of a connection is pretty paramount to the over-all experience. I haven't even mentioned game code yet, i'll leave that rant to Camlorn, but just from my experience with windows and linux system admin configuration, networking is a big part of the picture and you would do well to study up. Keep in mind i do a lot of this configuring stuff as a hobby, so i'm not saying this to put you off of your project. Just keep in mind your going to have to be willing to put in the days, weeks and months to learn your craft. It's no fun to have people hold your hand and do it for you. Certainly ask for the right path to take, but your going to be the one grinding out the product. Good luck, and have fun.

2014-08-17 22:43:30 (edited by frastlin 2014-08-17 23:10:16)

So I did some googling and found
PodSixNet
a python package that will do exactly what I want. It sends info from the client to the server, has code for the server to send info to the client and I just need to run the check once in my game loop to do all the checks. I'm going to play with this and see if I can get it working! It has a nice little chat example as well.
Here is a tutorial for a lines game using pod6


I'll see how it works and hopefully will understand a little more after working with it for a while.


*note*
After a little more searching around, I found:
Astral Networking
A more advanced library to stack on the top of podSix and it is built to work with pygame.
The only downside is that it is version 0.0...

2014-08-18 04:02:16

Be careful when choosing to use less popular packages; Twisted, Gevent, and Asyncio are written and used by many, many people.  I'm not saying that you should disregard the packages you have found.  But if your code starts getting ugly, it may be the fault of the package and not yourself, so keep that in mind.  Also, note that LGPL is arguably GPL when using Python, in that you basically can't do anything to close-source the client effectively and still obey the license.
Part of network programming is not blocking.  This means that you can never explicitly wait on anything.  Twisted achieves this through the concept of deferred, which is confusing the first 6 or so times you see it.  Gevent achieves it by being threads-but-not; other "greenlets" get a chance to run when you read from the network, so you can make blocking code that magically doesn't.  Asyncio combines these: you can code in the gevent style, but must syntactically note what is going to block (solving the a calls b calls c calls d which reads from the network, which is one of the big issues with Gevent).  If you want me to talk about why Gevent is not a free lunch, I will.  For a game, it ends up not mattering which of these you use, believe it or not.  There is a game loop on the server which processes available messages from clients every tick and queues up new ones to go out.  The networking should probably be kept completely separate from this loop, or so shows my first failed prototype.  I've seen the problems I am describing in real code.
Regardless, the first time you do network programming for the server side of things "right", it's going to feel very wrong.  Incredibly so.
Here are the estimates I have given myself for something akin to Swamp but more mud-like (quests, talking NPCs, etc): 1-6 months to finish Libaudioverse and write a simple game using it, 6 months to write an online server/client pair that contain everything needed for building, scripting, and basic gameplay, and an additional year for testing and worldbuilding before a launch.  Doing this successfully is not about knowing a specific programming language, believe it or not.  You will get much, much further with creative architecture; turn-based stuff is simple, i.e. board games, but realtime games definitely require it.  My blog posts may not help you yet, but I'll poke this thread with a link once I have the time to go tag everything, as I need to do a bunch of small stuff for my web site soon anyway.

My Blog
Twitter: @ajhicks1992

2014-08-18 20:27:02

Podsix is built for games, so hopefully it will not be that ugly when writing games. Also, there is a game engine that is like a wysiwyg game engine that uses podsix.
I don't know how it would be when trying to write a website though.
Twisted, Gevent, and Asyncio are all general pirpus and what I should eventually learn. Is django something that hides all of this stuff and just lets you worry about the website and content?
Asyncio honestly looks like the easiest of those three you said, and I think it is built-in...


In other news, I am eagerly eagerly eagerly awaiting libaudioverse!