You are not logged in. Please login or register.
AudioGames.net Forum → General Game Discussion → AGAIN Archive: 403 Forbidden
Hello. So recently I got a new computer and have tried to download the games by Jim Kitchin on the archive. However, every time I try and go to the developer page, my browser gives me a 403 error. Any reasons?
Yeah it's happening to me too and it's really worrying.
Yeah. I'm hoping it's not being targeted because of some copyright bot, but no one will know...
Why would it be? And isn't it on the archive?
I've just tested both the main developer listing and Jim's page, and both load normally.
It is very possible you're being impacted by a number of bans we put in place after last month's abuse if you're unable to access the entire website.
If this is the case, email our website at agarchive.net contact address, and I'll do my best to help you get the games you want and see if I can narrow the bans to exclude legitimate users.
Unfortunately our traffic is still higher than is acceptable, so the ban number is only likely to increase in the future, unless we're able to come up with a better way to reduce illegitimate traffic (users downloading the same file repeatedly).
It seems like every third page for me is consistently getting the 403. And I haven't downloaded anything from you guys for a month.
@5, do you mind elaborating on the bans you have?
@5: If this unwanted traffic follows the patterns of what I've seen in the past, it's most likely poorly or maliciously programmed bots. They don't care that they're downloading the same files over and over and over again. What's more, even if you move those files, remove them, etc. they don't seem to care that they're now getting HTTP error responses, they just keep hammering on the same URLs again and again and again.
If you have access to the server as Root, I'd suggest investigating Fail2Ban. I'm pretty sure it has a module which will let you ban IP addresses generating floods of errors at the Iptables level, which means they will start getting absolutely nothing at all back from the site for a while, and each ban is logged so you can find out where your trouble's coming from and take appropriate actions.
Hope this helps.
+3
Try to use cloudflare. Heard it caches pages and haves an under attack mode which waits 5 seconds and then redirects to your website. This should stop robots and decrease traffic.
+2
I was looking around agarchive a few days ago. When I clicked on the developer listing link from the news section, it 403'd, but the second time I tried it, this time from the home page it worked fine so the issue seems to be inconsistent.
+1 to both the people suggesting cloudflair and fail2ban. Either of these should make your life easier, I'm all for the service you're providing, game preservation is extremely important in my opinion.
Agreed. Much like there are sites for older pc games, I also agree that the audiogame archive is very important.
@All: I really appreciate the support for the site. It's a lot easier to deal with these issues when there's users clearly getting a lot from the project.
It's also worth noting that I'm not accusing anyone here of the abuse. If someone reading this thread is indeed treating the site like their personal internet-connected hard drive, then please stop breaking it for everyone else, but I don't have significant evidence one way or the other.
I've sent an email to Skynet, and hopefully we'll be able to start digging into this issue further. If you were getting hit by a ban, it would be every page, not just some of them.
Specific post responses below:
@Ethin: yes, sorry.
@Jaybird: a lot of this looks like legitimate traffic, especially since it's been varying heavily over the month. The one obviously broken user from June has not been back, hopefully because they too received a massive fine from their ISP.
@Mazen: Cloudflare is definitely near the top of my list of possible solutions, depending on how the next several weeks pan out.
I got a 403 error as well, but I can go to the homepage, the news page, and some others just fine
I sometimes get it and suddenly after few refreshes it works again.
I did, a few weeks ago, download quite a few items, but I wasn't trying to spam, I only was trying to see if that thingy that was supposed to fix the BSC games running on Windows 10 actually worked. So, I still have the error. Even though I wasn't trying to spamm, again, I didn't download the same file so can you unban me from the archive
As I've said previously, if you were banned, you wouldn't be able to access anything. That means refreshing the page wouldn't help you.
I'm looking into this issue as best I can, but especially since our high traffic is continuing, the only thing I'll be doing with bans is add more.
For the record, I don't care if you download a few games, or even a few developers. What I care about is you downloading the whole archive at once, especially if you do it repeatedly. Here are some traffic statistics for July (numbers in gb). To give you some context, the entire content of the archive, that's every game and recording on the site, is just over 37.
July 1: 24.8
July 2: 187.5 (unacceptable)
July 3: 22.0
July 4: 90.5 (unacceptable)
July 5: 33.8
July 6: 25.6
July 7: 12.5 (mostly acceptable)
July 8: 15.5
July 9: 25
July 10: 4 (normal)
July 11: 89.1 (unacceptable)
July 12: 22.4
July 13: 41
July 14: 7.6 (normal)
July 15: 18.2
July 16: 34.5
July 17: 21.3
July 18: 38.4
July 19: 23.9
July 20: 34.5
July 21: 51.9
July 22: 27.4
July 23: 30.8
July 24: 8.2 (normal)
July 25: 6 (normal)
July 26: 42.5
July 27: 40.4
July 28: 60.4
July 29: 104.4 (unacceptable)
July 30: 6.4 (normal)
July 31: 80.1 (unacceptable)
Here are monthly stats for 2019, keeping in mind that the issue started in june:
January: 206
February: 193
March: 243
April: 264
May: 209
June: 2.55tb
I would like to work with a couple of the people experiencing the 403 issue directly.
If you have been blocked from accessing the site but were able to successfully download a file either by retrying or waiting a while and you have a moderate degree of technical familiarity (or can follow instructions extremely well), please contact our website address via email. When you do so, please include a specific file you tried and failed to access (game or page), and the most precise timestamp you can give me.
I've made a change to the server that may resolve the issue, but unfortunately I may also need to revert it without any warning.
Well, I can access the site, so not sure if you want my input. But I can tell you that after checking every category, and a few of them three times for good measure, I no longer have any 403s.
I also downloaded like 10 of the smaller files from various categories and it worked fine.
Well I'll be damned. My server rarely ever gets such conditions. Hereare my baidwidth stats:
Total: 413.68 GiB received, 509.43 GiB transferred, 923.11 GiB total (since 2019-01-05)
monthly
rx | tx | total | avg. rate
------------------------+-------------+-------------+---------------
2019-06 77.71 GiB | 148.27 GiB | 225.98 GiB | 748.89 kbit/s
2019-07 38.91 GiB | 64.50 GiB | 103.41 GiB | 539.45 kbit/s
------------------------+-------------+-------------+---------------
estimated 63.29 GiB | 104.91 GiB | 168.20 GiB |
daily
rx | tx | total | avg. rate
------------------------+-------------+-------------+---------------
yesterday 2.24 GiB | 4.88 GiB | 7.12 GiB | 708.12 kbit/s
today 56.83 MiB | 56.48 MiB | 113.31 MiB | 186.37 kbit/s
------------------------+-------------+-------------+---------------
estimated 962.76 MiB | 956.78 MiB | 1.87 GiB |
So, yeah, my server gets a lot of traffic, but since I don't have daily results (I'm using vnstat), I can't give you extreme specifics. Plus my DB was inactive for about 3 months, so you have to take into consdieration that downtime.
@John: Have you considered offering a BTSync or similar of the archive so those who truly want absolutely everything can grab it all with a minimal of fuss, and stay up to date? The nice thing about some of these distributed systems is that people who are downloading are also helping you and others by uploading what they already have to others in need, so the entire bandwidth burden is distributed among the clients.
that is a good idea, post 21
As long as the website still exists for those who might not have or use bt sync anymore, as it's been a while since I've touched that program.
I don't see why both ways couldn't exist concurrently, much like how, for a time, the Blind Cool Tech archive was hosted on the now-defunct Blind Geek Zone site, but also in a BT Sync folder which those who wanted the entire archive at once were encouraged to use.
AudioGames.net Forum → General Game Discussion → AGAIN Archive: 403 Forbidden
Generated in 0.024 seconds (57% PHP - 43% DB) with 9 queries