Server problems (split from "Anacreon Era III Discussion Thread")

[Oddball]

Update: Unable to connect. Load screen comes up and then after a long while error: Time out

[TheBugKing]

I am fairly certain now that the reason the site is not loading is because of the shear amount of forces being moved (I see you Imperium). This must be overloading the server capacity somehow. I'm not a super nerd, so I can't be certain or offer awesome tips to fix it, but that seems to be the main reason for slowdowns in relation to movements. If someone moves 2 or 10 million of anything, it gets pretty choppy.

But this also is not good for playing, obviously. I hope George can work this one out quickly. We spent a month playing and building up our armies, but now they are so big they deny us access to the game when the real part of the game comes up.

Edit: I've decided that I will not bother myself with this sort problem again. It is taking 10 min get into the game and then times out at the next cycle. I can't move anything or do anything in the game screen. If I see that Imperium came in and wiped me out because of this I won't be coming back to this game for a very long time, if at all. The way we play the game is with massive forces, if the game can't handle that then it just isn't worth playing in my opinion.

[Oddball]

I abdicated. I couldn't move or complete attacks, bugz was able to take my southern sector as well as start on my main sector. I will try again with era iv. this time at the beginning not 2 weeks late lol

[Xephyr]

Something I noticed that may be related is I keep getting kicked off the devwiki; I'll return and be logged off, but going to the login page and selecting "use secure connection" I'll be logged in again without entering anything.

Could be that the anacreon server has a similar issue with secure connections?

[starxplor]

It definitely seems like certain times are worse than others. Sometimes, I have to reload many times to get a single successful game load, while other times, I get 30 minutes straight of perfectly fine response times to managing an empire.

EDIT 1:
Seriously, this game has become almost unplayable. I have no idea how anyone can do anything, let alone maintain a large empire like the big three empires from the other list.

[--Imperator--]

I've still been experiencing server issues as of this morning, exactly the same as other users have described, and since it's been for more than a week I thought I'd chip in as well. Yes, it's unplayable, however at times I CAN get a stable connection for about a few hours, then the timeout errors resume. Oftentimes I get the cloudflare "page is unavailable" message, which leads me to think it's an issue on their end but I haven't had any issues on other cloudflare based sites I frequent...

Edit: @TheBugKing, the units moving or in combat are just numbers and math. Shouldn't strain the server that much. I didn't have any issues moving millions of units around yesterday, and you were doing the same.

Edit: yeah, it's all working now. I also received a system message in game about a server reset.

[george moromisato]

I've been debugging this by trial and error, because I can't reproduce it on my end. There are lots of things I don't know, so I'm just adding debug code in various places and waiting to see if they yield any results.

Some of my latest findings:

1. I don't know if this is a network/communications problem or if it is a server responsiveness problem. Maybe it's both.
2. There is definitely a problem with garbage collection on the server which would cause timeouts, but times don't match up with when people have reported problems.
3. Turn update time continues to be normal. The maximum time it has taken the server to update a watch has been 400 milliseconds.
4. It's possible that in some cases the server takes too long to send data to the browser. This might cause a problem for other clients connected simultaneously, so I need to check this.

[TheBugKing]

I would say that it seems whatever updates you pushed through worked pretty well. Imperium and I were moving some pretty large forces around the map just now and it was as smooth as butter. Maybe it is fixed for awhile? I have a hunch that the game will be pushed to the edge over the next few days though, let's see what happens!

[starxplor]

It was working well, now I am getting errors again and just got a cloudflare error:

Error 502 Ray ID: 3604e4f404f42573 • 2017-05-17 07:34:27 UTC
Bad gateway

EDIT: Including more from the cloudflare error page:

You
Browser
Working

Chicago
Cloudflare
Working

anacreon.kronosaur.com
Host
Error

[starxplor]

Running into more issues with the game not loading properly.
First, got the timeout error, then trying to reload the galaxy it just hangs at connecting.
When switching to anacreon2 link, it loads immediately.

[george moromisato]

I've made some progress in debugging this, but I'm not done yet.

1. There's still a chance this could be CloudFlare caching problems. The fact that anacreon2 works usually indicates that the direct connection is working better than CloudFlare. But it's also possible that this is a transient issue and the minute you switch to anacreon2, the problem goes away.

2. I added debug code to check connection and traffic volume. So far, the volumes are far below the maximum capacity of the server and network, so I don't think there is a traffic or bandwidth problem.

3. I found a bug which caused the server to ignore certain HTTP messages under certain circumstances. This would definitely cause time-outs, but I'm not sure how common it was. In any case, this bug has been fixed for a couple of weeks, so any recent problems have a different cause.

4. I also found a bug which could have caused the server to stall for several seconds--probably not enough to cause a time-out, but enough to notice. I fixed this bug a couple of weeks ago.

5. My latest theory is that saving the game could be interfering with the server. Saving a game takes about 4 seconds; and we save about once a minute. It's possible that saving causes the database to stall until done (probably due to a mutex). If there are four games active, we could be waiting up to 16 seconds in some cases, possibly enough to trigger time outs. I'll debug this over the next few weeks.

[starxplor]

Thanks for the work on this, I know how much transient issue can be a pain to deal with.

Haven't had any issues so far today, and the issues the other day were the first time in a while (since back during the bug-imperium war). I was doing a lot of changes to trad routes prior to the issues, I could imagine calculating all the production and import/export could bog down a server or as you mentioned the save-game, lots of updates needing to be written out.

On that note, I have noticed a cyclic issue with production, example:
Planet A makes aetherium, constant quantity.
Planet C makes chtholon, consuming more aetherium that planet A produces.
Planet T is trade hub exporting chtholon, more than being produced on C.
10: At first, planet C tries to import aetherium, fails to get as much as requested, and starts up its own production of aetherium to make up the difference.
20: Next, planet T tries to stockpile aetherium, even with small surplus imported vs exported.
30: Then, planet C's request for more aetherium than previously received causes planet T to export more, dipping into its stockpile.
40: With more imported aetherium, the workforce moves more toward chtholon, increasing demand of aetherium.
50: The trade hub then runs out of surplus because of the higher demand, causing a missed quota for export.
60: Now, planet C increases aetherium production at the sacrifice of chtholon production... again....
70: Jump back to line 20

[george moromisato]

Apologies! There was a power outage and the server was down for two hours.

It is up again and running.