Seibertron.com Energon Pub Forums

Just an update on Seibertron.com ... our database server has been problematic for the past few months. I had been putting off fixing it until after the new year due to how busy our store has been. That server was behaving very much like Seibertron.com's old servers back in the day when the hard drives would go bad, but I've been on cloud servers with SSD for past 8 years and haven't had a problem with these servers since then.

2 days ago, the database server had other things in mind and ended up corrupting the OS and much of the main Seibertron.com database which made fixing the server a top priority. Fortunately, I was able to restore the database from a day old backup of the server so not much data was lost (if you posted on the forums 2 days ago, I apologize that your posts for that one day were lost).

Anyway, after spending time with tech support yesterday and not getting anything resolved plus spending another day working on the database server trying to fix it myself, I decided it was best to move the database server to a new server in hopes that I am right that the old server is corrupted somewhere. I don't have the luxury of waiting for tech support to help me resolve this problem eventually nor do I have the time or patience right now with New Years weekend. So hopefully I'm making the right decision here.

It's nerve-wracking trying to get a new server up and running with everything else I've got going on ... hopefully I don't forget something and hopefully this all works out! Fingers and eyes crossed!

If you're seeing this message, it means Seibertron.com is on the new database server. Fingers crossed. I'll check in on everything in a little bit once there's more traffic on the server.

Still a bit slow and Kremzeeky at times, but it's getting there.

Burn wrote:Still a bit slow and Kremzeeky at times, but it's getting there.

Thanks for the feedback, Burn.

I received a few other reports of this and saw it myself as well while running my script to update the seibertron store ads in the right column.

I made a few additional changes to the database server and purged the forum cache just now.

Please keep me posted what you guys are experiencing.

Ryan

The verification Captcha on the Submit News page is broken.

Sabrblade wrote:The verification Captcha on the Submit News page is broken.

Thanks for letting me know, Sabrblade. This has been fixed.

Sorry the site was down again for a couple of hours this afternoon. After the news crew ran into some difficulties getting a news post to show up on the homepage, I realized that I hadn't updated the forums database connection to match the new server. The rest of the site was using the new database, but the forums and the staff admin section both use different code to connect. I haven't messed with those settings in at least 6 or 7 years, so I had forgotten that I needed to make the changes there as well. Hopefully that's why we kept getting Kremzeeks on some parts of the site (because some of it was still using the old database).

Long story short. I made the updates to the database configuration in both of those areas of Seibertron.com. However, I didn't want to lose anything that was updated on the forums in the past couple of days, so I had to take the site down again in order to restore the forums on the new database server.

In order to avoid this problem going forward, I renamed the site's database on the old server so that it would be more obvious if there was an error somewhere on the site. If there's anywhere else on Seibertron.com that is connecting to the old server, it'll fail now because the database was renamed.

I also made an update to a very resource heavy database query that was in the forum code. On every page load, the code would check to see how many user sessions there were in past 15 minutes. The table was accessed so much that it would cause a bottleneck on the server at times when there's lots of activity. I modified the site code so that a script that runs in the background once per minute now stores that information in the database and now the forums just call that one field instead of counting the sessions on every forum page load. Hopefully this helps tremendously by putting less demand on the database server.

Kremzeek struck last night, but I think it was an issue with the main web server, instead of the database server this time. The error was actually because the database server was overloaded, however, because the site has been running so well this past 5 or 6 days, I've been uncovering some other areas on the server that needed to be addressed.

This time I uncovered that the system that is in place on the web server that automatically prevents brute-force attacks. I feel like this is what was really the cause of the problems. Seibertron.com just kept getting overloaded so quickly, that I feel like it was from bruce-force attacks.

I found some errors that were preventing this system from working, made the necessary fixes, restarted the server and it's back to blocking bad IP addresses and Seibertron.com seems to be doing fine this morning.

I'm getting 503 errors when I try to disapprove spam posts.

Emerje

Emerje wrote:I'm getting 503 errors when I try to disapprove spam posts.

Emerje

Can you post the link to the page you are on? The URL will require appropriate permissions to access so it's fine if you share the link for the purpose of debugging in this topic.

I'm assuming that 503 is a separate issue or an error in code that is causing the 503 you are getting and is actually unrelated to the others.

Sure thing:

Post: casino-color-psychology-t120486.php
Error: mcp.php?i=queue&p=2174909&f=4&confirm_key=2ZBH8AEKLI

Looks like the second link is just redirecting back to the mod panel.

Emerje

Emerje wrote:Sure thing:

Post: casino-color-psychology-t120486.php
Error: mcp.php?i=queue&p=2174909&f=4&confirm_key=2ZBH8AEKLI

Looks like the second link is just redirecting back to the mod panel.

Emerje

Interesting. I am able to replicate the problem. I am currently unable to resolve the issue. I'll have to keep working on it over next couple of days. It's a different problem from the typical issue that causes the Kremzeeks. It could be an error in code somewhere or a special character in the user's post or something that is breaking the phpBB code. I'm not certain at the moment but will continue working on it.

Emerje wrote:Sure thing:

Post: casino-color-psychology-t120486.php
Error: mcp.php?i=queue&p=2174909&f=4&confirm_key=2ZBH8AEKLI

Looks like the second link is just redirecting back to the mod panel.

Emerje

Interesting. I am able to replicate the problem. I am currently unable to resolve the issue. I'll have to keep working on it over next couple of days. It's a different problem from the typical issue that causes the Kremzeeks. It could be an error in code somewhere or a special character in the user's post or something that is breaking the phpBB code. I'm not certain at the moment but will continue working on it.

Deleting the user and their posts should fix it.

Burn wrote:Deleting the user and their posts should fix it.

The problem was that we were unable to delete the user or the posts. None of the administrative tools were working.

After doing some code tracing throughout phpBB, I isolated the problem. Somehow one of the phpbb reporting database tables didn't make it to the new server. I restored the table from one of early last week's backups and all is working.

Now I have to figure out how to restore First-Aid's user account which I accidentally deleted when I was cleaning up some spam bots in General Discussion. Always something!

The site is being a bit slow today, with some pages either taking much longer to load than normal, while others just don't load and instead lead to "502 Bad Gateway" warnings, forcing me to reload them.

No Kreemzeeking, at least, so it isn't that.

EDIT: I literally just got a "502 Bad Gateway" warning while trying to make this very post.

Sabrblade wrote:The site is being a bit slow today, with some pages either taking much longer to load than normal, while others just don't load and instead lead to "502 Bad Gateway" warnings, forcing me to reload them.

No Kreemzeeking, at least, so it isn't that.

EDIT: I literally just got a "502 Bad Gateway" warning while trying to make this very post.

I'm getting lots of slowness and 502 Bad Gateway errors too. I also got a 404 error every time I tried using the main and "First unread" links to go to this post; I had to use the direct link instead. So it seems like the "First unread" thing is acting up as well.

EDIT: And the site actually did briefly Kremzeek on me yesterday.

Just had some major Kremzeeking on Page 2 of the new Generations Prime thread, when trying to either quote another user's post, make a new post of my own, or even just trying to load the page.

And only on my laptop, too. It got so bad I had to switch to my phone in order to confirm if my post went through or not (it did, thankfully, despite the Kremzeeking).

Sabrblade wrote:Just had some major Kremzeeking on Page 2 of the new Generations Prime thread, when trying to either quote another user's post, make a new post of my own, or even just trying to load the page.

And only on my laptop, too. It got so bad I had to switch to my phone in order to confirm if my post went through or not (it did, thankfully, despite the Kremzeeking).

Which post specifically were you trying to quote? Maybe the post had a special character in it that the server didn't like. Everything seems good on my end with the server at the moment.

Seibertron wrote:

Sabrblade wrote:Just had some major Kremzeeking on Page 2 of the new Generations Prime thread, when trying to either quote another user's post, make a new post of my own, or even just trying to load the page.

And only on my laptop, too. It got so bad I had to switch to my phone in order to confirm if my post went through or not (it did, thankfully, despite the Kremzeeking).

Which post specifically were you trying to quote? Maybe the post had a special character in it that the server didn't like. Everything seems good on my end with the server at the moment.

This one.

Thanks for letting me know. Looks like the DB server is running a very high load avergage. I cleared out a few database tables and rebooted the DB server. Monitoring it. Load average went down from 48 to 24 but still very high.

Seibertron wrote:Thanks for letting me know. Looks like the DB server is running a very high load avergage. I cleared out a few database tables and rebooted the DB server. Monitoring it. Load average went down from 48 to 24 but still very high.

Yeah, yesterday evening, there was a period where I couldn't even go to the site at all for at least an hour.

Just got a 503 trying to post in the TF:One topic, couldn't load the rest of the site either.

Emerje

Got to work this morning and it seems like it's still experiencing some problems. I'm running some maintenance commands right now and making a few adjustments to server settings at the moment.

Site was down for past 30 minutes while it optimized the massive forum posts database table.