Shawn

http://www.macrossworld.com/mwf/index.php?showtopic=36284 Shawn

http://www.macrossworld.com/mwf/index.php?showtopic=36284 Shawn

http://www.macrossworld.com/mwf/index.php?showtopic=36284 Shawn

http://www.macrossworld.com/mwf/index.php?showtopic=36284 Shawn

Please be patient for the next week or so. We did the forums upgrade, and of course it blasted out the old 'skin', so we are using the default at the moment. Efforts are underway to get us back to something custom again. And like every single upgrade in the past, we had some mysql issues that we had to overcome. But all core functionality SHOULD be intact right now. FYI, next weekend (tentatively-give or take a couple days depending on my schedule) there will be another outage for a day or two as we do some hardware changes on the server. More info as it gets closer... Thanks for your patience throughout this latest upgrade!! Shawn

I am very,very glad to hear our members in Japan are ok!! I can't believe the videos they are showing on the news, the sheer power of the waves is mind-bending. Matt Alt from Toybox DX was on the US news this morning reporting on the status, so at least we know he is ok too. I'm sure there will be some donation efforts soon, and we will participate. Just wish there was something more we could do help! Everyone stay safe there!!!! Our prayers are with you. Shawn

I'll check the drives (last time we had some errors), and run some checks on the db this weekend. Again, I think it just boils down to the limitations of the ISAM format we are running. Locking entire tables for adding a post, when there are a lot of users doing the same, is probably leading to contention issues-and then everyone just times out. I just have to sit down and convert us to INNODB and be done with it all once and for all. I've let it carry on far too long. Time to clear out a weekend schedule this month... Shawn

Never seen that particular Orange Collection Elint box-very very rare!! Does it say which event it was sold at? Shawn

Finally had an outage for me right now Hit the server and saw it was swapping out A LOT of memory to disk-this might be the problem Shawn

I know its cliché and everything, but did you delete your Macross World cookie? Sounds like a corrupt cookie issue. Shawn

Passed the hardware tests. I did a mod in one of the cron jobs-will see if that helps. Please let me know times and how long any future outages occur. I have an open ticket with our hosting company. Thanks Shawn

Check it out Excerpts from recent outages Hurin: 07/22/2009 1:14pm Just had the outage myself from ~noon to 12:45 pacific. It seems to be almost certainly server-side. And judging by the behavior, it probably has something to do with php or mysql. I'm making that guess because the problem actually started small and then continued to ramp up until the entire IPB forum was inaccessible (just timed out, wheel spinning). Edit: Down again at 1:30pm. I also meant to mention that apache remains up and working throughout these outages as the main site (non-forum, so no php or mysql) remains available. Mr March: 07/31/2009 1:29pm The forums slowed and then I had an outage. However, there was a difference. The forums tried to load and after about a minute, Firefox came up with a "Connection Timed Out" error. Tried to load the MW forums again about a minute later, same error. That never happened before; before, the browser would just hang indefinitely until I lost patience and closed the window. Now I get errors messages after a minute. So something may have changed. Cent: 08/09/2009 02:10pm About an hour ago, I started to see a similar outage crop up. It went away surprisingly quickly though. About 20 minutes later. And the server bandwidth log below: During Hurins time, the server was producing 0 output. Same for Mr. March's outage. The opposite happened for Cent's-we were spiking. Going to run some hardware checks tonight (disk checks and such) Shawn

ok-i thought of something else I'm going to TEMPORARILY turn off the gallery tomorrow night, and leave it down until friday. Thats is an easy way to see if this could be server load related. S

I honestly am stumped. Sql logs show no issues other than the one error (that has no resolution I could find). Server loads appears fine. I did not try changing the DB driver, but that would be the next test. I was hoping the adjustments in load would help. I reeeeeeally do not want to prune. But we might consider a test. We could do a backup, prune the hell out of the forums, then live in wild-west land for a couple days knowing anything you posted would probably be lost. If the outages stopped, we know we are at the limit of the server and software somehow If the outages continued, we'd restore, lose those couple days of posts, and try something else (what, I have no clue) If no one has any ideas, this weekend is db driver change. I'll shut down sometime saturday morning and shift things over. http://forums.invisionpower.com/topic/1625...n-mysql-tables/ http://forums.techarena.in/software-development/1160112.htm S

Turning off the extra features doesn't seem to have made a difference if we still get the outage. again-you can still connect to the site, but it will say 'ips error' or something right? Or do you get nothing when going to the site (timeout 404 etc) S

wow-very observant!! was trying to turn off some extra features for a couple days to see if it improves anything. They'll be back shortly once we determine the root cause S

Here are the specific sql errors from a little while ago. Need to research what they mean. Date: Thu, 30 Jul 2009 16:49:28 -0700 Error Number: 2013 Error: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 95 IP Address: xx.xx.xx.xx Page: /mwf/index.php?showtopic=16227&st=240 =================================================== Date: Thu, 30 Jul 2009 17:21:39 -0700 Error Number: 2013 Error: Lost connection to MySQL server at 'sending authentication information', system error: 32 IP Address: xx.xx.xx.xx Page: /mwf/index.php?showtopic=21004&view=findpost&p=580850

hmmm... The forum DB is a little over 1GB right now, with 14GB of picture uploads. I'm wondering if some pruning is in order. There are much much larger forums than ours, wonder what the issue is. I hate to prune, because then we lose the old threads that inevitably someone asks about later on. Same reason the old forums are still asked about (and still offline) Or perhaps we need to change the MySql storage engine from MYISAM to INNODB. Perhaps the full table locks of MYISAM are causing issues across these large tables vs row level locking that INNODB provides. I think this weekend we'll flip the switch and see if it helps at all.

Ok-ran the repair options on the db and it did find some issues. Can you see if you still experience the errors? thanks! Shawn

ok-thanks, that helps a lot. i've been looking at http issues, but i think it is mysql i'll repair the db tonight-maybe tomorrow we'll see these issues go away! Will let you know when I start (well...it'll just look like another outage) but more importantly when I'm done maybe 2-3 hours max before I start. S

Can I get a definitive answer on WHAT you are experiencing when you say outage? I've been looking through the logs these last 2 weeks and do see spikes in traffic-but its brute force attack type of traffic-a lot heavier than usual. There have been some worldwide DOS attacks and I figured we somehow became one of the thousands of targets. Are you seeing the macrossworld.com main page go down (http error) or a ips board issue(sql error) this will help me immensely in figuring out what is broken thanks! Shawn

Restarted the server about an hour ago please let me know if the outages still continue! shawn

I'll restart the server tonight and see if that helps. Its been up a while and perhaps something has a memory leak or ??? Sorry! Shawn

Roger can't post for some reason, but he asked I offer his story to this entire mess. http://docs.google.com/Doc?id=df9xwj94_69hbzgvddb Shawn

I know once something is on the net its out of the box, but I certainly don't want encourage this and wont. I'm asking you please do not post those images on my site, now or ever. A lot of effort was put into getting those, and I will not be a part of this projects unraveling. Its a real shame it came to this. I was really looking forward to seeing these pictures get the credit they deserve and not just end up on a couple cheap pages in photobucket to be forgotten. Shawn