Annnnddd... we're back...

General information about upcoming maintenance, and other events which affect the forum.
Ciaran
Site Admin
Posts: 1824
Joined: Tue Apr 14, 2009 9:59 pm
Orga / RP numbers: 1234
5678
91011
Location: Ireland

Annnnddd... we're back...

Post by Ciaran » Mon Dec 07, 2009 4:06 pm

Christ... well that was a balls up and a half...

Sorry for so much downtime guys, what started out as a minor issue snowballed, about this time last week, and by Wednesday had become four or five seperate problems with the server / software. Not all of which were immediately apparent either, I had to do some digging and got quite a few red herrings, which meant the real problems were disguised and I was off troubleshooting non-existant issues.

It initially started out as a driver problem, crashing the machine about once a day, so I thoroughly updated them all, flashed a new BIOS onto the motherboard etc. Then it developed memory issues (or so I thought) so I replaced the RAM. It then exhibited strange behaviour and disk activity, would reboot for no reason, differing error messages each time, leading to a few WTF moments. At this point it was still serving out the site pages, though it was down more often than up. I finally traced the cause of this to be a very nasty viral infection, a 'rootkit', which had infected the OS at a low level using a 'code injection' method on some Windows system files, and was causing the instability. Luckily, upon identifying and removing it, I'm satisfied that no site data was compromised, as this thing was mainly concerned with peddling spam and desktop shite than anything else, thankfully. How it got onto the machine in the first place remains a mystery, because it isn't in contact with anything on my internal network, or directly to the web, other than serving out the forum pages. The possibility remains that it was some kind of vulneribility in the webserver which allowed it on, in light of that, the OS and webserver have been updated and patched to the latest versions available.

After that was sorted, the machine then began experiencing intermittent network disconnections, which I initially thought was a dodgy LAN card, so that, and the cable were changed. That was dandy for a whole half day or so, then it started again, this time looking like software problems on the machine. After I wasted a day chasing down that one, I discovered that it was actually dodgy firmware on the router, which was failing to route packets in from the internet to the webserver. Just as I resolved that, and thought we were all good to go again (was about Friday / Saturday night by this point), the motherboard in the server died without warning. It was running, I shut it down to change a DVD drive, and it never booted again. I subsequently discovered that a few capicitors on the motherboard had swollen and one or two had actually burst and were leaking fluid. Presumably this had been happening for some time and caused its inevitible demise.

At this point I was loosing the plot and just ripped out what salvagable components were left in the machine, stuffed them into a new motherboard and case, and installed a fresh OS, copying over the site data.
That takes us up to this morning, where the site was working briefly then suddenly stopped. Finally, after spending today troubleshooting weird PHPO permission issues, its finally working properly!

Lesson learned from this, in addition to the regular backups of the database and site itself, I'm planning to keep a redundant machine that can be booted up and kicked into action should this ever happen again in future...hopefully not!
We now do also have a very kind offer of termpory hosting from Jim on FCF, so there's that to fall back on, too, should the very worst ever happen.

Sorry for the extended messing about, it really was an utter, utter bollox!

Ciarán

SteveLP
Knows how to use the parking brake
Posts: 220
Joined: Sun Apr 26, 2009 12:51 pm
Location: Bristol

Re: Annnnddd... we're back...

Post by SteveLP » Mon Dec 07, 2009 4:13 pm

Well done. I know that we were all suffering total withdrawl symptoms :P .
Thanks for all you hard work in keeping the forum alive, we all appreciate it!

I've also spent all day wondering why my HP server is locking up randomly...
Cheers
Steve
3.0 V6 24V W reg: my new toy!!

Peter.N.
Global Moderator
Posts: 3999
Joined: Wed Apr 15, 2009 8:29 am
Location: Charmouth, Dorset

Re: Annnnddd... we're back...

Post by Peter.N. » Mon Dec 07, 2009 4:23 pm

Well done Ciaran, can the men in white coats go home now?
'96 'N' 2.1 td VSX manual estate White RP6695. Sadly gone
2008 C5 2.0. Hdi Estate, Red
2008 C5 2.0. Hdi Estate Silver


Located in Charmouth, Dorset

kenhall
Citroen go-to-guy
Posts: 845
Joined: Sat Apr 18, 2009 6:53 am
Location: West Cumbria

Re: Annnnddd... we're back...

Post by kenhall » Mon Dec 07, 2009 5:32 pm

You deserve a medal Ciarán - thanks for all your hard work keeping the forum afloat!

Ken

xmexclusive
Global Moderator
Posts: 5925
Joined: Wed Apr 15, 2009 8:11 am

Re: Annnnddd... we're back...

Post by xmexclusive » Mon Dec 07, 2009 5:36 pm

Ciaran

Thanks very much.

John

User avatar
colinxm
Has changed a sphere or two
Posts: 299
Joined: Wed Apr 15, 2009 4:13 pm
Location: Jørpeland Norway

Re: Annnnddd... we're back...

Post by colinxm » Mon Dec 07, 2009 5:41 pm

Thanks Ciarán, I've really missed this place :D

Question is, can we help out with funds for replacement/better hardware ? name your price ;)

Colin.P
1995 Cit XM 2,1 TD Silver Estate - deceased :-(
2004 Opel Vectra Estate
2008 Kia Ceed td
1989 Pug 205 Cabriolet
1998 Ford Puma
1979 Triumph TR7

User avatar
Dean
Global Moderator
Posts: 6116
Joined: Wed Apr 15, 2009 7:53 am
Location: Isle of wight

Re: Annnnddd... we're back...

Post by Dean » Mon Dec 07, 2009 6:02 pm

Its all been said....Thanks Ciaran its much appreciated ;)

D
92 Citroen XM Prestige 3.0i Auto R.P5678
14 Mitsubishi L200 Trojan
89 Talbot Express 2.0 coach built Auto-trail Chinook

Addicted to Crackanory

User avatar
Kit
Knows how to use the parking brake
Posts: 154
Joined: Mon Nov 02, 2009 7:50 pm

Re: Annnnddd... we're back...

Post by Kit » Mon Dec 07, 2009 6:02 pm

And a heartfelt thankyou from me as well, We are so lucky to have Ciarán. As a newby on this forum I am loath to stick my neck out, but what the Hell, it's too late to change my habits now. This hiatus does flag up how nice it would be to put the forum on a more secure footing, I too would be willing to contribute financially. Perhaps our founding fathers may have a cunning plan already? Kit
'95 2.1td manual VSX Estate Silver rp6791
'95 2.1td manual VSX Estate Green rp6769 sorn

bitonthesidexm
Knows how to use the parking brake
Posts: 109
Joined: Wed Apr 15, 2009 8:51 am
Location: Ellesmere Port

Re: Annnnddd... we're back...

Post by bitonthesidexm » Mon Dec 07, 2009 7:19 pm

Well done Ciaran - you say the whole thing was developing memory issues and you replaced the RAM. I am developing memory issues - could you not insert any RAM into me please? I thank you!
1994 XM Turbo Onyx in black.

User avatar
ash2020
Knows how to use the parking brake
Posts: 218
Joined: Tue Oct 20, 2009 9:14 am
Location: Mere, Wiltshire

Re: Annnnddd... we're back...

Post by ash2020 » Mon Dec 07, 2009 7:55 pm

Big round of applause for Ciaran. It's true that you don't miss it 'til its gone! Well done.

BTW, would it help if I Flashed a new BIOS on my XM segment display?

Cheers
Andrew
(97) XM 2.5TD VSX Green

Post Reply