Yes, The server was down today!

[mod 135639]

I was doing routine maintenance on the server and something bad happened and shut the entire site down for a number of hours while we repaired the server. I thought it was just a coincidence that it died just as I was working on stuff but needless to say... it crashed HARD!... and I am not sure yet what exactly it was.

 

It's back up now but I had to remove all my optimization code so the site may be a little slower today than normal. I will work on fixing that later tonight when the server is slow.

[mod 135639]

Ok, I had to add some of the optimization back in as being offline the server is getting hit hard all at once here as everyone is trying to get back in. I am done for now but some of you may still have encountered some database errors while I added the optimization settings back in.

Should be all good now so if you notice anything odd please PM me or post here as a reply.

 

Thanks, and sorry for the offline time today!

[Andee 220524]

I was in withdrawal without my CERB...whew so glad everyone's back!

[jara 476]

Thank goodness. I thought I broke it cause I was on it so much :-)

 

Thanks for the great service!

[A***** A***** 510]

It must have been dreadful trying to deal with that, I would have been freaking out. Did you ever find out what caused it to crash?:shock:

[mod 135639]

I did finally figure out what it was ... I edited some server setting to optimize the site a couple weeks ago and one of the settings I beleive was the reason the server would not reboot properly. We run a very unique web server here and not all "normal" settings work - in fact many settings can "break" the server as many systems are tied into each other in some complex ways.

 

For those who know about MySQL/Postgresql you would know about the setting skip-innodb (Popular optimization setting to invoke when running vbul as you don't need innodb).

 

For those who have never tried it on a server running a control panel (cpanel/plesk/etc...) a word of advise... don't use that setting as it breaks the control panel as the control panels apparently need innodb (Not sure why but that seams to be the case)

 

This was done a while ago but I had not done a full server reboot. Everything worked very well (until today)...

 

Today I went to routinely optimize the tables using the "optimize tables" settings built into the vbul software (You have to do that every couple weeks to keep the site running well) and 1/2 way through the optimization the MySQL server crashed and corrupted a table. (Problem on it's own).

 

The server was due for a reboot so instead of just restarting mysql I took the opportunity to reboot the server and it never came back online fully and the control panel locked up and postgresql would not start.

 

It took us HOURS to figure out what went wrong... I noticed MySQL was not loading as well as the postgresql so it dawned on me that I should just remove the optimization settings (back to default settings) and give it a try (Really I was just guessing) and sure enough the server came back online.

 

I systematically entered the optimization settings back in (Most of them one at a time) rebooting each time until it crashed again (Eventually finding my answer). I than went on google and searched for info to find out that others have had the same problem when using skip-innodb with a similar config like we have.

 

Lesson learned (the hard way as always but these are the lessons that you really do learn from) ... Talk about a stressful day! WHEW!

[antlerman 17064]

wow....that is exactly what I was thinking.....................:confused:

 

in the end.....Mod is the man!!!!!

[Mutau 2516]

I agree with Antlerman. Not only is the Mod's explanation beyond my limited technical capacities :confused: (although I quite fully understand that fellow CERBites who themselves are techies, or so inclined, will have loved his detailed explanation) but it only proves how well we are served by his mastery of all things technical in keeping our Board running smoothly, with only the odd little hiccup.

 

Well done Mod! Now relax and have a glass of champagne to celebrate a job well done!

[whatsup 11893]

Appreciated for sure, Thanks Mod

[Mikeyboy 27131]

I tried to log in at the time, and had a minor panic attack when the site wouldn't load. What if the site's gone for good? (Irrational I know but you never know). It made me realize how much I would miss this site if it weren't here. Good work mod. Keep it up. 8-)

[mod 135639]

Years ago (early to late 90's) when I started managing servers I would fall into a huge panic myself.... Some of the really huge companies now have so much redundancy it's insane... and they still go offline on occasion. I know a guy who's job is to make sure a bank server stays online at all costs... it has 12 backup drives for every one running drive! What are the chances of even 2 going down let alone 12!!

 

Some of our members are techies like me... sorry for the over the top explanation. Angela asked the question and she has expressed interest in the past on the technical info of things (I beleive it is Angela who is learning to do some computer programing in a obscure language and I would encourage anyone with a interest to do just that).

 

If it makes your hear hurt reading it... it's not for you (Just skip it and replace with Bla Bla Bla....)

[A***** A***** 510]

Years ago (early to late 90's) when I started managing servers I would fall into a huge panic myself.... Some of the really huge companies now have so much redundancy it's insane... and they still go offline on occasion. I know a guy who's job is to make sure a bank server stays online at all costs... it has 12 backup drives for every one running drive! What are the chances of even 2 going down let alone 12!!

 

Some of our members are techies like me... sorry for the over the top explanation. Angela asked the question and she has expressed interest in the past on the technical info of things (I beleive it is Angela who is learning to do some computer programing in a obscure language and I would encourage anyone with a interest to do just that).

 

If it makes your hear hurt reading it... it's not for you (Just skip it and replace with Bla Bla Bla....)

 

LOL @ obscure language

 

This is hardly that... http://www.alice.org

 

It's a noob 3D based on Java

 

They teach this to kids in school:lol:

 

Programming is brain draining, literally. Between functions and methods I would have one huge head ache.

 

I err towards the hardware, software and technical support. I have done Primus VOIP, Telus Dial Up, and basic hardware software.

[Meg O'Ryan 266444]

I just wanna give a shout out to MOD....like Angela, I went thru withdrawals. It happened right as I was posting my rant. When it didn't work, I had to try and remember what I said so that I could type it into notepad and paste in a thread when cerb was back up. God forbid we lose this wonderful forum! LONG LIVE CERB and LONG LIVE CERBITES!

[mod 135639]

LOL @ obscure language

 

This is hardly that... http://www.alice.org

 

That was meant as an encouraging comment (not a dis or anything) I appreciate that you are learning something that is not mainstream. You may just end up being on the cutting edge as these "less popular" programing languages often spring up and take over. Right now when you look at the job market for programmers it's all php/java/ajaz/ruby/.net and C programmers for the most part. Pretty hard to make a living right now with alice but that could change if it catches on more.

 

I learned COBOL back in the 80's (Worked for the government doing COBOL conversions off a unix mainframe to a IBM server and oracle DB).

 

I learned turbo pascal, fox pro, basic and a little C as well. C was mainstream and they had started teaching it in university... but most schools were teaching pascal, basic, cobol and foxpro and only cobol and C were really mainstream and COBOL was pretty much at the end of it's life.

 

The only reason I leaned COBOL was to help move the date from the main frame tape drives to the new digital DB as COBOL was on the way out the door along with fortran when I learned them. (Punch cards were before my time in case you are wondering)

 

I learned CGI/perl because of the internet and these days PHP is the most popular language around at least for web based stuff. C#, .net and some other stuff are really mainstream as well and of course java/ajax are really popular as well.

 

Stuff like CF and ASP and even perl are falling away now to make room for the once obscure stuff like Java/ajax, ruby on rails, and even the .net framework stuff from microsoft .. so picking the newer (not currently mainstream stuff) is a positive in my books as it could be tomorrows PHP! It may not make you lots of money (Most of the stuff they teach in schools don't - you need to branch out on your own or land a job willing to pay for specialized training)

 

If it does take off your a head of the game and I would rather be in a small group of specialized programmers and not trying to compete with a million other php programmers all outbidding each other. I never really bothered to learn php (I know just enough to hack this site as it runs on php) but I am not proficient with it... My reason for this is I can pay someone 30.00 per hour to do the work for us who is a php god and I would not work for 30.00 an hour doing programming work so it's just not feasible for me to spend the time to learn it.

[A***** A***** 510]

That was meant as an encouraging comment (not a dis or anything) I appreciate that you are learning something that is not mainstream. You may just end up being on the cutting edge as these "less popular" programing languages often spring up and take over. Right now when you look at the job market for programmers it's all php/java/ajaz/ruby/.net and C programmers for the most part. Pretty hard to make a living right now with alice but that could change if it catches on more.

 

I learned COBOL back in the 80's (Worked for the government doing COBOL conversions off a unix mainframe to a IBM server and oracle DB).

 

I learned turbo pascal, fox pro, basic and a little C as well. C was mainstream and they had started teaching it in university... but most schools were teaching pascal, basic, cobol and foxpro and only cobol and C were really mainstream and COBOL was pretty much at the end of it's life.

 

The only reason I leaned COBOL was to help move the date from the main frame tape drives to the new digital DB as COBOL was on the way out the door along with fortran when I learned them. (Punch cards were before my time in case you are wondering)

 

I learned CGI/perl because of the internet and these days PHP is the most popular language around at least for web based stuff. C#, .net and some other stuff are really mainstream as well and of course java/ajax are really popular as well.

 

Stuff like CF and ASP and even perl are falling away now to make room for the once obscure stuff like Java/ajax, ruby on rails, and even the .net framework stuff from microsoft .. so picking the newer (not currently mainstream stuff) is a positive in my books as it could be tomorrows PHP! It may not make you lots of money (Most of the stuff they teach in schools don't - you need to branch out on your own or land a job willing to pay for specialized training)

 

If it does take off your a head of the game and I would rather be in a small group of specialized programmers and not trying to compete with a million other php programmers all outbidding each other. I never really bothered to learn php (I know just enough to hack this site as it runs on php) but I am not proficient with it... My reason for this is I can pay someone 30.00 per hour to do the work for us who is a php god and I would not work for 30.00 an hour doing programming work so it's just not feasible for me to spend the time to learn it.

 

NO dis taken at all, I thought it was a funny comment8)

 

I might just have to get back into this programming some day.

 

http://www.avs34.com/portal/ is dual credit with UFV. Get a highschool with credits towards a degree sounds interesting.