Old kernels won't pop as easily as new kernels. My grandma always saved them anyway and microwaved them again in a paper bag.

(01-21-2013 01:46 AM)georgia_tech_swagger Wrote:  Yesterday it dropped out around 1:30 AM ET due to a power failure in the UPS on the rack. I'm waiting on the datacenter to rectify that situation. It was down for about an hour -- which is how long it took my wasted ass to wake up to all the phone alerts and re-enable a few services. By default if the server crashes uncleanly, the database will not come back live until I've manually inspected it to make sure there is no corruption. Trying to run this site on a corrupted database would create quite a horror show.

Tonight's downtime was for an enormous leap forward in kernel versions on all servers, and to enable some PHP features for some of my partners in crime in the Skunkworks team.

I'll let it sit as is this week to make sure there are no performance regressions, then next weekend I'll take it down for a few hours again to go to the latest kernels, install some special patchsets on those kernels, and bring userspace up to date to match.

Good, I thought it was because I tried to upload an uncompressed BlueRay rip of We are Marshall to my signature.

(01-21-2013 07:22 PM)georgia_tech_swagger Wrote:  

(01-21-2013 07:04 PM)Luckyshot Wrote:  

(01-21-2013 12:51 PM)georgia_tech_swagger Wrote:  Best. Upgrade. Ever.

Left: Old kernel, less traffic. Right: New kernel, more traffic.

Yeah, but there is more blood dripping down on the second graph. That's gotta be worrisome, right?

Nope. The red is CPU waiting on stuff to come in from disk. The huge spikes are from routine flushing of the database replication logs and the big one is from coming back after reboot. For about 5 minutes after initial startup, the database is pretty much going "I NEED EVERYTHING FROM DISK YESTERDAY!!!!!!11". Until we cache out the database into RAM it crawls.

Next time I'll put "j/k" somewhere in my post . . .

(01-21-2013 11:08 PM)TheEastisPurple Wrote:  I understand very little of this...

I know that the internet is a series of tubes but past that I'm lost.

(01-21-2013 01:46 AM)georgia_tech_swagger Wrote:  Yesterday it dropped out around 1:30 AM ET due to a power failure in the UPS on the rack. I'm waiting on the datacenter to rectify that situation. It was down for about an hour -- which is how long it took my wasted ass to wake up to all the phone alerts and re-enable a few services. By default if the server crashes uncleanly, the database will not come back live until I've manually inspected it to make sure there is no corruption. Trying to run this site on a corrupted database would create quite a horror show.

Tonight's downtime was for an enormous leap forward in kernel versions on all servers, and to enable some PHP features for some of my partners in crime in the Skunkworks team.

I'll let it sit as is this week to make sure there are no performance regressions, then next weekend I'll take it down for a few hours again to go to the latest kernels, install some special patchsets on those kernels, and bring userspace up to date to match.

Those may be actual words you are typing but all I read is blah blah blah.