Inheritance…..

Since this is my first post I thought that I might share some beginnings of where this blog comes from and the intentions for it.

Around April 2013 I inherited MySQL infrastructure containing several hundred MySQL 5.0 master-slave pairs. Yup, you read right, 2013 and these boxes were all running 5.0. So, before I get to far ahead of myself, lets back up a few years.

The year was 2010 and I was hired in to the Rackspace family as a Linux Systems Operations Administrator II, on 3rd shift non the less, and to all of the night walkers out there, you are loved. So 3rd shift breeds an interesting bunch, and I had worked with several. A very intelligent group of people that inspired me to push myself to new heights and dig my feet in deeper. Well, ~1.5 years later and my feet are nice and wet cutting my teeth on a fairly large and complex beast of a network and I find myself moving to 1st shift, then shortly there after, up the ranks to Ops III. All is going well, but I’m finding out quickly that I am getting rather bored with the day in – day out work of Ops. Low and behold an Engineering requisition opens up for the very product I have been working on for the past ~2 years! So I apply, get hired, and just like that, things start to find that I am back with a group of people inspiring me to do more!

Little did I know that almost 3 months later, I would be inheriting the entire MySQL infrastructure. Around January 2013, our DBA leaves the team. “Why Whatever are we to do!!!” That’t how everyone else remembers this going down. Me? I remember it more like “Fuck it, let the FNG do it!”

So there I am, 3 months green on the engineering team and I get all the technical debt that comes with a MySQL infrastructure that is hundreds of servers deep that has not been tuned in probably 3 years plus (even though hardware has been replaced and loads change near daily). My response to all this you ask? FML! So there I am faced with a decision…. I can 1) run screaming like a little bitch (probably the most sane response) 2) bend over and let the environment rape the will to live out of me 3) make these boxes my bitch and own the shit out of em’. I (perhaps stupidly) chose the 3rd option.

So here we are end of June and thus far I have successfully upgraded 1/3 of the infrastructure to at least a slightly more respectable version of …… 5.1 FML again. Turns out there was a decision before I inherited this mess that we would be going to 5.1, not newer. So, here I am today tasked with finishing off upgrading the other 2/3’s of the infrastructure, tuning the servers to not be steaming piles, as well as performing other Linux-y duties…. Oh and try to keep my sanity.

So, to get to the point, this blog is to document where my time and sanity has gone, and perhaps to save some of you from the same pit holes, or perhaps, for you to save me from my own pit holes.

In short, here is a list of project that I hope to complete in the coming months. By no means is this a complete list, but rather a launching pad.

• Completing the upgrade of the infrastructure to 5.1
• Develop a point-in-time backup procedure
• Go down the path to 5.5
• Improve tuning of the environment to account for new (to me) features/hardware in MySQL and the present workloads
• Develop a plan for no/minimal downtime cross server database migrations

….. well that’s a good start, we will see how its goes.

Let me know what ideas you all would like to see!

Scott