Quite a harsh week really. From the slew of tedious requests all week, we also had a nasty incident on wednesday that involved our two main servers being brought to their knees. As luck would have it Geoff and I managed to sort the problems out by resetting the RAID array without needing to reboot either server. A risk that paid off . Nonetheless we were shaken enough to call in an engineer during our scheduled downtime on friday.
This is where it becomes an interesting example of why outsourcing, and therefore privatisation, sucks.
The engineer arrives half an hour after we take the systems down and it is clear that he has never seen this particular device before. I don’t have a problem with this – he comes from some whore of a company that supports a million devices – but it makes me wonder what we’re paying for. He calls the guy who knows the answer s, and obeys his instructions. I COULD DO THAT! I can follow instructions…in fact I could probably follow them better than this guy, nice tho’ he was….
We ended up in a situation where the engineer may have fucked up our configuration, on the instructions of someone else. To the uninitiated that means we loose 2TB (2048 GB) of data and have to restore off a tape. A time consuming, and unsatisfactory solution. This engineer, helpless, was just sitting there waiting for a phone call from his engineers…who in turn were waiting for a call from the real engineers. I’d had two coffees and a curry the night before…I was not into pacing, waiting for a call, so, with Geoff’s agreement, I went for a long walk.
When we get Sun engineers in, they know the kit. When you call a Sun engineer out, he know the kit nd will get it fixed. When Peter G worked for Storcomp we felt content that his almost bizarre level of professional integrity would get the problem fixed PDQ. But no longer. How sad.
To cut a long story short he hand’t fucked things up and we were OK. There were some unresolved problems but we managed to “fix” them, without any of us understanding why it worked at long last. This is the first time I’ve felt scared about the technology. When the guy from the company who sold us the device called me, it was clear that he understood the technical problems I was talking about which gave me some hope. That alone gave me enough confidence to be able to sleep that night.
Ultimately Geoff and I managed to get it all back up and working, and with an extra chunk of space on all of the servers, including homepages.
The rest of friday was relaxing by comparison. Sadly this resulted in yet another night of no memories and falling asleep on the sofa…oh dear.