Trashed libc - getting out of the doodoo

By Veghead

After spending the best part of 5 hours trying to patch a Cobalt RaQ 4, I thought it might be an idea to document what I had to do in case someone, somewhere ever finds themselves in a similar position. Yet again, for the second time this week, it was essentially the fault of rpms. I hate rpms. They're so very nearly good, but as the cliche goes, a miss is as good as a mile.

Obediently obeying the patch order, I applied a few patches to the RaQ using its web interface. One patch failed to apply and of course it was probably most temperamental patch of the lot (aside from the kernel) - glibc. How many times has redhat managed to ruin my day with glibc rpms ?

Unbeknownst to me, the root partition was at about 99% capacity. Unlike Solaris patches, rpms don't give you a warning if space is tight - they just go ahead and fail. I was left with no libc on the system at all and thus neck deep in cack.

As you will know, the only way to rectify this is with physical access to the machine and a CD. As this is co-located god knows where and it's sunday this was going to be a major hassle so I decided instead to repeatedly scream a selection of swear words and generally terrify the parrot.

However, by some miracle I had a root shell onto the box - but this wasn't much use as even the most basic of commands were broken - ever tried to navigate a n*x box without ls ? The other thing that seemed to be running fine was apache/php.

To cut a long story short, I managed to get the box back in gear remotely, but it was a MAJOR hassle. If you ever find yourself in this position then I hope you find this useful. Here's what I did:

Use "echo *" and tab completion to see the contents of directories. Navigate to a webroot that allows you to put PHP scripts in place.
Create an html file upload form, and a php script to move the file somewhere useful (eg tmp). eg

 echo "<?php unlink(\"/tmp/cat\");
	print(move_uploaded_file(\$HTTP_POST_FILES['fred']['tmp_name'],\"/tmp/cat\"));
	chmod('/tmp/cat',777);?>">j.php

Upload some statically linked tools. I built a copy of 'cat' and uploaded that.
See if you can get hold of a copy of libc for the machine. In the case of RaQs, the .pkg files are actually gzipped tarballs containing rpms. rpm2cpio or rpm2tar should let you pull out the libc binary.
Upload it to the box using the php page you created.
If libc is too big to be uploaded in one go (as it was in my case) split it into manageable chunks with "split" and then upload them one by one. You can then use your static 'cat' to cat them together.
Then see if you can use your libc.

export LD_PRELOAD=/tmp/libc.so.6
ls

If it worked, then then do a little dance and give the parrot some nice seeds. Oh yes it might also be an idea to move libc into a sensible place (like /lib) and try to fix the broken patch that caused this problem in the first place.

<< Back