Announcement

Collapse
No announcement yet.

Recovering from a rm -rf /

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Recovering from a rm -rf /

    Best. Stack Exchange. Post. Ever.

    http://serverfault.com/questions/769...g-from-a-rm-rf

    I run a small hosting provider with more or less 1535 customers and I use Ansible to automate some operations to be run on all servers. Last night I accidentally ran, on all servers, a Bash script with a rm -rf {foo}/{bar} with those variables undefined due to a bug in the code above this line.

    All servers got deleted and the offsite backups too because the remote storage was mounted just before by the same script (that is a backup maintenance script).

    How I can recover from a rm -rf / now in a timely manner?

  • #2
    Does that also remove the dole queue??
    I've not failed. I've just found 10.000 ways that don't work!
    Dave Burnett

    Comment


    • #3
      1) insert bootable media
      2) start installation
      3) obtain copious amounts of caffeine, it's going to be a long night
      --
      Fractal R4 (shhh!) 3570k, z77x-d3h, 16gb, R9-270X, 240gb SSD, 2tb mirror w/ 60gb cache drive

      Comment


      • #4
        It's the 1535 customers sites that I worry for

        Comment


        • #5
          Surely a troll?

          Comment


          • #6
            It's hard to tell sometimes. Some of the comments made sound like an obvious troll (eg him getting the 'if' and 'of' flags wrong when creating a dd clone). But then he did mention that they (not him specifically, but the company generally) did recover the data and he thanked people for their help - which is very untroll like.

            My gut feeling is that was just an office junior - young, over confident and prone to making silly mistakes - who panicked and tried to fix things himself before he told his supervisors.

            Comment


            • #7
              Thread doesn't exist any more..



              Originally posted by coiler
              Stomach was rumbling like a fatman landing on Sanctuary

              Comment


              • #8
                Originally posted by Aaron View Post
                Thread doesn't exist any more..
                Ah pity. There's an Independant article on it: http://www.independent.co.uk/life-st...-a6984256.html

                edit: seems his account has been deleted from Stack Exchange as well. I'm now thinking Andyn was right.
                Last edited by cold fusion; 15-04-16, 14:40.

                Comment


                • #9
                  Surely this rang enough alarm bells of it being a troll account though?

                  {foo}/{bar}
                  Or is that a real thing?!



                  Originally posted by coiler
                  Stomach was rumbling like a fatman landing on Sanctuary

                  Comment


                  • #10
                    He was just using that to indicate some variables which should have been part of the path structure, $foo and $bar are widely used in programming as 'example variables'.

                    So his code might have looked like

                    $system_command = '/bin/rm -rf ' . $foo . '/' . $bar;

                    And then he managed to run this with empty values for $foo and $bar somehow.

                    But to say that's a bit of a noob error would be 'generous' to say the least. Also, although the article is gone I got the impression that commands were being issued through some platform for multiple server management. You would think that any such platform would have mechanisms in place to filter out 'suicide' commands really, although I don't remember the details well enough to be able to confirm that. Whole thing is a bit fishy tbh.

                    Comment


                    • #11
                      Originally posted by andyn View Post
                      He was just using that to indicate some variables which should have been part of the path structure, $foo and $bar are widely used in programming as 'example variables'.

                      So his code might have looked like

                      $system_command = '/bin/rm -rf ' . $foo . '/' . $bar;

                      And then he managed to run this with empty values for $foo and $bar somehow.

                      But to say that's a bit of a noob error would be 'generous' to say the least. Also, although the article is gone I got the impression that commands were being issued through some platform for multiple server management. You would think that any such platform would have mechanisms in place to filter out 'suicide' commands really, although I don't remember the details well enough to be able to confirm that. Whole thing is a bit fishy tbh.
                      It's very easy to accidentally drop in empty variables if you don't test your code properly. Both Valve (with their Steam installer) and Apple (with an OS X update) have managed to write code that rm -rf ~/$EMPTY and thus delete peoples home directories. Less severe than rm -rf $EMPTY/$EMPTY granted, but same program logic fail.

                      However you're right that newer versions of rm does refuse to run -rf on root. I think you need a special flag enabled. So not sure how he managed that.

                      Comment


                      • #12
                        Originally posted by cold fusion View Post
                        It's very easy to accidentally drop in empty variables if you don't test your code properly. Both Valve (with their Steam installer) and Apple (with an OS X update) have managed to write code that rm -rf ~/$EMPTY and thus delete peoples home directories. Less severe than rm -rf $EMPTY/$EMPTY granted, but same program logic fail.

                        However you're right that newer versions of rm does refuse to run -rf on root. I think you need a special flag enabled. So not sure how he managed that.
                        rm -rf --no-preserve-root /

                        (if I remember correctly )
                        Desktop: Intel i5-4690K | 16GB DDR3 | Gigabyte Z97N-WIFI | EVGA GTX 660 3GB | Windows 10
                        Server 0: Gen8 HP Microserver | Proxmox Hypervisor Server 1: Gen8 HP Microserver | FreeNAS

                        Comment


                        • #13
                          123reg have just managed to do similar - deleted a load of VPS & lost all data due to a dodgy script

                          http://forums.aria.co.uk/showthread....40#post2602440
                          Follow me @twitter

                          Comment


                          • #14
                            Seems the serverfault thing in this thread was a hoax anyway:

                            http://meta.serverfault.com/question...-hoax-question

                            Quite ironic it coming just before 123-reg seemingly managed to achieve the same result 'for reals'.

                            Comment

                            Working...
                            X