Purge it with fire!



  • -- "Alright Intern, let's figure out how to add PostgreSQL 9.4 side-by-side with current install of PostgreSQL 9.3 on our dev server...."

    Skype pings

    -- "Shit, have to put out a fire on a different front. BRB."

    15 minutes later, working on a hotfix in my dev environment

    -- "Hmm, I can't connect to my dev database. It's like it's.... gone? ... Wait... INTEEEERN!"

    Walk over to the kid, find him frantically trying to restore data from backups, having apt-get remove-d all PostgreSQL instances from the server.

    With a --purge flag. Because a tutorial said so.

    Bye bye config files and all data.

    I was super composed and nice. Didn't yell at all.


  • :belt_onion:

    You... just install it and it sets itself up to listen on the next available port (at least on Debian based systems)?



  • Yup. for some reason he then tried to remove 9.4. When it failed multiple times, he did purge with a wildcard, that somehow took 9.3 too.

    Still investigating.


  • mod

    @cartman82 said:

    Bye bye config files and all data.

    More and more I'm liking the DevOps idea of storing config files in version control so they can be restored on the fly.


  • :belt_onion:

    @cartman82 said:

    he did purge with a wildcard, that somehow took 9.3 too

    apt-get remove --purge postgres* instead of apt-get remove --purge postgres-9.4*?

    I'm guessing at package names, actually. postgres-x.y is a metapackage IIRC, but not sure if purging it purges all the things off the top of my head.



  • @Onyx said:

    not sure if purging it purges all the things

    TEST IT!



  • @Onyx said:

    apt-get remove --purge postgres* instead of apt-get remove --purge postgres-9.4*?

    I'm guessing at package names, actually. postgres-x.y is a metapackage IIRC, but not sure if purging it purges all the things off the top of my head.

    From the history, here's what he did

      267  sudo apt-get remove --purge postgresql-9.4*
      268  sudo apt-get remove --purge postgresql-9.4
    

    So seems fine. But somehow, this nuked 9.3 too. Strange.


  • :belt_onion:

    Depends on what he did before. I just now remembered - it's possible 9.4 replaces 9.3 by default and he just upgraded everything to 9.4, couldn't find 9.3 any more and then tried to roll back?



  • On Debian, 9.4 is installed side-by-side with 9.3. Everything in its own neat little dir. Very nice structure.

    Here's the whole history that I have. Unfortunately I don't have the console output.

      220  apt-get remove postgresql-9.4
      221  apt-get remove postgresql-9.4-ip4r
      222  apt-get remove postgresql-9.4-postgis-2.1
      223  apt-get remove postgresql-9.4-postgis-scripts
      224  apt-get remove postgresql-client-9.4
      225  apt-get remove postgresql-contrib-9.4
      226  ls
      227  cd /var/lib/
      228  ls
      229  cd postgresql/
      230  ls
      231  rm -rf 9.4/
      232  ls
      233  cd /usr/lib/postgresql/
      234  ls
      235  service postgres status
      236  service postgresql status
      237  su postgres
      238  cp /tmp/data4301* /
      239  cp /tmp/main4301* /
      240  ls /\
      241  ls /
      242  apt-get install postgresql-9.4
      243  apt-get install postgresql-9.4
      244  cd /var/lib/postgresql/9.4/main
      245  ls
      246  cd /var/lib/postgresql/9.4
      247  apt-get update
      248  apt-get upgrade
      249  apt-get remove postgresql-9.3
      250  apt-get remove postgresql-9.4
      251  apt-get --remove postgresql-9.4
      252  apt-get remove postgresql-9.4
      253  dpkg -r postgresql-9.4
      254  nano /var/lib/dpkg/info/postgresql-9.4.prerm
      255  dpkg -r postgresql-9.4
      256  nano /var/lib/dpkg/info/postgresql-9.4.prerm
      257  ls
      258  apt-get update
      259  apt-get upgrade
      260  apt-get install postgresql-9.4
      261  nano /var/lib/dpkg/info/postgresql-9.4.prerm
      262  dpkg -r postgresql-9.4
      263  apt-get remove postgresql-9.4
      264  ls
      265  cd /var/lib/postgresql/
      266  ls
      267  sudo apt-get remove --purge postgresql-9.4*
      268  sudo apt-get remove --purge postgresql-9.4
    

    At one point he did apt-get remove postgresql-9.3, but he says that was a mistake and he hit "no" before apt-get did anything (I believe him, that wouldn't have touched /etc stuff).

    At other point, he edited something inside "/var/lib/dpkg/info/postgresql-9.4.prerm". Have no clue.

    The only thing here that could have cleaned 9.3 config dirs is the purge.



  • Good. I wouldn't expect you to yell...

    I would expect you to reach into your Disaster Recovery plan, and like, recover... Because you WERE doing your backups right? Data + Config?

    The problem with things like backups is that they represent dull drudgery to most dev / operations folks. They're a lot more appealing once your back is to the wall though...



  • @mjmilan1 said:

    Good. I wouldn't expect you to yell...

    I would expect you to reach into your Disaster Recovery plan, and like, recover... Because you WERE doing your backups right? Data + Config?

    The problem with things like backups is that they represent dull drudgery to most dev / operations folks. They're a lot more appealing once your back is to the wall though...

    We have backup strategy, but not restore strategy. So we were back online after about half an hour, instead of few minutes.

    Live and learn. As soon as we are done with this, Intern will update his scripts and work out an actual restore strategy.


  • Winner of the 2016 Presidential Election

    @cartman82 said:

    purge with a wildcard

    yeah, that's never a good idea...


  • Discourse touched me in a no-no place

    @Yamikuronue said:

    More and more I'm liking the DevOps idea of storing config files in version control so they can be restored on the fly.

    People don't do this sort of thing as a matter of course? (This is my desktop PC, hence the local svn server - something more important would be remote.)

    [root@desktop ~]# svn info /etc
    Path: /etc
    Working Copy Root Path: /etc
    URL: svn://localhost/var/svn/desktop/trunk/etc
    Repository Root: svn://localhost/var/svn/desktop
    Repository UUID: 21d91ae4-40ce-41e3-937e-66014df0f365
    Revision: 26
    Node Kind: directory
    Schedule: normal
    Last Changed Rev: 26
    Last Changed Date: 2015-04-29 09:49:04 +0100 (Wed, 29 Apr 2015)
    


  • @cartman82 said:

    Bye bye config files and all data.

    I was super composed and nice. Didn't yell at all.

    Why did he have access to a live system? For that matter why would YOU? Or any Human?

    With a robust automatic deploy, this would have been tested in isolation, once working a trial deploy done to one or more target "test" environments, and only after they all passed would the trigger have been pulled and an automatic update of your "live" system done.....



  • @TheCPUWizard said:

    Why did he have access to a live system? For that matter why would YOU? Or any Human?

    With a robust automatic deploy, this would have been tested in isolation, once working a trial deploy done to one or more target "test" environments, and only after they all passed would the trigger have been pulled and an automatic update of your "live" system done.....

    Why would you have access to the commenting system? Why would any Human who didn't read the OP?

    It was training on a dev server.


  • Winner of the 2016 Presidential Election

    @Cartman82 wins the comeback award of the day.



  • @Yamikuronue said:

    More and more I'm liking the DevOps idea of storing config files in version control so they can be restored on the fly.

    No offense but... duh?

    Are there people who don't do that already?


  • mod

    @blakeyrat said:

    Are there people who don't do that already?

    Things like database config files? Yeah. I've never worked at a place that DID version anything other than code itself.



  • Why is he using an OS that's so difficult and buggy?

    That's a better question.

    Postgres runs on good OSes. It even has a nice graphical installer and a GUI management app on them.


  • Winner of the 2016 Presidential Election

    @blakeyrat said:

    Why is he using an OS that's so difficult and buggy?

    That's a better question.



  • @Yamikuronue said:

    Things like database config files?

    Well I usually work with SQL Server which doesn't really have such a thing, as far as I'm aware. You add the "master" and "model" databases to your normal DB backup and you're golden.

    I guess MongoDB has config files, and back when I was using that I didn't back them up-- they were like 4 lines long, and every line was a machine-specific path, so. Anyway.

    But for stuff where config files are important, for example, IIS' various config XML files, they're always in source control.



  • Intern can SUDO?



  • But what about the <!--IIS 6 :trolleybus: --> Metabase?



  • @TwelveBaud said:

    But what about the Metabase?

    They trashed that in IIS 7.

    In any case, the IIS model is that the application can dictate to the web server every feature/option it needs enabled and configured (in its own web.config), so that's basically the only part you actually need to back-up. The application-level configuration is the web server configuration.

    This is one of the things that makes IIS so much better than open source web servers, where you end up in an awful situation trying to juggle 47 php.ini files around.


  • sockdevs

    @blakeyrat said:

    This is one of the things that makes IIS so much better than open source web servers, where you end up in an awful situation trying to juggle 47 php.ini files around.

    Wouldn't that still have to be done if you're running PHP under IIS? Also, I'm not sure there's any real practical difference between juggling 47 .inis and 47 web.configs.


  • :belt_onion:

    @blakeyrat said:

    This is one of the things that makes IIS so much better than open source web servers, where you end up in an awful situation trying to juggle 47 php.ini files around.

    You can override php.ini with a per-application php.ini as well (or use ini_set). And you can control web server settings through .htaccess.

    I don't see your point.



  • @blakeyrat said:

    This is one of the things that makes IIS so much better than open source web servers, where you end up in an awful situation trying to juggle 47 php.ini files around.

    Haha! Joke's on you. I have 73 xml files floating around. Oh. :crying_cat_face:



  • @RaceProUK said:

    Wouldn't that still have to be done if you're running PHP under IIS?

    Yes, well, PHP just sucks.

    @RaceProUK said:

    Also, I'm not sure there's any real practical difference between juggling 47 .inis and 47 web.configs.

    The web.config travels with the application.

    The php.ini is (nominally) per-server. php.ini is fine as long as you have one application per server, or as long as every application on the server can work with the same configuration. But it's a shitty idea otherwise, as is all per-server configuration.

    @Onyx said:

    You can override php.ini with a per-application php.ini as well

    That's relatively new then.

    @Onyx said:

    as well (or use ini_set).

    That's shit.


  • sockdevs

    @blakeyrat said:

    Yes, well, PHP just sucks.

    Too true; give me ASP.NET any day. Or dysentery; at least that would keep me away from the PHP.
    @Onyx said:
    You can override php.ini with a per-application php.ini as well (or use ini_set).

    Can you override php.ini at the directory level? I know you can with .htaccess and web.config


  • :belt_onion:

    @RaceProUK said:

    Can you override php.ini at the directory level?

    Virtual host level. You have to set it up in the virtual host configuration though. Had to look it up myself, last time I used that the server was set up to do that for each virtual host by default. Been a long time, so I'm slightly off - it does require an extra step. I rarely touch any of the ini settings so it's been a while.

    Setting it up in .htaccess works as long as you have overrides allowed, yes.



  • Not for things that work during request parsing. If you need to accommodate a host that may have register_globals or magic_quotes_gpc enabled, you're gonna have to write PHP code to undo the damage.

    EDIT: oh, and in case of open_basedir and friends you're screwed.


  • :belt_onion:

    I'd rather nuke it from orbit in that case, TBQH.

    Also, with the prices on VMs these days, shared hosting is not something I really want to have to deal with ever again. And I'm going to use sane defaults on my own install.



  • Yeah, my PHP experiences are divided between the current company (where the product is always on a dedicated VM or server) and the open source project I worked on before that (which was ~8-9 years ago and had to accommodate the very cheapest of PHP/mysql stacks because of the target audience).


  • sockdevs

    @blakeyrat said:

    No offense but... duh?

    Are there people who don't do that already?

    You have no idea. Some of us don't even have version control yet.



  • I'm actually trying to write a Java webapp using JPA that has zero XML files just to say I did it.

    Apparently it is possible in Servlet 3.0+, JPA 2.1+, Spring 4.1+.

    Edit: Ugh, "easily" doing this requires Spring Data JPA, which I've been trying to avoid because it uses "magic" interfaces to create DAOs. Maybe living with just persistence.xml is OK after all.


  • Discourse touched me in a no-no place

    @Arantor said:

    Some of us don't even have version control yet.

    Some of us, you may recall, have to put up with people who derail a discussion about source control with "we need to write our own because [insert stupid here]".


  • Discourse touched me in a no-no place

    @powerlord said:

    I'm actually trying to write a Java webapp using JPA that has zero XML files just to say I did it.

    How are you planning to avoid having a deployment descriptor? (I don't mind having some XML config. But then I use Maven… :scream_cat:)



  • @mjmilan1 said:

    I would expect you to reach into your Disaster Recovery plan,

    Not into his back pocket? (No, he can't reach into my back pocket, thanks. He needs to have his own GAU-8.)



  • Time to put Docker + Vagrant/Puppet to good use. Welcome to 2015!



  • @dkf said:

    How are you planning to avoid having a deployment descriptor? (I don't mind having some XML config. But then I use Maven… :scream_cat:)

    Using a javax.servlet.ServletContext and javax.servlet.ServletRegistration.

    Having said that, my class that sets this up is implementing Spring's org.springframework.web.WebApplicationInitializer, which exists exactly for this purpose.

    The example for that uses an XmlWebApplicationContext, but I'm instead using an AnnotationConfigWebApplicationContext.


  • Discourse touched me in a no-no place

    Ah, and that's bootstrapping using the JAR Service mechanism. We've actually been purging that from our codebase as we've been switching to being OSGi-driven. :smile:



  • @blakeyrat said:

    [quote=Yamikuronue]
    More and more I'm liking the DevOps idea of storing config files in version control so they can be restored on the fly.

    No offense but... duh?

    Are there people who don't do that already?
    [/quote]
    At my last job, we tried... but the people that owned the servers weren't developers and absolutely refused to check any of the changes they made into source control (or even inform the development department). So we ended up with something in source control that kind of looks like the current config file.



  • @Eldelshell said:

    Time to put Docker + Vagrant/Puppet to good use. Welcome to 2015!

    I barely convinced boss to move off Centos 5.

    See you in 2027.


  • :belt_onion:

    @cartman82 said:

    I barely convinced boss to move off Centos 5.

    Ouch. I'm glad mine listens to me when it comes to that, then.

    We did use CentOS 6 in the olden days, and man was it a pain. Had to enable epel and then I still had to compile stuff from source. No thanks.



  • @Onyx said:

    We did use CentOS 6 in the olden days

    We are still on CentOS 6 but there is a plan to move to 7.


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.