Just watching the proverbial bodies stack up

  • thisbenzingring@lemmy.sdf.org
    link
    fedilink
    English
    arrow-up
    68
    ·
    edit-2
    3 months ago

    I hate you. Today I discovered a server that appeared to be in working order was actually just hanging on by a thread. The crash when the crowdstrike driver caused the blue screen of death must have corrupted some of the network configuration in the registry. When I couldnt make the failover cluster live migrate to or from that server I ejected it from the cluster and rebooted it, only it never came back up 😥

    Now all my servers are suspect until I can verify that they are healthy.

    • RegalPotoo@lemmy.world
      link
      fedilink
      English
      arrow-up
      31
      ·
      3 months ago

      Ah yes, the realisation that everything dependable and solid in the world is just a monument to hubris, built on foundations of sand

      • Laser@feddit.org
        link
        fedilink
        arrow-up
        6
        ·
        3 months ago

        In general, I don’t think the concept of a configuration database is bad - at least not inherently moreso than every application putting its own configuration file(s) in whichever format it wants somewhere on the filesystem.

        Whether the window registry is a good implementation of this concept is surely debatable.

  • mesamune@lemmy.world
    link
    fedilink
    English
    arrow-up
    44
    arrow-down
    1
    ·
    edit-2
    3 months ago

    This latest debacle is making my department move from windows to Linux. We were already planning it very slowly but then everything crashed at the same time…and all our other services worked except the ones on windows boxes. We can’t afford downtime so it was decided.

    It’s not a good reason but I’ll take it haha.

    • kautau@lemmy.world
      link
      fedilink
      arrow-up
      26
      arrow-down
      1
      ·
      3 months ago

      I don’t know how CrowdStrike works on Linux, but it’s worth remembering that if it’s a kernel level driver like it is on windows, and they release a driver that crashes the Linux kernel, there’s a chance for the same thing to happen.

        • ramble81@lemm.ee
          link
          fedilink
          arrow-up
          29
          arrow-down
          1
          ·
          3 months ago

          Thank you for mentioning that. I really hate how people on here think Linux is some panacea that will magically solve everything. It too it just another tool that depends on how it’s used. CrowdStrike exists for Linux, and it was crashing systems a few months ago.

          The bigger issue is most people who use Linux know what they’re doing. There are a lot of competent Windows Administrators too who didn’t have issues or were able to recover them in a timely manner. What happens though is you have a very large set of people who just need a computer, need is secured and don’t know how to administer or manage it. Doesn’t matter if they’re running Linux, or Windows, they’ll always have the greatest problems. They just happen to use Windows because it offers better Enterprise support options and usability. One day, Linux may be that, but I guarantee it won’t fix all of those pebkac issues.

          • frog_brawler@lemmy.world
            link
            fedilink
            arrow-up
            20
            ·
            3 months ago

            I mostly agree. As someone that’s worked with both Windows and Linux for over 15 years, I think we need to ask the question of “why do we see so many incompetent admins?”

            If you aren’t paying people enough to give a shit about what they are doing, they won’t.

            • themeatbridge@lemmy.world
              link
              fedilink
              arrow-up
              6
              ·
              3 months ago

              The answer is that companies are unwilling to allocate sufficient budget to infrastructure. So anyone competent leaves either because either there is better pay elsewhere, or they don’t want to be held responsible for the shoestring shitshow that companies are willing to pay for.

              • kautau@lemmy.world
                link
                fedilink
                arrow-up
                3
                ·
                3 months ago

                Which is sort of the reason crowdstrike is so popular in the first place. Technically inept leaders want to check a “secure” box in their infrastructure presentation to the board, and certainly don’t want to hire an actual cybersecurity team alongside what they already consider to be an expensive IT team. (Granted they can’t do the mental work of realizing that basically every one of their employee uses a computer every day for hours at a time, and connects to vast networks of computers sitting in datacenters). So to save money, and seeing the legally binding contract, they use crowdstrike.

      • mesamune@lemmy.world
        link
        fedilink
        English
        arrow-up
        6
        ·
        edit-2
        3 months ago

        I personally get nervous when any software wants to mess with drivers unless it’s graphic drivers.

        For work we don’t plan on using cloud strike. We needed to get everything up asap and the os allowed us to do so quickly. Seemingly unrelated systems and Azure was all down for quite some time.

  • Bakkoda@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    40
    ·
    3 months ago

    I was on vacation. They laid off my boss 2 weeks ago. They laid off my bosses boss 3 weeks ago. Guess who no one could contact?

    I found out about all of this 2 days after the fact. Best vacation ever.

      • Bakkoda@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        3
        ·
        3 months ago

        Way ahead of you. GSK/Haleon looks to be unraveling in their toothpaste division. I’m only a long term contact worker so I’ve always got another job lined up. I’ve got another 9 months on this tour but i don’t see them keeping me, i cost too much.

    • wheeldawg@sh.itjust.works
      link
      fedilink
      English
      arrow-up
      1
      ·
      3 months ago

      Wtf did they honestly expect from you in that situation anyway? Do they really think you would come back from vacation to deal with that? From what I can tell, fixing it is barely beyond a T1 hekpdesk amount of knowledge, just an annoying process. Unless there’s more to it, I haven’t looked heard at the details but that’s the gist I’m hearing.

      But unless they’re going to triple my vacation length (in terms of paid days off in other words) and maxing out all my upgrade options for the next one I’ma have to say they can get bent.

      Basically they gotta pay for the best version possible of that trip (assuming there was a trip) and give me enough time to take 2 more.

      I ain’t taking a work call on vacation.

      • Bakkoda@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        2
        ·
        3 months ago

        Funny story. My boss vetoed work phones as a cost savings and at first there was a squawking fest but honestly he handled all communication to us and protected us from so much bullshit. Now he’s gone and no one could figure out how to get in contact with us aside from email/teams.

  • radicalautonomy@lemmy.world
    link
    fedilink
    arrow-up
    15
    ·
    edit-2
    3 months ago

    The Feels Good guy has a stick figure hand sticking up out of his arm. I cannot unsee it, and I suspect now neither can you.