• Folstar@lemmus.org
    link
    fedilink
    English
    arrow-up
    19
    ·
    16 hours ago

    Fun anecdote. My dad went from line worker to maintenance lead to running a factory for a fortune 500 company back in the 20th century when working your way to the top was a thing. The factory went from average to the best in the company by a large margin. Dad gets promoted and here our anecdote splits into two tales that demonstrate the modern business world:

    I’m getting lunch in town and the guy who was brought in to take over the factory, we’ll call him Fuckface McFuckface, sees me. He comes over and wants to talk. He sings my dad’s praises then brags about how everything is going even better now. Then McFuckface throws a few jabs about all the wasteful spending on maintenance. I had been listening and smiling up to this point break my silence and ask “so you cut maintenance?”. Fuckface says with a big grin, “yeah, maintenance doesn’t make any money”. I tell him, loud enough that everyone (small town) can hear, that he’s a fucking moron and then walk away. Sure enough, the factory started failing shortly thereafter. McFuckface jumped ship and last I heard was a VP for an even bigger company.

    On the other side, my dad who by every real world metric knew more about successfully running a factory in this company than anyone else, joins this team that travels the globe setting up and fixing operations. He spends the next few years being ignored at every turn. No diploma and no connections in the C-suite. The company starts making things less and outsourcing more. It also gets really into stock buybacks.

  • jj4211@lemmy.world
    link
    fedilink
    English
    arrow-up
    20
    arrow-down
    1
    ·
    23 hours ago

    While frequently true, the recent events seem to be more the opposite, deploying improvements to systems that weren’t broken (yet).

    The FAA? Upgrading to a new piece of network equipment. An act of finally working toward refreshing long neglected infrastructure.

    Microsoft? Some developer pushed a windows os change in July, and circumstances fired off now to expose it as flawed.

    Banking? Again, some developer at one bank pushing some change.

    As to why all at once, could be coincidence, but if it isn’t the likely connecting factor is AI. You need to create the complex configuration for a piece of network gear? What a pain, just feed Claude the current config and ask to port it to new equipment. Want some nebulous software improvement but don’t have any ideas? Have Claude look for, provide a solution, and also have it “audit” and “test” its own solution. I have seen all of the above happen and go horribly wrong when I am actually in a position to intercept, but it is super popular.

    • AEsheron@lemmy.world
      link
      fedilink
      English
      arrow-up
      21
      ·
      23 hours ago

      Read a comment elsewhere that the banks aren’t quite comfortable enough with AI to let it make the code. But, what they are doing, is using AI to hunt for vulnerabilities, and get people to patch them. And the AI are finding tons of holes, so there is pressure to push tons of updates to fix them, because if their AI can find them, then hackers’ AI can find them. And so the workload has exploded with tasks labeled Priority 1, Critical! non-stop, resulting in sloppy updates.

      • rumba@lemmy.zip
        link
        fedilink
        English
        arrow-up
        4
        ·
        16 hours ago

        That’s 99% of what’s happening,

        Compounded by major governmental cybersecurity cuts over the past few administrations and a low-effort FAA update. (not to say the FAA update was lazy, but insufficiently designed/tested/handled)

    • Katana314@lemmy.world
      link
      fedilink
      English
      arrow-up
      3
      ·
      23 hours ago

      It is very possible they became pressured to perform an upgrade because something on the old system was compromised and out of support.

      Legacy stacks quite often maintain old tech, and then somewhere in there will be a library that has a critical exploit and has been totally replaced for future major versions.

      • korazail@eviltoast.org
        link
        fedilink
        English
        arrow-up
        1
        ·
        23 hours ago

        Unsure if you’re defending or objecting to the comment. Flawed and outdated code needs to be replaced and that pressure should be maintained. The ‘Go Fast And Break Things’ model works OK in an internal corporate environment, but not when we’re talking about the FAA or banking. Outlook can fuck right off.

        Outdated hardware needs to be replaced, but I’m with JJ about the likely root cause. Some boss decided that the cheapest way to fulfill their job/contract was to install or deploy whatever ChatGPT told them would work.

        • PuddleOfKittens@sh.itjust.works
          link
          fedilink
          English
          arrow-up
          2
          arrow-down
          1
          ·
          22 hours ago

          Outdated hardware needs to be replaced,

          No it doesn’t. Nuclear launch systems were running on floppy disks in the 2010s, IIRC, if it ain’t broke then “fixing it” is at best a nice to have.

          Now, if it’s insecure then we should replace it. Unless you’re replacing it with something even less secure (like AI, perhaps).

          • korazail@eviltoast.org
            link
            fedilink
            English
            arrow-up
            2
            ·
            20 hours ago

            fair - not broken (yet) doesn’t mean shouldn’t be maintained, though.

            Most solid state things are pretty much static, by definition, but there’s still stuff like plastic dry rot or metal corrosion. We’re talking decades, but we’re also talking decades. If that critical router failed outright the result might have been worse while a replacement was sourced, with a follow-on failure when the code bombed too.

            I’m not defending the outage – there should have been better testing – but I’ve been on the receiving end of a minimal but crucial server’s power supply capacitor failing (the post-mortem was wild to figure out what broke). That server was ancient and should have been replaced before it failed and the fire drill to recover it (even just taking a backup and restoring to a VM) was unpleasant.

      • jj4211@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        23 hours ago

        On the bank, a possibility, on Microsoft, based on what they detailed sounded like a go at performance optimization.

        For security, I get it, there is little time to lose in this scenario, usually. But huge volume of changes for change’s sake are also in play.

  • Mo.@lemmy.world
    link
    fedilink
    English
    arrow-up
    15
    arrow-down
    6
    ·
    21 hours ago

    No. Everyone doesn’t decide to do this shit all at once and pushing this narrative away from the true blame factor (the current administration) is fucked and scary and propaganda.

    Are there nuggets of truth here? Yes, absolutely. Anyone in tech knows that. But this is not that. There’s too much happening in core systems all at once at the same time and there’s no otherworldly reason that all of these people independent of one another all decided to update their systems at the same fucking time.

    Get off of it.

    Twisting the narrative or spinning it away from the ghouls that lit this fuse years ago is fucking dangerous and I’m really surprised to see it being pushed so very hard here of all places.

    We’re smarter than that.

    • Bahnd Rollard@lemmy.world
      link
      fedilink
      English
      arrow-up
      8
      ·
      16 hours ago

      Next week is Thanksgiving in the US, yes, everyone did decide to update critical things all at the same time so they can fuck off until next year. The american work calendar has about three fully functional uninturupted weeks left. I have a hunch that the timings are partially caused by people rushing to get things done by the start of the real holiday clusterfuck.

    • Optional@lemmy.worldOP
      link
      fedilink
      English
      arrow-up
      5
      ·
      19 hours ago

      In theory, but in practice the market bends towards bullshit and quadrant-leaders. Those companies (microsoft) are fucked up for the same reaons - they didn’t want to let the engineers apply redundancy beyond a paper thin excuse layer. Rinse, repeat, and wait.

    • JustEnoughDucks@feddit.nl
      link
      fedilink
      English
      arrow-up
      22
      ·
      1 day ago

      The market optimizes for extreme short term profit squeezing with the least amount of labor possible. The only efficiency it has is in exploitation and labor extraction.

  • jackal@infosec.pub
    link
    fedilink
    English
    arrow-up
    104
    ·
    1 day ago

    You missed the second post so I have it below. I’m dealing with this at my job and it’s so fucking aggravating to deal with executives who only care about this quarter’s performance.

    Mastodon post from @Viss@mastodon.social

    In case of missing image:

    you know when you cheap out and buy the cheap thing, but it turns out the cheap thing works like 5 times then breaks, so you have to decide whether to concede to defeat and spend the money on the right thing, but the sunken cost fallacy is tugging at you to buy the cheap thing and try again, but then exactly the same happens again and now youre kinda stuck?

    its that but its computers acting as digital infrastructure all over the country

    run by guys who abjectly refuse to admit defeat ever.

    • Valmond@lemmy.dbzer0.com
      link
      fedilink
      English
      arrow-up
      20
      ·
      1 day ago

      Yeah the problem is that the middle management has an order to carry out; make +2.5% this quarter. They do not have an order; make it sustainable.

      My guess is that it’s all about metrics, you can see the line go up, but you can’t assess sustainability without relying on trust (which can and will be broken and abused in our capitalist system, because we all need to eat), so line go up it is.

  • WhoIzDisIz@lemmy.today
    link
    fedilink
    English
    arrow-up
    205
    arrow-down
    1
    ·
    2 days ago

    Forgot to clarify that the reason they “don’t feel like fixing any of it” is because doing so doesn’t directly contribute a positive effect on the bottom line, and they’re too fucking short-sighted to understand how having invested in reliable infrastructure would have kept them from hemorrhaging money now.

    • credo@lemmy.world
      link
      fedilink
      English
      arrow-up
      11
      ·
      22 hours ago

      I’ve been trying to answer the question, “How do you show the value of preventitive action?” for a long time. I still don’t have a good answer, other than, you’ll know when you don’t do it.

      Same with public heath and I’m sure many other professions I can’t think of right now.

    • Seb the goblin@lemmy.world
      link
      fedilink
      English
      arrow-up
      36
      ·
      1 day ago

      Literally. I had to fight tooth and nail, repeatedly saying to people with Head in their title that if this [migration project that should have been done 2y ago] doesn’t go ahead, then in 6 months company will not have a product. Still ended up with lots of whinging, compromises, running over deadline due to said compromised, and then performance reviews coming in with “your product roadmap didn’t have much in it” like bruh, we were trying to put out the damn thermite trying to burn down everything

    • Boomer Humor Doomergod@lemmy.world
      link
      fedilink
      English
      arrow-up
      5
      ·
      1 day ago

      What do you mean? The rental cars and ride shares and hotels made bank. $300 hotel rooms and $600 rental cars.

      It’s just poor people’s money anyway and they don’t have enough to be worthy of consideration

    • jimmy90@lemmy.world
      link
      fedilink
      English
      arrow-up
      1
      arrow-down
      5
      ·
      edit-2
      22 hours ago

      that applies to the IOT bullshit that happened to the supermarket freezers yeah

      the “systemic” outages from Microsoft, the FAA and Linux kernel bug tsunami are incredibly complex systems that are tested to hell and back but human error will barely creep through

      not because the fucking boss is a greedy capitalist

      i know this is how the commie brain rationalises the complex world you live in which is why you’re such delayed learners

  • Stupidmanager@lemmy.world
    link
    fedilink
    English
    arrow-up
    8
    ·
    1 day ago

    I work in the industry, I have these friends and coworkers too. One of them is wrong, a lot - in this case He still is. But this doesn’t go unnoticed when the shit hits the fan. now we work in an industry that wants to move fast, so expect more of it with far more blame.

    The sky is falling folks.

    • Lemmilicious@feddit.nu
      link
      fedilink
      English
      arrow-up
      6
      arrow-down
      3
      ·
      1 day ago

      I know AI has been marketed as some big boost in finding vulnerabilities, but as far as I know that’s still all marketing talk without real substance. The important thing, using AI or not, is that finding vulnerabilities and fixing them requires intent, which seems to be the missing piece in this case.

      • Akuchimoya@startrek.website
        link
        fedilink
        English
        arrow-up
        5
        ·
        24 hours ago

        It turns out it’s much easier to destroy things than to build things well. I can destroy (greatly destabilize and render unsafe) a building by renting a sufficiently large vehicle and ploughing it through the building. I have literally zero skills it takes to design and/or build a structurally sound building. Finding vulnerabilities in code is orders of magnitude different than building secure code in the first place.

      • filcuk@feddit.uk
        link
        fedilink
        English
        arrow-up
        8
        ·
        1 day ago

        Have you missed the practically weekly news about hundreds of old CVEs found across various established codebases?
        It’s just a matter of which company wants to flex their new model as an advertisement.

        • Lemmilicious@feddit.nu
          link
          fedilink
          English
          arrow-up
          1
          ·
          24 hours ago

          Sorry for being obtuse, but I actually don’t understand how your comment relates to mine. But it does indeed seem to be a matter of which company wants to flex their model, like you say!

          • filcuk@feddit.uk
            link
            fedilink
            English
            arrow-up
            2
            ·
            23 hours ago

            I was pointing out that AI has been making waves in finding vulnerabilities, contrary to you saying it appears to be nothing but marketing.

            It is a double-edged sword. Anyone with an agent and a few hundred bucks can find a weakness to exploit somewhere. Scammers are making fully automated exploits that are easier than ever to fall for. Absolute chaos.

            • Lemmilicious@feddit.nu
              link
              fedilink
              English
              arrow-up
              1
              ·
              20 hours ago

              Ah yeah, then I get you, thanks! I’ve noticed those waves of marketing for sure, I haven’t looked into most of them but I was a bit burned by the Claude Opus crap that was very little bite in proportion to the bark. From what I read it mostly found the same vulnerability lots of times in different places, and seemingly not because it was an unusually hard to detect vulnerability, but rather because they took the time and effort to direct it in the right place.

              This is sort of what I mean, whether AI or not, it takes intent. Whether you think humans or AI do it better, in the success stories I’ve heard intent has been the common factor. That’s my main idea!

    • chonglibloodsport@lemmy.world
      link
      fedilink
      English
      arrow-up
      34
      ·
      1 day ago

      That’s close enough really. Because of AI they laid off tens of thousands of their most experienced tech experts. Now they’re letting the vibe coders run the show. “No one knows how any of this works” is the norm now.

      It’s like the airline laying off all the pilots mid-flight and they bail out of the plane in parachutes, leaving the passengers to fend for themselves.

    • gibmiser@lemmy.world
      link
      fedilink
      English
      arrow-up
      27
      arrow-down
      2
      ·
      edit-2
      2 days ago

      As far as I know WE DONT KNOW YET. Why is everyone acting like they know what is happening? They are deliberately keeping it a secret. We are just speculating.

      Pick any combo of the following:

      Failure to invest in infrastructure

      Cyber warfare

      Shitty AI

      Some other possibilities

      We don’t know yet.

      • LilB0kChoy@reddthat.com
        link
        fedilink
        English
        arrow-up
        20
        arrow-down
        1
        ·
        2 days ago

        What would make you feel like you “know”? Would you trust a government run by the Trump admin? Or the billionaire controlled mainstream media?

        I have worked in IT systems administration for a little over a decade now and, while I don’t have any direct knowledge, this has all the hallmarks of poor Infra management.

        Combine an increase in “vibe-coding”, the layoffs of senior technical staff, and an upcoming change freeze for the holidays and you get the scenario we’re seeing.

        Bad code being pushed in a hurry with nobody around who understands the systems enough to catch that these changes will break shit.

        It could be other things but Occam’s Razor would indicate it is not.

        • Spawn7586@lemmy.blahaj.zone
          link
          fedilink
          English
          arrow-up
          6
          arrow-down
          2
          ·
          2 days ago

          The guys that know told you, but their explanation was boring and you didn’t understand the technical terms. You know, the friend the post was talking about… You will always think it’s AI or some other shit anyway. Dependency on vulnerable libraries and lack of security updates is way too boring for ya all

      • amniotic druid@lemmy.world
        link
        fedilink
        English
        arrow-up
        4
        arrow-down
        4
        ·
        2 days ago

        Cyber attack by whom? The US, China, and Russia are literally summiting this week. Putin would’ve surely told Iran to cool it with the funny business while he’s working. It would be the dumbest possible move for either of those countries or their allies to launch a cyber attack right now

        • Valmond@lemmy.dbzer0.com
          link
          fedilink
          English
          arrow-up
          6
          arrow-down
          2
          ·
          1 day ago

          I know it’s complicated, but it isn’t actually Putin himself who does the hacking, he has other people do it for him.

        • gibmiser@lemmy.world
          link
          fedilink
          English
          arrow-up
          14
          ·
          2 days ago

          No, any one of them could do it at any time as part of ongoing operations. You don’t know what they are doing, and neither do I.

          • amniotic druid@lemmy.world
            link
            fedilink
            English
            arrow-up
            3
            arrow-down
            5
            ·
            2 days ago

            Can you not make an educated guess? A cyber attack is not educated. You didn’t say alien invasion in your post but iT cOuLd bE AnYtHinG

            • gibmiser@lemmy.world
              link
              fedilink
              English
              arrow-up
              5
              ·
              2 days ago

              Right, well I try not to fall into the trap of letting my biases color my reality as best I can. But go ahead and equate my position to stupid bullshit and have fun choosing your own adventure.

              • amniotic druid@lemmy.world
                link
                fedilink
                English
                arrow-up
                2
                arrow-down
                4
                ·
                2 days ago

                Laundering the cyberattack narrative is, being as charitable as I possibly can, contributing nothing but hype and coldwarmongering for the Defense industry. This is a very dangerous line of thought to pursue.

                It’s also exactly what the current Broligarchy wants you to do. Externalize it onto one of the Bad Guys… who have zero reason at all to launch what would be the largest act of cyber warfare in history. I guess a layman like me just isn’t capable of making sense of these 5D chess maneuvers.

                • protist@retrofed.com
                  link
                  fedilink
                  English
                  arrow-up
                  4
                  ·
                  2 days ago

                  Laundering the cyberattack narrative

                  They literally said no one knows what’s going on and to not assume

        • ReptilianCleric@lemmy.today
          link
          fedilink
          English
          arrow-up
          1
          ·
          1 day ago

          What? The op specifically talks about security being ignored. And security vulnerabilities don’t exploit themselves. So what are you even talking about?

    • mushroommunk@lemmy.today
      link
      fedilink
      English
      arrow-up
      11
      ·
      2 days ago

      I really don’t get it. I’ve had several non tech people ask my opinion on it and then do the exact same thing. They don’t want to hear that script kiddies with LLM might be amplifying the problem but it’s a symptom.

    • RedGreenBlue@lemmy.zip
      link
      fedilink
      English
      arrow-up
      4
      ·
      1 day ago

      AI or no AI; the systemic issues and gaping holes have to be fixed.

      AI only accelerate the urgency to fix things. Might be a good thing, in the long run, that it makes the problem visible. Just gonna hurt in the short term.