Copied from discussion on c/Technology:

OSS projects have already started to recognize that LLM is just a tool and how you use that tool matters a lot more than whether you are using said tool at all.

This tool is however not neutral but has a lot of steering and bias built in. Like an advisor who optimizes heavily for personal gain. For example, it constantly tells you that it knows it all and also that your ideas are the best thing since sliced bread. A lot of stuff around them is engineered to increase engagement and make you dependent on them. Precisely like social media which is full of “brain hacks” - no, this isn’t hyperbole, this is the name that engineers at Facebook gave to their creations. As social media, commercial LLMs are “addictive by design”.

Lets bring StackOverflow for analogy: we didn’t ban browsers for code development because one can navigate to StackOverflow and copy-paste some really crappy code.

Stack Overflow is a really good example. I am programming over 40 years, since 35 years as hired programmer, since 25 years on PhD level and I used Stack Overflow a lot to find interesting new angles on issues with seemingly obvious answers. Like “what is the most portable, clearest way to efficiently serialize message data in a C struct, given that type punning is only defined for some compilers and can lead to undefined behaviour”.

Stack overflow has typically various opinionated commented answers here, which gives you different angles and you can decide what is best in your case. This is crucial since a lot - one could say, almost all - of software engineering is about trade-offs.

ChatGPT is the know-all of stack overflow responders: It gives you a single smart-allecky answer and is purposefully built so it tries to take decisions out of your hands. It currently also makes a ton of suggestions of which some have a point, some might apply in other cases but not yours, and some are bullshit. If you let it run free you will easily get ten times more code than needed. And ten times more code means ten times more bugs - this is well-established.

So grabbing LLM to get a sense of direction or shape is an OK use in general, but copy-pasting it’s results blindly is silly and dangerous.

The thing is that commercial LLMs in practice are engineered to short-circuit your judgment and decision-making.

There’s a second argument that AI is ruining OSS communities which I believe is true, but refusing to accept AI code is a self-defeating action. Why? Because AI training needs unadulterated source of knowledge, one not tainted with AI and OSS communities rejecting AI would be that source.

First: These companies will siphon off open source code anyway, be it legal or not. That is outside of the influence of single open source developers or projects. Maybe a matter of high courts and civil society lawsuits to decide. And these will have to fight an uphill battle against the current political climate around favouring large companies in general and that kind of exploitative AI in particular.

But what projects do have influence over is the quality of their code base and what ingresses into it.

Further, it is to expected that the quality of commercial code bases continues to degrade at a rapid pace and also that companies will depend even more on open source infrastructure and libraries - and FOSS developers training the LLMs they use on anything that is new. Because the average commercial project or github project just will turn to shit. Companies will soon wistfully remember old times where Sturgeon’s law was a thing and when only 90% of all public code was shit, and not 99.999% . (Congratulations to that future five nines!).

There is an asymmetry between open source code and proprietary software: Open source code is protected by copyright law. Which GenAI vendor companies practically do ignore. Proprietary software is usually protected by copyright law, and also obfuscation by it being binary or behind server APIs. Currently, this works in favour of companies: They get to use GPL-proteced public code, but are not exposing their own, uh, creations. (And of course, LLM vendors will make the solemn pinky promise that they never ever will publish commercial code their customers are working on as LLM output (ask top-notch mathematicians how that worked out).)

But will this stay like this? Because another issue is that the same methods can be used to decompile binary code, and LLMs can also be trained on decompiled proprietary code. Which will, for example, make its many hidden security issues much more acute. (Another thing is I think one can transform decompiled control structures in binary code to control flow graphs, and then use standard graph search algorithms and graph databases in order to find out which GPL’d open-source code some proprietary product has included, including which potentially vulnerable versions. So-to-speak a software-bill-of-materials by code inspection.)