The One Thing That AI-Generated Code Can't Do
In which the author notes that someone has inadvertently left a gun on our industry's mantlepiece
I’ve written before about an experiment in vibe coding I did where I (a professional software developer) prompted Claude Code to create a guitar practice application for me with as little input or review from me as possible. It performed very well indeed. The application performs exactly as desired and my guitar technique is improving steadily, all without me ever looking at or modifying the application’s code. It took me longer than I like to realize that one thing was missing from the AI’s capability. It came when I looked at the Github repository that Claude had created and saw that it was marked as MIT licensed. Not surprising, you might think, MIT is a reasonable “default” open source license, which you’ve seen on hundreds of Github repos.
Did you catch the issue?
I didn’t ask Claude to put that license there. Claude just did it. Because of course you put an MIT license on an open source project. That’s just what you do, like telling your wife that her dress is pretty before going to a party. The problem is that Claude may not have had the standing to do it, and I may not have had the standing to ratify it, and the license may therefore be granting permissions that nobody actually has the authority to grant, over a work that may legally belong to nobody at all.
If you thought from the title that this was going to be some dorm-room bong-hit rant on how AI-generated code couldn’t be “truly creative” or “generate production-quality code” or “handle legacy codebases”, I’m happy to disappoint. There are still plenty of other websites to find those takes, although they are getting fewer and more screechy of late. I know AI-generated code can do all those things, and has been able to for about seven months at this point. No, the thing that AI-generated code can’t do is both less complex technically and more fundamental economically than that.
Purely AI-generated code can’t be copyrighted. (Probably.) And if it can’t be copyrighted, then none of the open source licenses will work with it.
Before we go any further, I will caution that nothing in this post is legal analysis or advice. I am not a lawyer. I’m extremely not a lawyer. You know your idiot cousin-in-law who keeps saying things are “moot” but couldn’t cut it as a notary? I’m less of a lawyer than him. Who’s reading this? Are you on a cellphone? CRANK CALL! CRANK CALL!
Okay, with that out of the way, lemme start on the nature of copyright law as understood by an everyday engineer who has needed just enough of it to build stuff.
Issues with content produced by other than human hands aren’t new. Back when photography was the hot new technology, it was argued that photographs could not be copyrighted, as they were the mere production of a machine. The courts took this seriously, but eventually ruled that some photographs could be copyrighted, if they required some amount of creative work on the part of a human photographer in choosing and composing the photograph. Crucially, they left open the question of whether photographs without such human creative work were available to be copyrighted. It has since been ruled that photographs created entirely by animals (like the “monkey selfie” above) cannot be copyrighted by anyone — the animals in question included. Recently courts have extended this to AI, indicating that purely AI-generated images cannot be copyrighted. Overall, this as as close to settled law in the US as anything gets.
Careful readers will note that I said “purely” AI-written code cannot be copyrighted, which raises questions of just what is meant by “purely.” Is it enough for a human to make changes to the AI-produced code? To suggest changes? Is it enough to specify what the application should do? My personal guess is the answers are “yes,” “no,” and “no,” but the issue isn’t going to be settled by my personal guess. It’s going to be answered eventually by teams of lawyers fighting it out in marble-lined courtrooms at rates starting at $500/billable hour.
In the case of my guitar practice application experiment, I prompted the AI to produce the application. I never modified or even looked at the source code it produced, and indeed I’ve never bothered to learn the computer language (Swift) that the application is written in. I went as far as I possibly could to avoid the possibility that I might have written the application, simply as an experiment in the state of the art of agentic software development. What this inadvertently managed to create was a near-perfect instance of code that is probably not copyrightable, and literally owned by no one.
At this point, you might be asking yourself just why this matters. The answer is simple. All of the open source licenses are based on copyright.
Every one of them — MIT, GPL, Apache, BSD, all the way down to the ones nobody uses except that one guy at your company who really cares about licensing (and dubstep, for some reason) — is a copyright license. They work by saying: the author of this work holds copyright, and hereby grants you certain permissions. No copyright, no permissions to grant. The license becomes a legal instrument with no valid grantor, like a deed of sale signed by nobody.
Oh look, someone left a can of worms, strangely unopened.
Open source licensing isn’t just a bureaucratic formality. It’s the legal load-bearing structure underneath most of the software industry. The GPL’s famous “copyleft” clause — the one that requires derivative works to remain open — only functions because copyright exists to enforce it. Remove the copyright substrate and the viral property of copyleft evaporates entirely for AI-generated code. The MIT license on my guitar app isn’t just possibly invalid. It’s possibly void in a way that has implications for every project that might incorporate that code downstream.
From there it gets worse.
There are entire business models built on dual licensing — companies that release software under GPL for open source use and sell commercial licenses to enterprises who don’t want the copyleft obligations. The commercial license is what pays the engineers. If the code has no copyright, there’s no commercial license to sell either. The business model doesn’t bend. It just stops.
And then there’s the question of what “contributing” to an open source project even means when your contribution was written entirely by an AI. You submitted a pull request. The code is probably not yours. The project maintainer accepted it under a license that may not apply to it. Everybody did everything right by current norms and the result is legally undefined — a null-pointer exception that has the First Circuit Court of Appeals as its catch-block.
I want to be careful here, because this is the point at which I most acutely feel the absence of a law degree. (I am not a lawyer. We’ve covered this. CRANK CALL.) These questions aren’t settled. The courts haven’t gotten to most of them yet, because the volume of purely AI-generated code in production codebases has only recently reached the point where someone sufficiently aggrieved would have sufficient financial motive to fund the litigation. That day is approaching faster than either the legal system or the software industry is prepared for.
So there’s Chekhov’s gun. It’s sitting on the mantelpiece, it was placed there by nobody in particular, and the play is still in the early acts. The software industry has been writing AI-generated code into production systems, open source projects, and commercial products for months now, at rapidly increasing rates. The legal surface area is now enormous. We just haven’t had the lawsuit yet that forces anyone to look at it carefully.
When those lawsuits start coming, a lot of things that currently seem settled are going to become very interesting very quickly. “Interesting” is lawyer-speak for “I need to look into buying a really nice boat.”
I’ll leave the worms here, in this can, which I am handing to you, dear reader. I’ve done my part. I built the guitar app, I watched the AI put a license on it, I thought about it longer than was probably good for me, and I wrote this down.
I am not a lawyer. But I can see the gun from here.
As ever for this blog and as seems especially appropriate given the subject matter: this post was co-written with the able assistance of Claude Sonnet 4.6. This raises its own interesting questions about who holds the copyright on this particular piece of writing. I have some guesses. I’m not telling you what they are.
Open source is a way of approaching software and it's distribution. But software doesn't care and certainly technology doesn't either. Consequently, open source will inevitably have to change it's modality as culture and business evolve.
"Open source" won't mean the same thing in the future and that's entirely ok. No one should feel necessarily threatened by that. If they are, then, they are missing the meta.