It doesn’t really matter whether the original data is present in the model

Yeah it does. One of the arguments people make is that AI models are just a form of compression, and as a result distributing the model is akin to distributing all the component parts. This fact invalidates that argument.

This isn’t a slam dunk argument that there’s nothing wrong with what an AI does even if we grant it is transformative. It may also simply be proving that the copyright law we have fails to protect artists in the new era of AI.

If we change the law to make it illegal it’s illegal.