I think it follows that if you ask it to implement something, it will implement the average thing.
But then it also follows that if you give it the average thing and ask it to improve it by doing something clever, it may identify a clever improvement if such an improvement has ever been done in an analogous setting that exists in its training set.
> Your own cleverness would have to be below average
In this case the difference between myself and the LLM is that I’ve never seen the clever thing before (or I cannot recall it as effectively), but the LLM has and the LLM is specialized to recall it
How probable is that given that LLMs output the average of their training data? Your own cleverness would have to be below average for this to hold.