Even though I don't like it for myself, I thought about trying to do a full-fledged thesis work for a friend with the help of LLM. But unfortunately, I failed. Either it was a prompting skill-issue or that LLMs haven't reached that state yet.
My thoughts lean towards the latter. Although Claude is fairly good at writing functional code to achieve the desired result, it like its counterparts is not even nearly close at understanding statistical nuances. I think, it's about datasets and training. Training dataset even for complex programming problems is, I suppose, available in plenty amount, and all the documentations for programming languages do not have any conflicts. But firstly, the training amount on real world dirty statistics is available in less amount, and that different textbooks are written for different levels of understanding. And many a times, different authors use different terminologies for same things, and...
My brain's auto-forming connections with former thoughts, but yeah, LLMs basically extract the concepts from labels, and wherever the labels are not universally consensually defined for concepts, it might give LLMs a hard time extracting the concepts.
I don't think there's something fundamentally different about statistics. If it's tacit so is programming. The only thing I can think of is the log, or in other words, data for training. Getting LLMs to do non-nonsensical statistical work by giving them published research papers, will only be as good as showing LLMs only the GUIs or command outputs of programs instead of actually including the code for the program in the training dataset.
The concept of reproducibility in research does have been something people have started to talk about, but still, it's scarce. What we need is a log of statistician -- something like a field diary, which contains all assumptions, hypothesis, decisions, the rationale behind decisions, the bad results, the deciding of an alternate route, or in short, the whole truthful process of the work. And I believe it's not limited to statisticians only. More fields should be giving a thought about getting the tacit ideas documented down somewhere, so that we are able to build better LLMs.