I just finished a coding experiment using Claude AI and Codex.
In this experiment, I assigned Claude the role of architect and prompt designer, and Codex the role of coder. I served as the human in the loop, responsible for the high-level system design, architecture stack, functional requirements, and final testing.
The project covered five problems in quantum mechanics. The generated code and results were then tested against peer-reviewed published data to verify the mathematical accuracy. The goal was not to solve quantum mechanics with LLMs, but to document where and how the models make mistakes.
To my surprise, Codex performed nearly flawlessly. Claude struggled in the role of architect (prompt designer)
The main problems came from Claude, especially its tendency to drift away from the overall project goals and to prompt Codex to implement functionality that was outside the original design, even after those goals were clearly restated multiple times.
If you are interested in failure modes using LLMs to write software and in the details, here are the links to the code and the results:
GitHub (The Code)
Zenodo (The Report / Paper)
.. or if you prefer
ResearchGate
The bottom line is that I was able to complete the project and obtain confirmed, correct quantum mechanics results, but it took a great deal of effort to keep Claude aligned with the overall goals. Claude repeatedly drifted toward objectives and implementation paths that were not part of my direct instructions.