Had a very similar experience in pretty niche-use cases. LLMs are great if you understand the what you are dealing with, but they are no magical automation tool (at least in somewhat niche, semi-technical use cases where seemingly small errors can have serious consequences).