The difference is that with arithmetic and calculators I am simply playing the rules of a formal game without regard to the semantics of those numerical symbols. I don't care so much what the numbers mean as much as I care that I have executed the rules correctly and arrive at the "right number."
With language however we are attempting to refer to some kind of underlying meaning or reality, and an LLM will not give you the understanding of the things being referred to - only the outward representations themselves. If you are indeed interested in meaningless exchange of symbols a la Searle's Chinese Room then perhaps there is some utility in this; otherwise it's the act of digesting and comprehending meaning, and the internal cognitive processes that go into the production of the written artifact that matter, not just the written artifact itself.
With language however we are attempting to refer to some kind of underlying meaning or reality, and an LLM will not give you the understanding of the things being referred to - only the outward representations themselves. If you are indeed interested in meaningless exchange of symbols a la Searle's Chinese Room then perhaps there is some utility in this; otherwise it's the act of digesting and comprehending meaning, and the internal cognitive processes that go into the production of the written artifact that matter, not just the written artifact itself.