I have this macro to grab the contents of a (licensed) Windows app that I want to use in a more convenient way. It is absolutely amazing how few commands are needed to perform the advanced task. The strange thing is that the macros perform well if I let it run for, say, 2000 reps. However, when I let it run for 5000 reps, the output ends with:
It could be a memory leak or a cache of some sort filling up. The system has lots of things that are not quite unlimited that are cleared out periodically, and that can cause problems if they are used faster than they are cleared. And they aren't really designed with the assumption that you will be doing things continuously for very long periods of time without breaks.
Sometimes simply adding a large pause periodically in your macro (say every 1000 times, pause for five minutes) to allow the system to “catch its breath”, clean things out and flush things back to a normal state can resolve such issues. Other times more drastic measures might be required, such as relaunching applications (potentially including Keyboard Maestro) or restarting.
It probably isn't that. I ran a test yesterday that blitzed 10,000 iterations of up-arrowing/OCRing/appending using a Terminal window, and KM didn't miss a beat.
I then realised that my single short line was (probably) way less data to process than your OCR grab, so I increased the area to the whole Terminal window. It hadn't finished processing by the time I had to leave for work this morning, but there does seem to be a memory leak or similar from watching Activity Monitor (around 1.5GB "missing").
If I can find out from the output file when that started happening I can then try some mitigation testing, starting with longer pauses in the loop and @peternlewis's "periodic pause" suggestion.