Keystroke not simulated after many loops

Questions & Suggestions
, ,
1

I want keyboard maestro to loop approximately 10000 times through a list of URLs in excel. Each link is loaded in Safari and returns a table with information (which sometimes might be empty). Up to this point, everything works as expected, however, the macro skips certain keystrokes after a certain amount of repetitions. In this case, after it copies the text from the browser, the macro should go over to another sheet in Excel an paste the information into the next empty row (CMD + Down Arrow). This step is on that is skipped, because it pastes the information on top of another cell already containing previous data.

This happens in my case after the 2104th URL and the respective 18406th pasted dataset (other sheet). I ran this macro on another computer with a higher CPU and more RAM and somehow the result is the same. It somehow start skipping at 18000-20000 datasets. Each URL may return many datasets, which is why the other sheet contains more datasets than actual URLs.

I added certain timed pauses in between each action. Is there a way to wait for each action to be simulated? There is an option to pause until the keystroke or key combination is pressed, but in this case it is not pressed but rather simulated.

Maybe use another spreadsheet application which is more lightweight than Excel? Any recommendation?

I want this macro to be as fast as possible given that I have to run sometimes even more than 10000 perhaps 100000 loops.

It is important to note that Keyboard Maestro increases the CPU% usage. I deduce that this might be due to the loops accumulating. Each loop should not really have too much CPU% usage, hence, how can this macro be built more efficiently?

This is the macro:
Parser.kmmacros (15.9 KB)

Thanks,
Anthony

2

It’s hard to say for sure, and I doubt anyone can accurately duplicate your conditions.

Keep in mind that the event queue is just that, a queue of events. Keys, when simulated, are added to that queue. And the system will pluck them off when it feels like it. And that queue is finite in length, so if it overflows, keys will be lost.

It is, of course, always possible that there is some leak or other issue in Keyboard Maestro (or Excel or your browser or whatever) that doesn’t show up until the 10,000th operation, it would be very hard to tract down such a thing.

If Keyboard Maestro’s CPU is going up over time, then maybe get an Activity Monitor Sample of the Keyboard Maestro Engine to see where it is spending its time.

One technique I have used for situations like this is to periodically insert a larger delay to let the system fully complete everything outstanding. So maybe every 20 or 50 URLs, add a 60 second pause to ensure there is no accumulation of delays.

3

Thanks for your extensive answer @peternlewis. Wouldn’t adding a 60 second pause lead to an excess in time? Keep in mind that there are 10000 URLs and doing so will keep my macro running for days and perhaps even for a week.

4

The point is to allow the system to catch up and catch its breath. How long and how frequent the pause is would depend on how serious the backlog is and whether that is even the issue.

If the issue turns out to be another application getting warn out (or even Keyboard Maestro getting warn out), then part of the automation process will need to deal with that, perhaps by quitting and relaunching applications periodically.

The thing with automation is the make them as far as possible, but no faster.