The issue you are seeing is because you are looking for a state (key is down) but you really need the event (key is pressed), because the key only stays down a short time and if you do not check it while in the short time the key is down you will miss it. See:
That has the same solution you need to reliably detect pressing a key (as opposed to detecting a key being down at a specific time, which is what you are doing).