You should be able to use the tab key to jump over to the time. However, it is still frustrating as it doesn’t highlight the entire time when tabbing over for you to replace the value. Instead, it puts the cursor at the end of the time so that you have to backspace and/or highlight the existing time before replacing it with something else.
