A-Z is the only option for the output of FROMNOW(). Yet FROMNOW() is categorized as a Date/Time Function, and the description in the Formula field reference says “Calculates the number of days between the current date and another date.”
Doesn’t it seem more normal that I would want to put things in order of when they’re next due?
@Elias_Gomez_Sainz has it right: If the result of a formula field is, in part, a string, the entire thing is a string — and the string '5 years' is less than the string '6 days'. That the only available sort options are 'A-Z'/'Z-A' is a dead giveaway the result is a string.
And Date/Time Functions return dates, numbers, or strings, depending on the function.
Frankly, I’d suspect it is a rare to-do list that commingles items due in days and years.
One possibility would be to create a hidden field with DATETIME_DIFF(), sort FROMNOW(), and group by bins of days: < 30 (or is it < 7; does FROMNOW() ever return in weeks?), < 365, and > 365).