Ha, I never noticed that before! It is not possible to turn that behavior off. It would be nice if it didn’t actually create the new row until we started typing data. You may want to add this as a feature request in the #show-and-tell:product-suggestions topic.
Yes, it is annoying that clicking anywhere on the entire bar creates a new row, not just clicking the +. For me, it is only a problem when I have grouped records.
It is possible to turn off the ability to create a new row in a view by grouping by a formula field, instead of an input field. Unfortunately, the only way to create a new record in such view is by duplicating a record, which has its problems.