Even though you are rounding the display of A and B in their fields, Airtable still holds the full float in the background - down to the greatest level of specificity the float originally went to.
For better or for worse (probably for better), when you perform calculations via formulas on a number field that you have rounded in the display, the formula acts on the float behind the display, not the displayed/rounded version.
So, in your (A - B) formula, try rounding the end result of the calculation like this: ROUND(A - B).
If that still displays a negative 0, try rounding each number involved in the calculation like this: ROUND(A) - ROUND(B).