Skip to main content

Incorrect calculations

  • January 14, 2020
  • 5 replies
  • 48 views

Forum|alt.badge.img+11

I have a width entry and a height entry entered in columns

W cm [defined as a number field with precision 2 decimal places]
H cm [defined as a number field with precision 2 decimal places]

I have a formula to bring these together into one field:
{W cm} & " x " & {H cm}

Result is incorrect because …
19.68 25.08
Becomes
19.685 x 25.0825

Can anybody tell me why this is happening please?

5 replies

Forum|alt.badge.img+19
  • Inspiring
  • January 14, 2020

The true value of {H cm} is actually 25.0825, but it’s showing a display value of 25.08.

There’s several approaches to solving this. If {H cm} is a formula, I’d recommend using the ROUND() function within that field so that your summary bars don’t get thrown off (though, this might not matter if you don’t use them).

If {H cm} is not a formula, use the ROUND() function in the formula you’re asking about.

ROUND({W cm}, 2) & " x " & ROUND({H cm}, 2)

Hope this helps!


Forum|alt.badge.img+11
  • Author
  • Known Participant
  • January 14, 2020

I am confused as to why a number field becomes a different number in a function call field?
{W cm} is a number field with 2 decimal places as is {H cm}. Why is the function field mangling the numbers?


Forum|alt.badge.img+11
  • Author
  • Known Participant
  • January 14, 2020

I am confused as to why a number field becomes a different number in a function call field?
{W cm} is a number field with 2 decimal places as is {H cm}. Why is the function field mangling the numbers?


By the way … it’s not happening consistently


Forum|alt.badge.img+19
  • Inspiring
  • January 19, 2020

I am confused as to why a number field becomes a different number in a function call field?
{W cm} is a number field with 2 decimal places as is {H cm}. Why is the function field mangling the numbers?


Hi there - sorry for the delay, I wasn’t notified of your reply to this post.

The function is not mangling the numbers, math is :slightly_smiling_face:

Can you give more information on how your getting to the numbers in question? Are they manually entered into a Number field? Or are they calculated via a formula? If it’s a formula, can you share the formula?


Forum|alt.badge.img+19
  • Inspiring
  • January 19, 2020

I am confused as to why a number field becomes a different number in a function call field?
{W cm} is a number field with 2 decimal places as is {H cm}. Why is the function field mangling the numbers?


Yep - extremely annoying and I think this is the infamous floating point bug which was first documented in 2017 and as far as I know, remains until this day.