You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This discrepancy is could be due to floating point imprecision
Yes. Numbat is a pure numerical calculator at the moment. Internally, it uses 64 bit floating point numbers, and 10^100 is outside the range of exactly representable integers (https://stackoverflow.com/a/1848762). So this is expected behavior, but see #4
When performing arithmetic with very large numbers in numbat, the order of operations appears to affect the result:
Mathematically, both expressions should yield 1. This discrepancy is could be due to floating point imprecision.
I'm using version 1.16.0.
The text was updated successfully, but these errors were encountered: