Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inconsistent arithmetic with large numbers (possible floating point imprecision) #690

Closed
apcamargo opened this issue Feb 16, 2025 · 1 comment

Comments

@apcamargo
Copy link

When performing arithmetic with very large numbers in numbat, the order of operations appears to affect the result:

>>> let big_number = 10^100

  let big_number: Scalar = 10^100

>>> big_number + 1 - big_number

  (big_number + 1) - big_number

    = 0

>>> big_number - big_number + 1

  (big_number - big_number) + 1

    = 1

Mathematically, both expressions should yield 1. This discrepancy is could be due to floating point imprecision.

I'm using version 1.16.0.

@sharkdp
Copy link
Owner

sharkdp commented Feb 16, 2025

This discrepancy is could be due to floating point imprecision

Yes. Numbat is a pure numerical calculator at the moment. Internally, it uses 64 bit floating point numbers, and 10^100 is outside the range of exactly representable integers (https://stackoverflow.com/a/1848762). So this is expected behavior, but see #4

@sharkdp sharkdp closed this as completed Feb 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants