Number Precision Comparison Calculator
Compare a number rounded to different levels of precision and see the absolute error, relative error and how it is stored as a 32-bit float.
Compare Your Options And Get Clear Answers
Compare a number rounded to different levels of precision and see the absolute error, relative error and how it is stored as a 32-bit float.
Number Precision Comparison Estimator
Enter your numbers below for an instant estimate.
How It Works
Key concepts behind this calculator
Round
Round the value to each number of decimal places.
Absolute Error
Original minus rounded value.
Relative Error
Absolute error รท original, as a percentage.
Use As A Guide
Results are estimates based on your inputs. Check real quotes, rates and professional advice before deciding.
Number Precision Guide
Approximate precision of number formats.
| Format | Bits | Significant Digits | Typical Use |
|---|---|---|---|
| Half float | 16 | 3 โ 4 | Graphics, ML |
| Single float | 32 | 7 | Games, sensors |
| Double float | 64 | 15 โ 16 | Most computing |
| Decimal | 128 | 28 โ 29 | Money, finance |
| Rule of Thumb | Double is ~16 digits | 0.1 is not exact | Round only at the end |
Number Precision Comparison Calculator FAQ
Answers to common questions about this calculator.
Computers store most decimals in binary, which cannot represent them exactly, so tiny rounding errors appear.
A digit that carries meaningful precision, starting from the first non-zero digit.
Match the precision of your measurements and avoid reporting more digits than you can justify.
The size of the error as a fraction of the true value.