Resolution uncertainty calculator
Turn an instrument’s resolution into a GUM standard uncertainty, optionally combine it with your repeatability, and read off the expanded uncertainty at k = 2. A quick companion to a full uncertainty budget. Everything runs in your browser.
Your numbers
The smallest increment the instrument displays (one count), e.g. 0.01 if the last digit steps by 0.01.
The specification half-width (±), same units. Used only to judge whether the resolution is fine enough.
Result
Resolution uncertainty, in plain terms
Why resolution adds uncertainty
A display rounds. If a meter reads 10.00 V to a resolution of 0.01 V, the true value could be anywhere from 9.995 to 10.005 V and still show 10.00. You have no way to know where inside that one-count window the truth lies, and that limit on what the instrument can tell you is a genuine uncertainty component every bit as real as drift or repeatability. It does not vanish just because the number on the screen looks exact.
The rectangular distribution and 2√3
Since any value within that window is equally consistent with the reading, the GUM (JCGM 100) models it as a rectangular, or uniform, distribution one resolution wide. The half-width is resolution / 2. The standard uncertainty of a rectangular distribution is its half-width divided by √3, so combining the two gives u_res = (resolution / 2) / √3 = resolution / (2√3), about 0.2887 times the resolution. That single factor, 2√3 ≈ 3.464, is all that turns a least significant digit into a standard uncertainty.
Combining with repeatability
Repeatability is a Type A component: take several readings on a stable artifact and compute the experimental standard deviation s. If each reported result is a single reading, the repeatability standard uncertainty is s. If you average n repeats and report the mean, it is s / √n, the standard uncertainty of the mean. Because the resolution and repeatability components are independent, they combine in quadrature: u_c = √(u_res² + u_rep²). Multiply by the coverage factor k = 2 for the expanded uncertainty U at roughly 95% coverage.
This is an honest component estimate, not a complete budget. A real measurement also carries the reference standard’s uncertainty, drift and stability, environmental effects, and more. For a fuller worked budget see the thermocouple & RTD uncertainty calculator; the combination rule used here is the same one you would apply to every line of that budget.
Common questions
-
Why does instrument resolution add uncertainty?
A digital display can only resolve to its least significant digit, so a true value anywhere within half a count of the shown reading rounds to the same number. You cannot tell where inside that window the true value sits, and that ignorance is a real component of measurement uncertainty. The GUM treats it as a rectangular (uniform) distribution one resolution wide, centred on the reading.
-
Why divide the resolution by 2√3?
The resolution defines a window one full count wide, so its half-width is resolution / 2. For a rectangular distribution the standard uncertainty is the half-width divided by √3. Combining the two gives u_res = (resolution / 2) / √3 = resolution / (2√3) ≈ 0.2887 × resolution. The √3 is the standard divisor the GUM assigns to a uniform distribution.
-
Should I use a single reading or the mean of repeats for repeatability?
Use the experimental standard deviation s when each result is a single reading. When you average n repeat readings and report the mean, the relevant figure is the standard uncertainty of the mean, s / √n, which is smaller. This tool lets you pick whichever matches how the value is actually reported, so the repeatability term is consistent with your method.
-
How is the resolution uncertainty combined with repeatability?
The two are independent components, so they combine in quadrature (root-sum-square): u_c = √(u_res² + u_rep²). The expanded uncertainty is U = k × u_c with a coverage factor of k = 2, which corresponds to roughly 95% coverage for an approximately normal result. A full budget would add more terms, but the combination rule is the same.
-
Is the resolution fine enough for my tolerance?
A common rule of thumb is that resolution should be no larger than about 5 to 10% of the tolerance you are judging against. If you enter a tolerance, the tool compares the resolution to it and flags whether the display is fine enough, marginal, or too coarse. It is guidance, not a hard requirement; your own method and decision rule have the final say.
This calculator estimates two GUM (JCGM 100) uncertainty components and combines them; it is not a complete uncertainty budget and does not replace the GUM, ISO/IEC 17025, or your laboratory’s own procedures. A full evaluation includes every significant contributor for your specific measurement. Review any figure against your quality system before you report it.
Keep the uncertainty with the record
Axiospec keeps every calibration, uncertainty, and certificate in one tamper-evident ledger built for ISO/IEC 17025. See it on real calibration data, no signup.