5   Numbers, Precision, and the Display

A calculator can retain many more digits than it shows. In SpicyRPN, working precision determines the precision available to new calculations, while the display setting determines how a value is presented. Neither setting establishes the accuracy of the numbers you supplied. Keeping these three ideas separate helps you read results, preserve useful intermediate values, and recognize when more digits would help.

The precision and display settings are shared by RPN and ALG. Their entry procedures differ in one important place: FIX takes a count from the stack in RPN, but opens a separate count prompt in ALG. This chapter covers both procedures.

5.1   Three meanings of precision

Working precision describes the numerical resolution used to create a value or calculate a result. SpicyRPN starts with 600 decimal working digits; PREC allows 16 through 10 000. Internally, the calculator uses binary floating-point values with enough bits for the selected decimal precision. For example, 600 decimal working digits correspond to 1 994 bits. This is a significant-digit precision, not a fixed number of decimal places: very large and very small numbers can use the same working precision.

Finite binary precision cannot represent every decimal or fraction exactly. Integers that fit the available precision, and fractions such as \(1/8=0.125\), can be exact. A decimal such as \(0.1\), a repeating fraction such as \(1/3\), and an irrational number such as \(\pi \) require approximations. Individual MPFR real-number operations use rounding to nearest, with an exact halfway case rounded to the even choice. A calculation made from several operations can accumulate or amplify their rounding errors.

Display precision controls the number of places shown after the decimal point, either in the ordinary number or in the coefficient of a power of ten. The initial FIX 4 normally shows four decimal places. A short readout does not discard the remaining digits of the value. Likewise, requesting a long readout does not prove that all its digits are accurate.

Input accuracy comes from the source of the number. A length reported as \(12.3\) cm may be a rounded measurement; the calculator cannot discover the omitted measurement detail. An exact count of 123 objects is a different kind of input. Entering extra zeros, increasing working precision, or displaying more digits cannot turn an uncertain measurement into an exact one. SpicyRPN does not infer measurement uncertainty from the number of digits you type.

5.2   Setting working and display digits with PREC

With no pending command or active shift, press g 7, the blue PREC function. The Precision & display panel shows the current settings in two typed fields:

If you lower Working digits, also choose a Display digits count no larger than that new setting. For example, working 80 with display 81 is rejected. Raising the working count alone does not automatically raise the display count. The panel retains the selected notation, so applying new counts while ENG is showing leaves ENG selected.

Replace the field contents with the desired counts, then press Apply. The panel closes after a successful change. An invalid count leaves it open with a message so that you can correct the fields; it does not partly apply the new settings. Press Close to leave without applying the edited counts. The fields are settings, not operands: typing in them does not put numbers on the RPN stack or into an ALG expression.

Opening or applying PREC does not finish a pending numeric entry or evaluate an unfinished ALG expression. The characters already typed remain available for editing. A numeric entry that is still open will be converted to a stored number at the working precision in effect when you finish it. In an ALG expression, numbers already finished before the change retain their earlier precision; changing the setting does not re-create them from their original text.

The INFIX label

The calculator’s help uses INFIX as a name for working precision, including the upper bound for a display-digit count. In this usage, “current INFIX precision” means the value of Working digits in PREC. It does not select or identify algebraic entry. The entry-mode buttons above the display say RPN and ALG; those are the controls to consult when choosing a key sequence. Working precision applies in either mode.

5.3   Fixed, scientific, and engineering notation

The display’s notation flag combines a format and a count, such as FIX 4, SCI 4, or ENG 4. The count means places after the decimal point in all three formats. It does not mean the same total number of significant digits in each format. The ALG expression line has its own compact four-decimal presentation, described in section 4.1; changing the main display’s count does not lengthen the numbers in that line.

With a count of zero, the coefficient has no fractional part or decimal point. SCI still has one significant digit, while ENG can have one, two, or three. Zero itself is shown as an ordinary zero in FIX and with exponent zero in SCI or ENG, with the requested fractional zeros.

Press f 8, the orange FSE function, to cycle FIX to SCI to ENG and back to FIX. Each press sequence advances one step and retains the count. It does not calculate, move stack values, finish an entry, or evaluate an expression. While you are typing a number, the display continues to show the entry as typed; the selected format applies to the completed value.

Why FIX sometimes shows an exponent

FIX is a preference for ordinary decimal notation, not a requirement to show every number that way. If the integer portion, sign, grouping, and decimal boundary cannot fit the main display’s width, the display uses scientific notation. Rounding can itself cause the integer portion to grow: a value close to 999.999 may round up to 1,000.00. Resizing a window can therefore change whether a particular value uses an exponent.

FIX also uses scientific notation when ordinary rounding would turn a nonzero value into displayed zero. For example, \(0.000001\) under FIX 2 appears as \(1.00\times 10^{-6}\), not 0.00. An actual zero still displays as 0.00. This helps distinguish a small retained result from zero, though it cannot recover a value that arithmetic has already rounded to zero.

The flag remains FIX with its selected count during fallback. The scientific coefficient uses that many fractional places. Long fractional tails remain available by scrolling; SpicyRPN does not reduce the requested number of decimal places merely to squeeze them onto one screen. A secondary line or expanded-stack row can have a different width from the main display, so judge the value together with its exponent rather than expecting identical line breaks or notation in every location.

5.4   Setting FIX in each entry mode

FIX is the orange function of 7. It both selects FIX notation and sets its decimal-place count. The accepted count is a whole number from zero through the current working precision. To change the count while keeping SCI or ENG selected, use PREC; alternatively, set FIX and then cycle with FSE.

RPN: enter the count first

In the RPN stack, X is S0, the displayed operand, and Y is S1, the next position. Entering the count follows the normal stack-lift rule. On a full fixed stack, that lift discards the deepest value; the subsequent drop repeats the new deepest value. Thus the sequence returns the previous X to the display, but does not necessarily restore every deeper register. Use the PREC panel when you want to change display digits without using a stack position. FIX itself leaves the separately saved Last X and Last Y operands unchanged.

In RPN, enter the count and then press f 7. For example, 6 f 7 selects FIX 6. There is no digit prompt after the FIX key. It reads the count already in X, finishes that entry if necessary, and drops it. The preceding Y becomes the displayed X. You do not need ENTER between a freshly typed count and FIX.

An invalid count, such as 1.5, a negative integer, or a number above the working precision, reports an error and leaves the setting and operand intact. If the count was still being entered, its entry remains editable. Correct the count before trying FIX again.

ALG: select FIX, enter the count, then accept it

In ALG, press f 7 first. The \(\Delta \)FIX flag and a status prompt identify a separate digit-count entry. Type the count, then press =. The count does not enter the expression, and this = accepts the display setting without evaluating the expression. Both an unfinished number and the rest of the expression resume exactly where you left them.

Only digits and the accepting = belong to this prompt. Any other key cancels it and is consumed as cancellation; it does not also perform that key’s normal function. In particular, backspace cancels the prompt rather than deleting one count digit. To correct a mistyped count, cancel, press f 7 again, and enter the count afresh. A mode-button press also cancels without switching modes; press the desired mode button again afterward.

An empty count or one above the current working precision is rejected when you press =. The prompt ends, the previous display setting remains, and the suspended calculation is unchanged. Leading zeros in a valid count are allowed.

5.5   Reading long numbers

When a number is wider than its display region, drag the digits horizontally. Real RPN and both ALG displays use larger digits and an overflow-only ruler without scrollbars. This also applies to the real RPN full-stack view. Complex RPN offers a scrollbar beneath each component. Moving the digits left reveals later digits; moving them right returns toward the beginning. Scrolling changes only the view. It neither changes the stored value nor rounds it again. A new completed result normally starts at its leading digits.

In the main complex display, a completed result that changes either component returns both rows to their leading digits. This includes a change to the imaginary component alone. The rows remain independently scrollable afterward.

In scientific or engineering notation, the coefficient scrolls while the \(\times 10\) and raised exponent stay pinned at the right. Always read that exponent with the visible digits. A portion such as 23456 seen midway through a long coefficient is not a complete number on its own. During exponent entry, the display preserves the sign and digits you are typing, including an unfinished exponent; pinning does not complete it for you.

The CAL control in the header appears when a number overflows. It shows or hides a digit ruler without consuming a pending shift or command. If CANCEL is also showing, it stays in its fixed position beside the branding. The ruler marks the decimal point with 0 and then counts digits in groups of five outward on either side: 5, 10, 15, and so on. For an integer without a visible decimal point, the zero position is immediately after its last digit. Signs, commas, and exponent digits do not count as coefficient digits. The ruler moves with the coefficient, so its marks stay aligned as you scroll. It is a counting aid, not a statement about how many digits are trustworthy.

Each row of the expanded RPN stack has its own horizontal inspection area. Vertical movement selects deeper registers, while horizontal movement reveals more digits within a row; see section 2.6. In complex mode the two component rows have separate scrolling regions and pinned exponents. Polar angles also retain their angle-unit suffix. The meaning of those components is covered in chapter 8.

5.6   Display rounding and value rounding

FIX, SCI, and ENG round a presentation of the stored value. They do not replace that value with the displayed decimal. Display rounding uses nearest, ties to even: if a value is exactly halfway between two display choices, the last retained digit is even. For example, exact 0.125 under FIX 2 displays as 0.12, while exact 0.375 displays as 0.38. Under FIX 0, 2.5 displays as 2 and 3.5 as 4.

Those examples use numbers exactly representable in binary. A decimal that merely looks like a tie may be stored just above or below the mathematical halfway point because of input or earlier arithmetic rounding. Judge the actual stored value, not only the visible prefix of its decimal expansion.

RND deliberately creates a rounded value. It is the blue function of PMT on the Financial panel and is available in both entry modes. It rounds the real component to the current display count of decimal places, using halfway cases away from zero. In RPN it replaces the current X (S0) and saves that operand’s unrounded value separately in Last X. The current Y (S1), deeper stack positions, and the saved Last Y operand are unchanged. It does not overwrite a financial or ordinary memory register merely because a value originally came from one.

The count used by RND is a number of decimal places in the value itself, even when SCI or ENG is selected. For example, SCI 2 displays 1234.567 as \(1.23\times 10^3\), but RND at that count rounds the value to 1234.57, not 1230. Choose FIX before financial rounding when you want the readout to make the decimal-place convention clear.

In ALG, first finish a compound expression with =, or enter a single number. Press g PMT to round that value and make it the displayed result and ANS. An unfinished compound expression is preserved with the message Finish the expression first; RND does not implicitly evaluate it. After a successful rounding, digits start a new expression and an operator continues from the rounded result. The saved RPN workspace is untouched.

NINT, the blue function of 9, changes a value by rounding it to the nearest integer, with ties to even. It is available in both entry modes. Thus NINT sends 2.5 to 2, 3.5 to 4, and \(-2.5\) to \(-2\); its integer rounding is independent of the display count. Other integer operations use different rules: INT truncates toward zero, floor rounds downward, and ceiling rounds upward. Their key sequences and domains are in chapter 7; financial rounding is also discussed in chapter 13.

5.7   Existing values and a new working precision

Every stored number retains the precision at which it was created. Changing Working digits does not rewrite the RPN stack, ordinary memories, financial or statistical registers, the previous ALG answer, or numbers already present in an ALG expression. Recalling an older value brings back that value, with its existing precision. Increasing the display count may reveal more of its binary approximation, but cannot supply digits lost when it was created.

A new operation uses the current working precision for its result, even if its operands were created at other precisions. Raising the setting therefore allows more precise subsequent arithmetic, but the new result still depends on the information in those operands. Adding zero to a low-precision approximation at a higher setting does not restore the intended exact number. Lowering the setting likewise leaves old values intact, while new results may retain less detail.

When higher precision matters, select it before entering the original data and performing the calculation. Re-enter approximate decimals from their source, generate constants again, and repeat the necessary operations. Reusing a previously rounded intermediate result is not the same as recomputing that intermediate result at the new precision.

This example illustrates cancellation: the leading digits of nearly equal quantities disappear on subtraction, exposing the accuracy of their much smaller difference. The same issue can arise inside a formula even when its final answer has an ordinary size.

5.8   Choosing useful precision and recognizing limits

The default 600 working digits provide considerable room for ordinary arithmetic. You usually need only a small display count for a price, measurement, or quick comparison. Choose the display for the task: FIX for a familiar decimal scale, SCI for orders of magnitude, or ENG when exponents in groups of three are convenient. Keep working precision comfortably above the answer digits you need, especially when intermediate values differ greatly in size.

For a demanding calculation, repeat it from the original inputs at a higher working precision and compare the digits you intend to use. Agreement is a useful check, but not a proof of correctness: an incorrect formula, an uncertain input, or a numerical limitation can persist at several settings. A more stable mathematical form can be more effective than simply asking for more digits. More precision can also require more time and memory, particularly for long formulas or complex special functions.

Magnitude, domains, and practical bounds

Working precision and exponent range are separate limits. SpicyRPN can represent finite values such as \(10^{5000}\) and \(10^{-5000}\); the formatter also handles powers such as \(10^{100000}\) without expanding them into an enormous line of integer zeros. A setting of 600 digits does not mean that values must lie between \(10^{-600}\) and \(10^{600}\). The underlying exponent range is nevertheless finite. Increasing precision is not a general cure for overflow or underflow.

Other limits belong to particular controls or functions:

• PREC accepts 16–10 000 working digits, with the display count from zero through that working count. These are the calculator’s exposed setting bounds.
• Real-mode factorial accepts nonnegative integers through 10 000. Permutations and combinations require integers with \(0\leq r\leq n\leq 10\,000\). More working digits do not enlarge those argument ranges.
• For a nonreal argument, complex erf and erfc have an approximate magnitude limit of 100 and may report a failure to converge. Their real-argument path has different numerical behavior; see section 8.9 and chapter C.
• ALG limits a streaming numeric literal to 10 000 characters and an expression to 4 096 tokens and 128 nested groups. These are entry limits, not a promise of answer accuracy.

A mathematically invalid real operation, such as the square root of a negative number in real mode, is not fixed by increasing working precision. Some RPN arithmetic can display \(\infty \), \(-\infty \), or NaN rather than a finite result; NaN means “not a number.” ALG rejects nonfinite values during expression evaluation. Other commands issue specific domain or convergence messages. Read the status area and check the operands and selected mode before continuing.

Complex arithmetic and elementary functions use MPC, which rounds each component of an individual primitive at the precision requested by the calculator. A composed calculation, angle conversion, or custom special-function approximation does not inherit a blanket guarantee of correct rounding for its complete result. Such algorithms can still lose digits through cancellation or fail to converge; a convergence message does not mean that the mathematical function is undefined. Chapter 8 and chapter C discusses complex conventions and numerical interpretation.

When an answer surprises you, first check the displayed exponent, notation, entry mode, angle unit, and complex setting. Next check the operands and any reused intermediate values. Only then decide whether to repeat the calculation at a higher working precision. The diagnosis and recovery procedures in chapter 14 complement these numerical checks.