Context I'm implementing a Julia set fractal renderer using CubeCL (a Rust GPU compute framework). I want to achieve "infinite zoom" similar to deep Mandelbrot zoom videos, which requires ...

TL;DR: In Emacs Lisp, is the sign of a fixnum part of its binary representation or stored in some metadata? What's the distinction between positive and negative fixnums? Can the sign be switched in ...

I am writing a program called moments++ (https://github.com/gvbarroso/momentspp). This program uses Eigen 3.4. Some scenarios lead to precision issues in the matrices (using long double). Thus, I want ...

I have the following code where I try to use fmpz_t to prime factor rationales of a system constant. /* gcc -I/usr/local/include/flint -o arb arb.c -lflint */ # include "arb.h" ...

Fiddle select 5.0 / 2 , pg_typeof(5.0 / 2); pg_typeof 2.5000000000000000 numeric Why the result is 2.5000000000000000? When we insert 2.5 into a numeric column or select 2.5 from a numeric column ...

Boost multiprecision libraries highest value possible Can the boost multiprecision handle values as large as googolplex , like maybe the boost multiprecision cpp dec float library? So can someone ...

I have a recursive computation code of mine (containing only simple +, *, exp, etc arithmetic on complex valuated numbers) that starts to loose precision when increasing the depth of iterations and I ...

I need to store some really tiny positive and negative numbers in Matlab. One way is to store their signs separately, and the log10 of their magnitude, which lets me store numbers as small as 10^(-...

I'm currently writing an arbitrary precision library for fun (yes, I know that GMP exists) and I'm using it to generate large prime numbers (~5000 digits) with the Miller-Rabin primality test. I ...

edited because my main issue lies with incorrect results from mp.math. I need to eventually do very precise calculations with complex numbers, which is why I am trying to use the mp module from mp....

Regarding arbitrary precision arithmetic: Can AVX-512 be used to calculate the multiplication of two 256-bit Integers on a 64-bit computer in C++? Is there an intrinsic Integer data type of 512 bits ...

I have a constant 0.29289321881345247559915563789515..., which can be calculated using the equation (1 - SQRT(0.5)) and then transformed into fixed-point format in order to be used with various sizes ...

i am using java big decimal. but still i am not getting expected result.. public class DecimalAdjustEx { public static void main(String[] args) { BigDecimal v1 = new BigDecimal("100&...

I want to find powers of a relatively small matrix, but this matrix consists of rational numbers of type Rational{BigInt}. By default, Julia utilizes only a single thread for such computations. I want ...

For a statistics application, I need to evaluate tiny chi-square probabilities. But when this becomes smaller than matlab's double-precision limit realmin (around 1e-308), it returns 0. x=50; p=...