Re: rounding problems

On Wed, May 14, 2008 at 11:47:52AM -0400, Justin wrote:
> I have forgotten how much i hate C++

What we're talking about doesn't have much to do with C++, it's floating
point maths in general.

> Its not doing what you say it would but it did do other odd ball 
> things.  I miss my foxpro :-(. 

What does foxpro use for storing numbers? or is it just that you never
pushed it hard enough for the abstractions to show through.

> Plus its not holding 15 precision points

after changing the output to be:

  printf("%.10f %.10f\n", d, d-c);

I get:

  100000000.0999999940 0.0999999940
  100000000.0100000054 0.0100000054
  100000000.0010000020 0.0010000020
  100000000.0001000017 0.0001000017
  100000000.0000099987 0.0000099987
  100000000.0000009984 0.0000009984
  100000000.0000001043 0.0000001043
  100000000.0000000149 0.0000000149
  100000000.0000000000 0.0000000000

Which looks reasonable.  Remember that floating point numbers store
their state in base two, not base ten.  All of those numbers look good
to 15 decimal digits.


  Sam

-- 
Sent via pgsql-general mailing list (pgsql-general@[EMAIL PROTECTED]
)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general