Talk About Network

Google


Register and Login
Nick
Password
Register create new account Sign up is FREE and you can post replies, new topics, bookmark posts and more!
Recover lost password


Data Bases > Pgsql General > Re: rounding pr...
Latest [ Topics | Posts ] Archive Post A New Topic Post a Reply
<< Topic < Post Post 33 of 36 Topic 15511 of 17437
Post > Topic >>

Re: rounding problems

by justin@[EMAIL PROTECTED] (Justin) May 20, 2008 at 10:28 AM

This is a multi-part message in MIME format.
--------------020904030503040302000306
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit



glene77is wrote:
> On May 14, 3:27 pm, s...@[EMAIL PROTECTED]
 (Sam Mason) wrote:
>   
>> On Wed, May 14, 2008 at 02:08:47PM -0400, Justin wrote:
>>     
>>> Sam Mason wrote:
>>>       
>>>> What doesfoxprouse for storing numbers? or is it just that you never
>>>> pushed it hard enough for the abstractions to show through.
>>>>         
>>> I know i pushed it.  Foxpro for the most has only  4 basic data types
>>> Numeric (similar to Posgresql numeric), Boolean, Date, Text aka
>>> (string)  Thefoxprotables sup****ted far more data types but when every
>>> it was dumped to variable it acted like one of the 4.
>>>       
>> I really meant how much did you check the results, or did you accept
>> that they were correct?
>>
>>     
>>> Foxprodid not suffer floating point math errors.  I normally used 8 to
>>> 10 points precision.  Foxprowas limited to 15 points of precision
>>> period.   No more and no less, once you hit that was it.
>>>       
>> 15 places seems very similar to what a 64bit IEEE floating point number
>> will give you, i.e. a double in C/C++.
>>
>>     
>>> My problem is we calculate resistance of parts in aFoxproapp that we
>>> want to move because we want to bring all the custom apps into one
>>> framework and single database.
>>>       
>>> Take this calculation  (0.05/30000* 1.0025) which is used to calculate
>>> parts resistance and Tolerance. (its Ohms Law)  The value returned 
from
>>> C++ = .0000016708 which is wrong
>>> it should be .00000167418.  We just shrank the tolerance on the part
we
>>> make
>>>       
>> Why are you so sure about theFoxProresult?  I've just checked a few
>> calculators and get results consistent with your C++ version.
>>
>>   Justin C: 0.0000016708
>>   JFoxPro: 0.00000167418
>>       My C: 0.000001670833
>>      bc[1]: 0.0000016708333333333333333333333333333332
>>      PG[2]: 0.0000016708333333333333336675
>>  Google[3]: 0.00000167083333 (actually gives 1.67083333e-6)
>>
>> Both bc and Postgres use their own code (i.e. not the CPU's FPU) to do
>> the math, and as they all agree I'm thinkingFoxProis incorrect!  Next
>> I tried doing it accurately (in Haskell if it makes any difference) and
>> get an answer of 401/240000000 out, which would agree with everything
>> butFoxPro.  If I calculate the ratio back out forFoxProI get
>> 401/239520242 which is a little way out.
>>
>>     
>>> The Do***entation from MS says 15 points of precision but the result
say
>>> otherwise.
>>>       
>> The docs for what?FoxProor their C compiler?
>>
>> If you meanFoxPro, I think this is another case of MS screwing up.
>>
>>     
>>> I'm glad You and others are taking the time to explain to me
>>> the odd results before i get into redoing that application.
>>>       
>> Welcome to the PG community, lots of people to get interested in lots
of
>> things!
>>
>>     
>>> Why oh Why did MS killFoxpro. :'(   I understood it, knew its quirks
>>> and it worked very well with Postgresql
>>>       
>> Are you sure you want to stay with it if its answers are wrong?
>>
>>   Sam
>>     
>
>
*********************************************************************************
> This is fun, at 0400 AM.  I enjoy reading  Experts having serious fun!
>
> VFP 6.0, using my defaults
> ? (0.05/30000* 1.00250000000000000)
> displays  "0l.000001670833333000"
>
> SET DECIMALS TO 15
> ? ((0.05/30000)* 1.0025)
> displays "0.000001670833333"
>
> and a frivolous example:
> SET DECIMALS TO 18
> ? ((0.050000/30000.00000000)* 1.0025000000000000)
> displays "0.000001670833333000"
>   
Foxpro always stops at 15 decimals points,  Even though some of the 
do***entation says 20  and 22 points of precision depending on the 
version.  I have versions 5 to 9
> Anybody tried to reckon this math
> the way we used to do it with a Slide-Rule ???
> (In VFP of course)
>   
A slide what??.  I have never touched one or seen a slide rule in real 
life, just pretty pictures  :-)
> glene77is
>   





--------------020904030503040302000306
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
  <title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
<br>
<br>
glene77is wrote:
<blockquote

cite="mid:50a48f05-173a-42c6-88b8-00d6aadda1c4@[EMAIL PROTECTED]
"
 type="cite">
  <pre wrap="">On May 14, 3:27 pm, <a class="moz-txt-link-abbreviated"
href="mailto:s...@[EMAIL PROTECTED]
">s...@[EMAIL PROTECTED]
> (Sam Mason) wrote:
  </pre>
  <blockquote type="cite">
    <pre wrap="">On Wed, May 14, 2008 at 02:08:47PM -0400, Justin wrote:
    </pre>
    <blockquote type="cite">
      <pre wrap="">Sam Mason wrote:
      </pre>
      <blockquote type="cite">
        <pre wrap="">What doesfoxprouse for storing numbers? or is it just
that you never
pushed it hard enough for the abstractions to show through.
        </pre>
      </blockquote>
    </blockquote>
    <blockquote type="cite">
      <pre wrap="">I know i pushed it.  Foxpro for the most has only  4
basic data types
Numeric (similar to Posgresql numeric), Boolean, Date, Text aka
(string)  Thefoxprotables sup****ted far more data types but when every
it was dumped to variable it acted like one of the 4.
      </pre>
    </blockquote>
    <pre wrap="">I really meant how much did you check the results, or did
you accept
that they were correct?

    </pre>
    <blockquote type="cite">
      <pre wrap="">Foxprodid not suffer floating point math errors.  I
normally used 8 to
10 points precision.  Foxprowas limited to 15 points of precision
period.   No more and no less, once you hit that was it.
      </pre>
    </blockquote>
    <pre wrap="">15 places seems very similar to what a 64bit IEEE
floating point number
will give you, i.e. a double in C/C++.

    </pre>
    <blockquote type="cite">
      <pre wrap="">My problem is we calculate resistance of parts in
aFoxproapp that we
want to move because we want to bring all the custom apps into one
framework and single database.
      </pre>
    </blockquote>
    <blockquote type="cite">
      <pre wrap="">Take this calculation  (0.05/30000* 1.0025) which is
used to calculate
parts resistance and Tolerance. (its Ohms Law)  The value returned  from
C++ = .0000016708 which is wrong
it should be .00000167418.  We just shrank the tolerance on the part we
make
      </pre>
    </blockquote>
    <pre wrap="">Why are you so sure about theFoxProresult?  I've just
checked a few
calculators and get results consistent with your C++ version.

  Justin C: 0.0000016708
  JFoxPro: 0.00000167418
      My C: 0.000001670833
     bc[1]: 0.0000016708333333333333333333333333333332
     PG[2]: 0.0000016708333333333333336675
 Google[3]: 0.00000167083333 (actually gives 1.67083333e-6)

Both bc and Postgres use their own code (i.e. not the CPU's FPU) to do
the math, and as they all agree I'm thinkingFoxProis incorrect!  Next
I tried doing it accurately (in Haskell if it makes any difference) and
get an answer of 401/240000000 out, which would agree with everything
butFoxPro.  If I calculate the ratio back out forFoxProI get
401/239520242 which is a little way out.

    </pre>
    <blockquote type="cite">
      <pre wrap="">The Do***entation from MS says 15 points of precision
but the result say
otherwise.
      </pre>
    </blockquote>
    <pre wrap="">The docs for what?FoxProor their C compiler?

If you meanFoxPro, I think this is another case of MS screwing up.

    </pre>
    <blockquote type="cite">
      <pre wrap="">I'm glad You and others are taking the time to explain
to me
the odd results before i get into redoing that application.
      </pre>
    </blockquote>
    <pre wrap="">Welcome to the PG community, lots of people to get
interested in lots of
things!

    </pre>
    <blockquote type="cite">
      <pre wrap="">Why oh Why did MS killFoxpro. :'(   I understood it,
knew its quirks
and it worked very well with Postgresql
      </pre>
    </blockquote>
    <pre wrap="">Are you sure you want to stay with it if its answers are
wrong?

  Sam
    </pre>
  </blockquote>
  <pre wrap=""><!---->
*********************************************************************************
This is fun, at 0400 AM.  I enjoy reading  Experts having serious fun!

VFP 6.0, using my defaults
? (0.05/30000* 1.00250000000000000)
displays  "0l.000001670833333000"

SET DECIMALS TO 15
? ((0.05/30000)* 1.0025)
displays "0.000001670833333"

and a frivolous example:
SET DECIMALS TO 18
? ((0.050000/30000.00000000)* 1.0025000000000000)
displays "0.000001670833333000"
  </pre>
</blockquote>
Foxpro always stops at 15 decimals points,&nbsp; Even though some of the
do***entation says 20&nbsp; and 22 points of precision depending on the
version.&nbsp; I have versions 5 to 9<br>
<blockquote

cite="mid:50a48f05-173a-42c6-88b8-00d6aadda1c4@[EMAIL PROTECTED]
"
 type="cite">
  <pre wrap="">
Anybody tried to reckon this math
the way we used to do it with a Slide-Rule ???
(In VFP of course)
  </pre>
</blockquote>
A slide what??.&nbsp; I have never touched one or seen a slide rule in
real
life, just pretty pictures&nbsp; :-) <br>
<blockquote

cite="mid:50a48f05-173a-42c6-88b8-00d6aadda1c4@[EMAIL PROTECTED]
"
 type="cite">
  <pre wrap="">
glene77is
  </pre>
</blockquote>
<br>
<br>
<br>
<br>
</body>
</html>

--------------020904030503040302000306--
 




 36 Posts in Topic:
rounding problems
justin@[EMAIL PROTECTED]   2008-05-12 13:48:16 
Re: rounding problems
lyeoh@[EMAIL PROTECTED]   2008-05-13 02:07:09 
Re: rounding problems
justin@[EMAIL PROTECTED]   2008-05-12 14:54:47 
Re: rounding problems
justin@[EMAIL PROTECTED]   2008-05-12 15:08:11 
Re: rounding problems
craig@[EMAIL PROTECTED]   2008-05-13 04:29:06 
Re: rounding problems
justin@[EMAIL PROTECTED]   2008-05-12 17:02:30 
Re: rounding problems
justin@[EMAIL PROTECTED]   2008-05-12 17:09:57 
Re: rounding problems
justin@[EMAIL PROTECTED]   2008-05-12 17:23:15 
Re: rounding problems
craig@[EMAIL PROTECTED]   2008-05-13 05:50:16 
Re: rounding problems
justin@[EMAIL PROTECTED]   2008-05-12 18:37:02 
Re: rounding problems
aanderson@[EMAIL PROTECTE  2008-05-12 19:04:13 
Re: rounding problems
sam@[EMAIL PROTECTED] (S  2008-05-13 02:20:18 
Re: rounding problems
justin@[EMAIL PROTECTED]   2008-05-12 22:51:42 
Re: rounding problems
craig@[EMAIL PROTECTED]   2008-05-13 11:42:09 
Re: rounding problems
justin@[EMAIL PROTECTED]   2008-05-13 00:16:06 
Re: rounding problems
xof@[EMAIL PROTECTED] (C  2008-05-12 14:53:21 
Re: rounding problems
craig@[EMAIL PROTECTED]   2008-05-13 05:26:46 
Re: rounding problems
aanderson@[EMAIL PROTECTE  2008-05-12 14:18:57 
Re: rounding problems
justin@[EMAIL PROTECTED]   2008-05-12 14:49:48 
Re: rounding problems
tometzky@[EMAIL PROTECTED  2008-05-13 08:40:41 
Re: rounding problems
justin@[EMAIL PROTECTED]   2008-05-13 11:30:48 
Re: rounding problems
justin@[EMAIL PROTECTED]   2008-05-13 13:29:07 
Re: rounding problems
justin@[EMAIL PROTECTED]   2008-05-13 14:36:18 
Re: rounding problems
sam@[EMAIL PROTECTED] (S  2008-05-14 15:38:17 
Re: rounding problems
justin@[EMAIL PROTECTED]   2008-05-14 11:47:52 
Re: rounding problems
sam@[EMAIL PROTECTED] (S  2008-05-14 17:26:30 
Re: rounding problems
sam@[EMAIL PROTECTED] (S  2008-05-14 20:27:19 
Re: rounding problems
aanderson@[EMAIL PROTECTE  2008-05-14 16:02:32 
Re: rounding problems
justin@[EMAIL PROTECTED]   2008-05-14 16:08:54 
Re: rounding problems
aanderson@[EMAIL PROTECTE  2008-05-14 16:24:58 
Re: rounding problems
justin@[EMAIL PROTECTED]   2008-05-14 16:55:49 
Re: rounding problems
glene77is <glen.e77is@  2008-05-20 03:19:20 
Re: rounding problems
justin@[EMAIL PROTECTED]   2008-05-20 10:28:28 
Re: rounding problems
aanderson@[EMAIL PROTECTE  2008-05-12 15:56:30 
Re: rounding problems
justin@[EMAIL PROTECTED]   2008-05-13 13:01:11 
Re: rounding problems
doug@[EMAIL PROTECTED] (  2008-05-13 12:12:07 

Post A Reply:
  Go here to Signup

AddThis Feed Button


About - Advertising - Contact - Frequently Asked Questions - Privacy Policy - Terms of Use - Signup

Contact
tan12V112 Sat Nov 22 12:34:43 CST 2008.