Re: gsoc, text search selectivity and dllist enhancments

Jan Urba=C5=84ski wrote:
> If you think the Lossy Counting method has potential, I could test it=20
> somehow. Using my current work I could extract a stream of lexemes as=20
> ANALYZE sees it and run it through a python implementation of the=20
> algorithm to see if the result makes sense.

I hacked together a simplistic python implementation and ran it on a=20
table with 244901 tsvectors, 45624891 lexemes total. I was comparing=20
results from my current approach with the results I'd get from a Lossy=20
Counting algorithm.
I experimented with statistics_target set to 10 and 100, and ran
pruning=20
in the LC algorithm every 3, 10 or 100 tsvectors.
The sample size with statistics_target set to 100 was 30000 rows and=20
that's what the input to the script was - lexemes from these 30000=20
tsvectors.
I found out that with pruning happening every 10 tsvectors I got=20
precisely the same results as with the original algorithm (same most=20
common lexemes, same frequencies). When I tried pruning after every 100=20
tsvectors the results changed very slightly (they were a tiny bit more=20
distant from the ones from the original algorithm, and I think a tiny=20
bit more precise, but I didn't give it much attention).

Bottom line seems to be: the Lossy Counting algorithm gives roughly the=20
same results as the algorithm used currently and is also possibly
faster=20
(and more scalable wrt. statistics_target).

This should probably get more testing than just running some script 5=20
times over a fixed set of data, but I had trouble already sucking ~300=20
MB of tsvectors from one of my production sites, putting it on my
laptop=20
and so on.
Do you think it's worthwhile to implement the LC algorithm in C and
send=20
it out, so others could try it out? Heck, maybe it's worthwhile to=20
replace the current compute_minimal_stats() algorithm with LC and see=20
how that compares?

Anyway, I can share the python script if someone would like to do some=20
more tests (I suppose no-one would,  'cause you first need to apply my=20
ts_typanalyze patch and then change it some more to extract lexemes
from=20
the sample).

Cheers,
Jan

--=20
Jan Urbanski
GPG key ID: E583D7D2

ouden estin


--=20
Sent via pgsql-hackers mailing list (pgsql-hackers@[EMAIL PROTECTED]
)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers