World: r3wp

100000 is easy

but... it's a good idea to study some things here about which loop 
functions are faster than others.

great - I don't think anything I would pull thru would contain more 
than 100,000  rows x 15 cols

Thanks Henrik.....For large blocks, you can help REBOL's memory managemeny 
by pre-allocating the needed space. So, applying Henrik's optimisation:

    oldblock: copy [ fld1 fld2 fld3 fld4 fld1 fld2 fld3 fld4 ]
    newblock: make block! (length? oldblock) / 4

    foreach [a b c d] oldblock [
       append/only newblock reduce [a b c d]
      ]

some functions can give simple code, but are slower, because they 
are mezzanines based on other loops.

in sunanda's example, APPEND is actually a mezzanine. if you wanted 
a tiny speed up, INSERT/ONLY TAIL is a little faster than APPEND/ONLY

You could do an easy timing test by creating some test data:
    loop 1000000 [append oldblock random/secure "abcdefghijklm"]

....My computer took about a second to reformat the test data. Fast 
enough?

Brilliant, thanks Sunanda. you guys answer questiions faster than 
I can ask them ;-)

you can explore this yourself by studying the source for various 
loop functions.


 http://www.hmkdesign.dk/rebol/files/385cc7420409fa08f768a537a52ce4d8-76.html

That blog post might be of help.

be aware that pretty code is not always fast, even in REBOL. :-)

Thanks Henrik for the append vs insert optimization too!

Thanks Henrik, I was a bit confused by looking thru the core and 
dictionary functions - the examples don't really show advantages/disadvantages 
for real world apps

one thing that amazes me is that PARSE is very fast, despite being 
one of the most complicated functions to understand in REBOL. if 
you learn how to use it, it can perform some quite amazing tasks.

and once again, we'll be turning this upside down for REBOL3. :-) 
there are a lot of new functions to aid in the creation and manipulation 
of blocks with simple code that also performs very well.

Yes I've only used it for my licence creator applet - I't's one of 
the core items I need to master. At the moment I'm trying to get 
blocks/series to fit into recursions I have created in legacy apps

I don't mind re-learning these for R3, I thnk I need to learn R2 
to master R3

sorry this wireless keyboard has a stutter ;-)

So the immediate thing I've picked up is as you use append Rebol 
has to re-form the block on each iteration! Is that right?

generally about speed: I rarely worry about REBOL/Core speed. I made 
a database once that was mainly about in memory block manipulation 
in REBOL. On an old 500 Mhz Celeron laptop, I couldn't saturate it, 
when throwing queries at it from 3 fast PCs over a 100 MBit LAN. 
I think it was about 100-1000 times faster than a MySQL database, 
but of course with the price that it's in-memory and that it had 
very few features.

be careful about wording, as 'reform is a function in REBOL. :-) 
all it has to do, is allocate more space. Just imagine a cramped 
desk and you want to put some things on the desk. Which is faster? 
Is it to remove one item at a time from the desk to place a new one 
there, or just swipe the desk clean in one go in order to free all 
desk space immediately?

The processing seem's amazing compared to say one of my c# apps.

yes, and they say interpreted languages are slow, right? :-)

Yep sorry Henrik, I meant allocate more space

Well really VB and C# are interpreted as well

the code for a c# app is compiled into an intermediary psuedo code 
then interpreted by the CLR (Common language runtime)

generally you just want to reduce the cases where you need to allocate 
space. if you reduce the number of cases and that reduces the number 
of garbage collections needed and there is a greater guarantee that 
all the allocated space will be freed up. REBOL is good, but the 
garbage collector isn't super intelligent so it helps to simplify 
its work.

I was going to ask how efficient the GC was in Rebol - it's one thing 
that is good in c'# compared to c++

how it works exactly is unfortunately undocumented. Only Carl knows, 
so everything we know is based on observations and tests.

when your re-painting screens and such you have to force GC or the 
display struggles

View is a bit different, because it hogs memory quite badly. You 
can do a few limited things there to speed up display, but not much 
in terms of helping GC.

Video processing on R3 looks amazing

the difference in performance on R2 vs. R3 in graphics is also quite 
amazing, mostly thanks to much reduced memory requirements to create 
and manage graphics.

Using the alpha channel allows for some great transparencies and 
other effects

btw, about your append question... no, append does not need to allocate 
memory at each call. each series has some space free, and when that's 
up, a bigger space is allocated, usually with 2x increments (until 
a certain size, then it's linear). however, the whole series needs 
to be copied when that happens. so, if you append 1000 times, it 
may happen just 2 or 3 times, but still, if you know the final size 
already, you can save a lot by just preallocating it.

if you don't know the final size... it's usually better to let rebol 
do its magic. but, i'd suggest timing the code and deciding which 
is really going to perform better, as there are many variables involved.

gabriele, doing "magic", isn't that degrading performance in the 
long run, if he has to perform the operation thousands of times?

Thanks, I think with the speed that's been mentioned compared to 
my old recursions in c# I'm worrying about nothing.

Biggest customer i've got has 250,000 x 20 rows x 15 tables

Gabriele, what I'm mostly concerned about is how memory is eaten 
up over time, because GC isn't done well enough. Do you know anything 
about what will be done here in R3? Will the GC be documented?

steveT, if you have the memory to fit that amount of data, then REBOL 
will process it without problems.

cool, I think I used to suffer a little because if your try to open 
a recordSet in visual studio it creates a natural bottle-neck because 
some idiots might have bound controls - and it has to handle that. 
I had to do all my processing of data on a 'SQL disconnected basis' 
 know what I mean by that?

I think from what everyone says once you get data into REBOL it's 
wonderful. I just wanted some options for pulling the data in.

for those cases:  Learn PARSE. I haven't fully learned it yet, but 
the things it can do is just.. wow :-)

I will. Will there be something standard in R3 to say pull a csv 
file in to blocks?

PARSE in R2 already does that to some extent.

an example of that would be much appreciated ? I need to print this 
thread out and digest what everyones suggested. Thanks so much for 
all the help.

I don't think much time will be spent on building special import 
functions unless its for media formats, which would really require 
performance.

read/lines is a start to turning CSV into blocks -- you get one block 
entry per record.

right , thanks I'll have a go with that. I'll save all these snipets 
and perhaps we could build a database how-to to put on the new users 
docs.