World: r3wp

does JOIN still reduce like it did in R2?

Yup, so it's not a direct correspondance.

I always wondered why it reduced... I find that very annoying... 
many times I'd use it and It ends up mucking up my data, so I just 
almost never use it.

I mostly use REJOIN and AJOIN instead of JOIN, or maybe APPEND COPY.

me to.

I am pretty sure, that:


1) the set operations in Rebol are in fact GC-safe using the standard 
meaning of the sentence

2) it is always necessary to use auxiliary data, if the wish is to 
do the set operation efficiently

3) nobody pretending to need a modifying version really needs an 
inefficient variant, which does not use any auxiliary data

why do you say we "pretend"?

I think that people who need a modifying version really need it, 
but the rest of us need the non-modifying default :)

really need it

 - does that mean they need what I specified - an inefficient variant 
 not using any auxiliary data?

I suppose that is nonsense

Nope, it just means they need the first argument modified to contain 
the result instead of what it originally contained.

no one is saying to use anything less efficient than the copy returning 
version.

The only difference between the DEDUPLICATE code in the ticket and 
a native version is that the auxiliary data could be deleted immediately 
after use instead of at the next GC run.

and the data is managed directly by C, not by the interpreter, which 
is faster for sure.

which also means that some of that can possibly be optimised by the 
compiler... something that cannot happen within rebol.

its also going to use less ram, even if it does use some auxilliary 
data... since that auxilliary data is not wrapped within REBOL interpreter 
wrappers.

or at least, parts of it wont.

Not much less RAM. The "interpreter wrapper" is pretty much constant, 
no matter the size of the data. Remember, the data you are doing 
set operations on is REBOL data already.

yes, but the extra data used to build it as a mezz, including the 
stack frames and stuff is prevented.   


I know I'm being picky here.  but we're doing a detailed analysis.. 
 :-)

but in the end, its the usability which everyone wants, even if its 
only slightly more effective.

The only difference between the DEDUPLICATE code in the ticket and 
a native version is that the auxiliary data could be deleted immediately 
after use instead of at the next GC run.
 - that would be inefficient as well

INSERT, CLEAR and UNIQUE are already native, so the actual time-consuming 
portions are already optimized. The only overhead you would be reducing 
by making DEDUPLICATE native is constant per function call, and freeing 
the memory immediately just takes a little pressure off the GC at 
collection time. You don't get as much benefit as adding /into to 
REDUCE and COMPOSE gave, but it might be worth adding as a /no-copy 
option, or just as useful to add as a library function.

right now the GC is very cumbersome. it waits for it to have 3-5MB 
before working. and it can take a noticeable amount of time to do 
when there is a lot of ram.  I've had it freeze for a second in some 
apps.

everything we can do to prevent memory being scanned by the GC is 
a good thing.

by 3-5MB, I mean that it will usually accumulate ~ 3-5 MB of new 
data before running.

Mark and sweep only scans the referenced data, not the unreferenced 
data, but adding a lot of unreferenced data makes the GC run more 
often.

yep.

right now the GC is very cumbersome. it waits for it to have 3-5MB 
before working. and it can take a noticeable amount of time to do 
when there is a lot of ram.  I've had it freeze for a second in some 
apps.

 - what exactly does the GC have in common with the "Deduplicate issue"?

I demonstrated above, that, in fact, nothing.

But that doesn't  mean that deallocating immediately will be any 
more efficient; likely it won't.

This is all just pretending, if, what is needed, is a kind of incremental/generational/whichever 
other GC variant, then no "Deduplicate" can help with that

We don't need DEDUPLICATE to help with the GC. He was suggesting 
that having it be native would help reduce the pressure on the GC 
when used for other reasons instead of a mezzanine version. I don't 
think it will by much.

He needs DEDUPLICATE for his own code. The GC also needs work, but 
that is another issue :)

if we implement deduplicate as a mezz, we are juggling data which 
invariably tampers the GC.  doing this native, helps to prevent the 
GC from working to hard.


the problem is not how long/fast the allocation/deallocation is... 
its the fact that cramming data for the GC to manage, will make the 
GC trigger longer/more often.

But you would need to deallocate every time the function is called, 
instead of just when the GC is called. This is often slower.

Did you read Carl's note to the subject?

The subject of the set function implementation, or the GC implementation 
and how it compares to direct deallocation? If the latter then no.

I meant the note more to Max, and it was about the set function

Ah, cool :)

The GC is not a slow approach to the garbage collection. The main 
problem is, that it is "unpredictable", and possibly producing delays, 
when other processing stops. (but that does not mean, that immediate 
collection would be faster)

yes I did.

the "stop the world" approach is disturbing for user interface, which 
might need a different type of GC...

Also, multitasking could require a different kind of GC. Any thoughts 
on this, Ladislav?

just adding a generational system to the GC would help a lot.  I've 
read that some systems also use reference counting and mark and sweep 
together to provide better performance on data which is highly subject 
to volatility.

though I guess it requires a bit more structured code than rebol 
to properly predict what is expected to be volatile.

re DEDUPLICATE ... it's not just GUI code that would benefit, DB 
code often needs to apply this on the result set. "buffer: unique 
buffer" is a common idiom, which can be problematic with very large 
datasets.

You still need an intermediate dupe even for DEDUPLICATE. This just 
makes it so the argument is modified, in case there are multiple 
references to it.

it's not just GUI code that would benefit, DB code often needs to 
apply this on the result set. 

buffer: unique buffer" is a common idiom, which can be problematic 
with very large datasets" - that is exactly where I was trying to 
explain it was just a superstition - buffer: unique buffer is as 
memory hungry as you can get any Deduplicate is just pretentding 
it does not happen, when, in fact, that is not true

Does DEDUPLICATE *have to* create a new series. How inefficient would 
it be to SORT the series then REMOVE duplicates starting from the 
TAIL. Inefficient as a mezz, but as native?

Deduplicate does not have to use auxiliary data, if the goal is to 
use an inefficient algorithm. But, in that case, there's no need 
to have it as a native.

even if done without aux data, it will still be MUCH faster to do 
in native.