AltME groups: search

and now it's also running on his Linux router :-)

yeksoon - what? I expected R3 running on Linux or OS-X not sooner 
than maybe in one year, and you are unhappy, that PPC ancient architecture 
is not supported yet? :-)

btiffin: thanks for letting me know, I was checking out QM a little 
and Chris has helped me few times here already... but about the frameworks.. 
I don't like to use (mvc) frameworks , in any language

also #505 and #503 should be reviewed

Just did, and they look related. I think MAKE IMAGE! is hosed - it's 
probably a good thing we don't have LOAD-JPEG at this point.

As an alternative to DIR-EXISTS? and FILE-EXISTS? we could change 
EXISTS? so it returns more information.

; R3 version:
exists?: func [

    "If a file or URL exists returns 'file or 'dir, otherwise none."
    target [file! url!]
][
    select attempt [query target] 'type
]

; R2 version:
exists?: func [

    "If a file or URL exists returns 'file or 'dir, otherwise none."
    target [file! url!]
][
    unless error? try [
        target: make port! target
        query target
    ] [

        either 'directory = target/status ['dir] [target/status] ; To work 
        around a current incompatibility
    ]
]


EXISTS? could still be used in conditional code, with the exception 
of AND and OR, but would have more info if you need it.

(And AND and OR could still be done with ALL and ANY, probably what 
I would use in preference anyway.)

I've almost never seen EXISTS? used with AND or OR, though I rarely 
see AND or OR anyways. You can always use FOUND? or TRUE? if you 
want to turn it into a logic value :)

I've always found AND and OR not very rebolish, maybe because they 
can be infix (operators), where ALL and ANY are prefix (like functions). 
And you often need parenthesis, when used infix:

>> or 1 = 2 2 * 2 = 4
== true
>> 1 = 2 or 2 * 2 = 4
** Script Error: Expected one of: logic! - not: integer!
** Where: halt-view
** Near: 1 = 2 or 2
>> 1 = 2 or (2 * 2 = 4)
== true

and ANY is faster!

Well, we really need the information returned by the EXISTS? function 
above, and my last attempt to get that information out in a R2-R3 
compatible way (the above *-EXISTS? functions) got a lot of complaints 
(mostly from Gregg, as I recall). This is hopefully a less annoying 
change, and is compatible now even without the 'dir tweak if you 
check against 'file instead.


My opinion of the *? functions that are meant to be predicates is 
that they should be usable as predicates, but don't necessarily need 
to be simple predicates. As long as you can use them in IF statements, 
they're fine. We have methods to convert from REBOL truth values 
to logic! if we need to.

Sure. In REBOL 2 there are 2 functions, EXISTS? and DIR?, that check 
for whether a file! refers to an existing file and whether the existing 
file is a directory, respectively. Both of these functions wrap around 
QUERY, a low-level native that works very differently between R2 
and R3, mostly because of the port model change. In addition, DIR? 
has a design shortcoming in R2 (mentioned in CureCode ticket #602) 
and both DIR? and EXISTS? share the same bug in QUERY in R3 (#606, 
affects #602 and #604).

All of these combine into a few problems:

- People who want to write file and directory management code that 
is portable between R2 and R3 have trouble doing so.

- Bugs of the kind mentioned in #602 are not likely to be fixed in 
R2, so we have to consider DIR? broken for non-existing directories.

- Using both DIR? and EXISTS? means two QUERY calls, which has overhead, 
particularly for networked files.

- Attempts to get around this using QUERY require completely different 
code in R2 and R3, so wrappers would be nice.


As it specifically relates to 2.7.6, for people who don't care about 
forwards compatibility, there is only one problem:
>> DIR? %nonexistingdirectory/
== false  ; Should be true, unlikely to change

Also, R2 and R3 could use a standard function that does the opposite 
of DIRIZE. Current proposed names are UNDIRIZE or FILEIZE.

I can't fix problems like modifying the date on directories - that 
is native code, and I just work on mezzanines.

Rather, there be a separate distribution for just the main bin and 
then the mezzaines.

We only include the mezzanines we use, and I wouldn't suggest something 
unless there is already a need for it. Your TRIM/with code is wrong, 
btw, we only trim the last / and from a copy at that.

Well that would be nice.  We shall wait and see.

Think of these as a standard library of helper functions that you 
don't have to use if you don't need to. If you do use them, you can 
count on them working as correctly as the REBOL experts can make 
them work, and as efficiently. Either way REBOL is better.

So mezzanines are different on the inside. As long as they work the 
same on the outside, your code doesn't need to change. That is why 
the mezzanines are there. And code that is not part or the REBOL 
distribution is not mezzanine code, just REBOL code. If you want 
it to be mezzanine code (with all of the optimization benefits mezzanine 
code gets), submit it :)

Yeah and at less evals then yours.

I'm getting to where I use less and less mezzanines.

The simpler and faster I can make them the better. If this means 
imporovements to the natives to make the mezzanines better, then 
any code you write that also uses the natives will also be better. 
And you get good library funnctions too :)

And I didn't fine remove to be better consistently. Clear won half 
the time with the same code.

I ran a dozen profiles of each, and they were 50/50 on which was 
faster. That is well within the profiler variance.

Well, my get-block function is an example.  I used it on a series 
of block data and get different results that don't seem to jive with 
my expectations.

I had done a test where I read a small 5000 record file and compared 
to a 100000 record file and the 100000 record file proviled better 
performance than the smaller one.

And while is slower than until and foreach.  It gets complicated.

The trick is that it is harder to write and optimize parse code, 
so it can get really slow without you understanding why. The downside.

It gets complicated.

 It gets more complicated with R3, since R3 has made most of the previously 
 mezzanine loop functions native. This means that FORALL and FORSKIP 
 are faster than FOREACH in R3, while they are much slower in R2.

Yeah, that is my hope that some of those spark some interests.  Feel 
free to add comments on the site as well or updates to the functions 
that you feel could be more beneficial.  The goal of that thread 
was really to open the discussion of mezzanines and how to improve 
them.

ALTME is not the best medium for that type of  effort.  The forum 
remains with some clean code and examples.

Really, that is it.  I think there would be much more involvement 
if the UI was much better.  I think the problem is that Carl is still 
in honeymoon stage with RebDev and doesn't yet understand how much 
work is still really needed to get it embraced.  At this point I 
think ALTME is far better medium than RebDev.

Chris, yes, but that would double the code of DIRIZE and add refinement 
checking overhead to every call, so the effect is worse.

Of course it takes a lot of work to make a function simple to learn 
and use.

The list! type is gone from R3, and that was the only type with index 
overhead.

in that light ...  does this solve >>exists? ; exists?/dir ; exists?/file 
exists?/...<< does this solve that problem too? >>- Using both DIR? 
and EXISTS? means two QUERY calls, which has overhead, particularly 
for networked files.<<

maybe I understood Brian wrong.. I thought in current situatuion 
you need to call exists? somepath and dir? somepath to know that 
something exists and is a directory (which also means two query calls 
I suppose)

Janko, yes, the current situation is exactly that; to know that a 
directory exists, you need to call exists? and dir?, which causes 
two QUERY calls.

In R2, there are only two 'un verbs: 'unset and 'unprotect.  'undirize 
seems contrived (yep, dirize is contrived too, but necessary for 
a state that has no other name).

This is a particular case.  I can see the useful ness from a mezzanine 
standpoint of having a function that does both add the "/" and subtracts 
the "/".  Because in the case of looping we can easily homegrow our 
own need there that would be more efficient.  But I agree the name 
of dirize is not so elegant.

There's a big difference between an inverting refinment and a logic! 
parameter: default behavior. 


I'm all for a better name. Even better than that, a convention. Adding 
"ize" (dirize) or "ify" (blockify) isn't a great solution, but there 
is some basis for them (compartmentalize, normalize, scarify, terrify). 


TO-* and AS-* have specific meanings, and are core funcs. What should 
the standard derivation be for this kind of behavior?

I found hash! a very useful datatype, I still don't get why it has 
to be removed. Map! looks less flexible because you have to conform 
to the key/value data model and it doesn't seem possible to navigate 
in a map! like in a hash!. Why can't we have both hash! and map! 
in R3?

probably you have to modify do-request.
save %do-req.r mold do-request
edit in do.req.r the line 
User-Agent: "REBOL"
to whatever you desire, add the rebol [] header and do the file.

I liked list and hash and did use them a lot.  List was was buggy 
though.

there is someone using list!? I wonder if there is any advantage 
of list! over hash! and block!

lists are fast with insertion and deletion, i use them sometimes

Oldes, yes there is an advantage to each.  Lists are faster for inserting 
and traversal.  Hash is faster for find operations.

by the way, some times ago, i made an algorhitm to simulate hash 
with blocks, it's quite fast and use much less memory than hash.

But it works only with integers key. i could search for it if someone 
is interested...

Anyway.. list! and hash! are no longer in R3

I don't know much about vector or map but I hope that we haven't 
loss the functionality of list and hash in R3.

Map! has issues for now (see CureCode), and the traversal thing is 
being worked on. Vector! is almost completely nonfunctional now.

There are plans to extend FOREACH to map! and object!, and MAP could 
make sense too I suppose, but no plans for that. What did you use 
hash! for that wasn't keyed search, Doc, and what advantages did 
it give you over block! aside from a different datatype?

You give up position and persistent ordering, and in theory you also 
give up duplicate keys though there is a bug ticket about that.

We are extending support of many of the functions that people traditionally 
associated with series to objects and maps, without making them into 
series. So that means that APPEND works, but INSERT doesn't because 
of the position stuff.

SELECT works on object! and map! but FIND doesn't. PICK and POKE 
work, but they take keys rather than indexes. There are some (at 
this point undocumented) limits on what can be used as keys too (at 
least you can't use a block as a key in practice), but that may be 
a bug.

Well, in some cases to fix, but yes, lots of exceptions. The guideline 
is that maps and objects don't have position, but series do. So the 
position-dependent functions don't work but their non-position-dependent 
counterparts do. Ports don't work like series either, so there is 
another whole set of differences to remember.

I agree, Doc, that sounds like a good plan. Map! storage and implementation 
is in theory more efficient than hash!, so having two of them shouldn't 
be a problem, as long as your wrapper functions keep track of the 
mirroring.

The only trick is that you remember that none means nothing, so don't 
bother with a none key. The theory of map! is either:

- Keys that are associated with none don't exist in the map!, so 
assigning none to a key makes it go away.

- All possible keys are in the map! but only the ones associated 
with something other than none are displayed.

There is no difference between these two theories in practice, and 
whether there is memory allocated for keys that you can't see is 
an implementation detail that is irrelevant to the use of map! (though 
there is usually not).

something related, in the past i made some tests to simulate hashs 
with integer keys in R2. I used a bitset as an index, mixed with 
block of blocks to store data.

my tests show that for 10000 records,  finding data is near as fast 
as with hashs. 
actually it's incomplete but you have the idea with this:

REBOL []
f: fast-dic: context [
	size: 100000

 hash: 128 - 1	;** hash size speed up the search, must be a power 
 of 2 - 1 (ie. 15, 31, 63, 127, 257 ...)
	master: copy/deep head insert/dup/only [] [] hash + 1
	index: make bitset! size
	flag: func [idx [integer!]][
		unless find index idx [
			insert index idx

   insert/only insert tail pick master idx and hash + 1 idx copy []
		]
	]
	flag?: func [idx [integer!]][find index idx]
	deflag: func [idx [integer!]][
		remove/part index idx
		remove/part find pick master idx and hash + 1 idx 2
	]
] 

t: now/time/precise
loop 10000 bind [flag random 99999] f
print now/time/precise - t
t: now/time/precise
loop 10000 bind [flag? random 99999] f
print now/time/precise - t

People in general didn't use list!, because of the bugs and because 
block! was good enough for most uses. We haven't felt its lack for 
a year now. Most of the advantage of list! in regular code was handled 
by allowing insert and remove at the head of a block! to expand or 
contract the block, just like it does at the end, without a block 
copy. Just by making block! more efficient for inserts and removes, 
we have made list! even less necessary.

I tested vector! extensively about one and half year ago. Not much 
changed, I think. The BIGGEST problem with vector! is that it's zero 
based.

Trust me, vector! has bigger problems than that right now. I expect 
that it will not end up 0-based though, even with all of the advantages 
that 0-based vectors give us. It is likely that we will instead have 
0-based versions of PICK and POKE that work on all series.

Brian, all the vector! tickets are mine, so I know very well about 
vector! problems :) I'm not sure about zero based PICK and POKE, 
I would probably prefer PICKZ and POKEZ, but I can get used to it.

Yes, those would be good names for the aformentioned 0-based versions 
of PICK and POKE. PICK0 and POKE0 have also been suggested as names, 
though I prefer *Z. They could be natives (or mezzanines?) that call 
PICK and POKE internally. I think the only reason we don't have them 
already is that PICK is an action!, and the naming disagreement.

How can inserts at the head and tail be done without moving the block, 
then?

this ? "Just by making block! more efficient for inserts and removes, 
we have made list! even less necessary."

The only way I can think of to keep the memory contiguous is advanced 
use of the hardware MMU, which would lead to partially used memory 
pages at the start and end of each block

¨People in general didn't use list!, because of the bugs and because 
block! was good enough for most uses. We haven't felt its lack for 
a year now. Most of the advantage of list! in regular code was handled 
by allowing insert and remove at the head of a block! to expand or 
contract the block, just like it does at the end, without a block 
copy. Just by making block! more efficient for inserts and removes, 
we have made list! even less necessary.¨

If the block needs to be expanded because there isn't enough allocated 
the system does a block copy. If there *is* enough allocated, as 
when you preallocate using make block!, then the system doesn't have 
to do a block copy. That is R2 and R3.


What is new in R3 is that the "head" pointer of a block doesn't have 
to point to the beginning of  the allocated memory, just like the 
"tail" pointer in R2. This means a remove from the head of the block 
just shifts the pointer over one in R3, while in R2 you had to copy 
over the rest of the block contents to shift it towards the head 
of the allocated memory. Preallocated memory can also exist before 
the head of the block contents in R3. This means that there is no 
difference in overhead between inserts at the head or the tail of 
a block in R3.


In theory, inserts inside the block in the first half could be more 
efficient because you would only have to shift from the nearest end, 
not the tail. I don't know whether this optimization has been implemented.


Block operations in general could be faster because with no list! 
type we wouldn't have to special-case as much code, so we could make 
our code much faster through more aggressive optimization.


Btw, I submitted a tweak to DP to make it more accurate by subtracting 
its own overhead. It still has some variance though - have to tweak 
the native to fix that. Plus there is the extreme variance caused 
by Windows.

There is a proposal (looking likely to be implemented) to have FOREACH 
work on object! and map! types. The word list syntax would be restricted, 
but you could do your traversal that way. In the meanwhile you have 
WORDS-OF to get the keys in a block, VALUES-OF to get the values 
in a block, BODY-OF object! to get both in a block (map! proposed 
too) and TO-BLOCK of map! to get both in a block. It works, but the 
FOREACH proposal would create fewer intermediate blocks.

Of course FOREACH of map! would operate in the order that TO-BLOCK 
map! would return the keys and values at that moment. In the long 
run you would have to consider the order of FOREACH map! to be non-deterministic 
between calls. The map! type has no inherent ordering, so position 
and sorting are meaningless for it.

It doesn't. SELECT works on object! and map!, but there is no comparison 
except equality for those types.

REBOL is a lower-level language than SQL. No set operations (for 
map!, ar least). Those kinds of SQL operations need to be done procedurally 
in REBOL, using block copies and such.

No, there are no plans to implement range. The head and tail of the 
series are attributes of the series, not the reference to the series. 
It's like the difference between the position attribute of a port 
and a series: For a series, the position is an attribute of the reference 
to the series, while for port, the position is an internal attribute. 
Still, a subseries reference could be implemented as a user-defined 
datatype as long as it is careful to make up for changes in the underlying 
series.

I think that the idea was to be able store internal REBOL values. 
I mean that you for example close console with some defined values 
and after restart of the console you could load into the state where 
you ended.

At least I think Carl was somewhere talking about that. And that 
it could for example speed the boot time, which could be useful for 
cgi aps.

Rebin is intended to be a binary syntax for REBOL, like EBML for 
XML. All REBOL values (or maybe just the literal values) will be 
stored in a binary format. I suppose rebin would be an output format 
of MOLD and SAVE.

No, just the value and the reference to the value with the position. 
The poisition isn't a reference, just an integer (internally). So 
you need the series, the length of the series, a datatype tag, the 
reference to the series, and the offset. The reference and offset 
would be part of the contents of anotther series. It would take less 
space in binary than in memory or REBOL syntax.

As for looking over Carl''s shoulder, I've been more following the 
process and discussions. If you know the REBOL semantics and have 
experience implementing this kind of thing yourself, the big picture 
is obvious. Carl's better at this than I am though - I can only guess 
so far.

B: next A: "blabla"

A (contains a logical reference) --> (physical reference)-->"blabla" 
(value)

B (contains a second logical reference to the same physical reference) 
--> 

if "blabla" is expanded the physical reference is modified and "blabla***" 
is stored in another one place.
The references in A and B are not modified.
That why i said there is 2 indirections (2 references)

Sqlab and Steeve: Thank you for the user-agent issue

Theoreticaly empty space may be in the middle, that is question of 
implementation with influence to performance of course (and diference 
betwen block, hash, list, map etc.)

Brian, Vector datatype can't be expanded or appended, ie the size 
is fixed in the moment of creation, for use as fix lookup maybe good, 
practically not, it is kind of series and should behave similarly

What we need to worry about is clarity of how each series type behaves 
with the series functions (as uniformly as possible) and fixing the 
bugs for AT on blocks and PICK on strings.

You are right, those functionalities behave consistently. I probably 
need some "rebol philosophy" explanation for such stuff. Why e.g. 
head is at first series element, and why tail occupies one past last 
series element, etc.

I have to think and come up with example, when you use index? something, 
to get your position, I think that then the change shifts the calculation 
in such a case...

It's perfectly consistent and useful for subblocks. If INDEX? was 
local to the block in use, a lot of things wouldn't be possible.

Petr, I have been proposing that new PICKZ and POKEZ functions be 
added to do a 0-based PICK/POKE, instead of having vector! be 0-based. 
This would give us 0-based referencing abilities for all series, 
not just vectors, then we could make vectors 1-based like the rest. 
There are real advantages to 0-based indexing so it would be good 
to have it, but consistency is better here.


Carl was not proposing to make a change to PICK and POKE in his blog: 
he already (half) made the change. He was asking us in his blog if 
he should change it  *back* to the old, buggy R2 behavior. I think 
he should finish fixing PICK, POKE and AT instead.


Henrik, INDEX? returns a 1-based index from the *head* of the series 
- that's why it's always positive.

Indexes are calculated by adding the base to the offset. The new 
behavior for PICK and POKE makes that consistent, though AT should 
be fixed as well as it is the only index-based native left (unless 
I've forgotten something). I've added bug tickets for all of this.

Personally I prefer 0-based indexing since it's more useful and less 
confusing, but we are stuck with 1-based.

In R3? Tasks don't really work yet, and we may be changing the model. 
Aside from that, I don't know.

then I hope the final implementation will get something like freeze 
and kill

erlangs concurrency model, hot code loading, easy process communication, 
robust message passing 

and a nice database (let's call it rebnesia)...all ported to our 
fabulous rebol.
...when did you said you are ready with this brian? :-))

I think special thing about erlang is that each green process has 
very very little memory overhead (just 16-32 words) that is why there 
can be soo many of processes and that it can switch between them 
very fast. I imagine Erlang vm spreads N green processes on some 
number of native threads/processes (but I am not sure)... Another 
interesting language to look in this regards is stackless python

Erlang uses their own lightweight process implementation (like you 
could write in REBOL .... hint); remember that the first version 
were written in Prolog. The Erlang VM uses all cores on a machine. 
The "no shared memory" aprroach makes this easy.


R3 should be able to utilize multiple cores. Then with async networking 
and people finally understanding dialecting who needs tasks? Just 
roll your own.

Maarten - could you please explain your "who needs tasks" remark? 
You need some low level support, to be async, no? And by async I 
don't think only networking ....

He might mean "who needs threads?", because "tasks" aren't defined 
yet. As Erlang proves, there are advantages to having both green 
processes and full processes, especially if they are basically interchangeable 
as far as the code running in them is concerned. You *could* just 
go with full processes like R2 does ("Just roll your own."), but 
that wouldn't be as efficient as a hybrid green/full approach, particularly 
on platforms like Windows that have huge creation and switching overhead 
for full processes.

Henrik, everyone who can, please get into R3 chat and read messages 
#1786 and #1436. I need help.

Paul, the R option exists and works like you specify.

Ahhh, that must mean that me and the Great mind of Carl think alike. 
 ;-)