AltME groups: search

>> r: does [recycle set 'a "11111111" system/stats]
>> r
== 4460038
>> r: does [recycle set 'a 11111111 system/stats]
>> r
== 4460038

>> r: does [recycle set 'a ["1 1 1"] system/stats]
>>r
== 4460518
>> r: does [recycle set 'a [1 1 1] system/stats]
>> r
== 4460518

>> a: 1
== 1
>> to-binary a
== #{31}
>> a
== 1
>> a: "1"
== "1"
>> to-binary a
== #{31}
>> a
== "1"

It may be fairer and more accurate to say REBOL has some subtle optimizations 
in its memory allocation.

Watch, for example, the memory drop here.....you'll see that as the 
block 'x grows, REBOL allocates space in anticipation, so a single 
insert into 'x does not always change the stats.
x: copy []
st: system/stats
loop 500 [
    print [(st - system/stats) length? x]
    append x length? x
    recycle
   ]

>>r: does [recycle set 'a [#{7FFFFFFF}] system/stats]
>> r
== 4559204

>> r: does [recycle set 'a [6442450943] system/stats]
>> r
== 4562852

the first is the binary version of the second integer..  safe to 
say that a binary uses more mem than the integer?

Im trying to determine the most memory efficient method of storing 
a large block of numbers (can be integers, bits.. i dont care, as 
long as i can convert it to an integer in the end)

Also cosidering the best storage for finding a particular binary 
or integer in the block?

A binary does use more than an integer, but the above doesn't prove 
it. You're only checking one value. As Sunanda wrote, rebol's memory 
allocations are not obvious. It uses pools of memory, which allows 
reuse of memory.

why would a binary use more mem.. shouldn't it be the other way around?

Gabriele has explained it previously, something like this:
Values life in "value slots". Words point to value slots.

Some values live entirely in their value slot -- chars, integers: 
ie the short ones with a determinate maximum length.

Other values live in memory pointed to the by value slot -- such 
as strings.

There is memory allocation optimisation both for value slots (as 
seen in the growing block of integers example above), and elsewhere.

So a single allocation is not enough to deriive the underlying algorithm.

And a binary is just a type of string, so yes, the value slot contains 
a pointer to the actual data.

so then a block of integers  ie: [1234432   345   45345   5435   
2345  5435353]  .. .
is the most efficient way to store?

Storing a bunch of integers in a binary should be more efficient, 
only one value slot used, and each integer takes only 32bits.

I guess that's my point.. i need to use 32 bits to store a single 
integer??

no negatives..  and a MAX of 32 bits is more than enough for the 
largest number

A value slot is 16 bytes, so  a single integer takes 16 bytes -- 
it has to live in a value slot.

No, the slot size is fixed so it can fit a whole lot of different 
types in it, eg. time! values.

ahh .. ok..  that's a fairly heavy price to pay (relatively speaking) 
just so I can store non binary data types

For compact representation of large integer sets, I often uses the 
format:
    [10x4 50 76x1000] 
ir REBOL pairs! (each taking 1 value slot) meaning
     [10 11 12 13  50  76 77 78 ..... 1076]
This works for me for large blocks of semi-sparse integers.

There is a script for handling this format at REBOL.org -- look for 
rse-ids.r

Ok, so you could pack each 21 (or 24) bits into a binary.

so is that a block of binaries then?

No, a single binary.

That would be a block of binary....each binar would take a slot
Anton is suggesting this sort of approach:
x: make binary! 25000
>> loop 18 [insert x to-char random 255]
== #{5E218289FC8B65B86A1C597C232F9E8C79}
Just one binary string to hold the data.

is there a function to convert an integer to 3 bytes?

would there be a performance trade off as opposed to just a block 
of integers? (finding/ appending etc.)

Note that converting integer <-> binary is a little tricky. But we 
can use a struct! (re-use a shared struct for optimal performance).

i thought crawling a block of binary (hash?) would be faster than 
a block of integers

Here's what Im trying to do.. 

I have a dictionary...   dict: ["one" "two" "three" ... ]    with 
 a couple of million values

I want to build an index block .. dex: [ 2 3 2 1]  that represents 
the index? of each word in my dictionary

but there's a compression value as well..

a value in the dictionary could be a page of rebol code (as a string) 
 .. and it could be represented with a single bit  (or in this case.. 
3 bytes)

so then my code becomes a block of 3 byte 'symbols' that I use to 
'pick' the actual values out of the dictionary with.

Is the aim to create a fixed-width number representation for the 
index ?

(I mean, a fixed-width of 3 characters.)

dict: ["Rebol"  "Carl"  "isa"]    are common..  should  be  represented 
as [1  2  3 ]  .. and dict:["This is a rare string.. probably only 
used once"]   shoud be  [ ZZZ ]

Im going to store the entire dict in memory as a hash!, and my code 
as a separate block of 3 byte "symbols"

an integer is a symbol

the sound you utter when you say "symbol" is a symbol

a symbol is not (only) an integer

my main concern is crawling the block of integers... looking for 
 needles in haystacks.. whatever is the FASTEST method for this is 
best.. compaction is a by-product

Here's a question.. when a block of integers is converted to a HASH!.. 
what actually happens to it?

A hash just has a different way of linking the items it contains 
together, so it performs faster in some operations at the expense 
of others.

That's interesting, using a bit in each byte to "escape" and open 
up the next, more significant byte. So it's a variable-length encoding 
scheme. (I guess, kind of like UTF8)

It should be faster and simpler to just use integers. When you want 
to cut down the size of your 2 million integers (=30MB), you can 
then look at implementing 3-byte integers packed in a binary.

Here's a couple of conversion routines.

I'll add these functions to the dictionary.. 

dict: [{integer-to-3-byte-binary: func [integer [integer!]][
	struct/int: integer
	copy/part third struct 3
]}]


now whenever i want to use that function.. i can represent it with 
a single integer.. ie:  do pick dict 1

>> a: compress "a"
== #{789C4B04000062006201000000}

>> a: compress "aaaa"
== #{789C4B4C4C4C040003CE018504000000}
>> a: compress "aaaaa"
== #{789C4B4C04020005B401E605000000}

the size is used to make a result buffer for decompress:
>> decompress rejoin [#{789C4B4C} #{04020005B401E6} #{00000000}]
** Script Error: Not enough memory
>> decompress rejoin [#{789C4B4C} #{04020005B401E6} #{FF000000}]
== "aaaaa"

Terminology question;  I know I could probably RTFM, but sometimes

Ask A Friendly Human is more fun.  What is the correct terminology 
for the global

REBOL context.  I'm describing (or trying to at least) the parse 
dialect "copy" versus

the REBOL "copy".  Is there a one word term for the "no context" 
context?  Or is
the REBOL global namespace
 good enough (and not too confusing to new rebols)?

Is there a function which can flat a block. For example,
>> FLAT [ 1 2 [ 3 4 ] 5 6 ]
== [ 1 2 3 4 5 6]

what if the record size is 2, consists of a binary and an object 
and you want to sort on a value in the object?

Here's one way -- though it assumes (for simplicity) that the binary 
is a string of equal length in all keys:


data: reduce ["z" make object! [key: 1] "y" make object! [key: 2] 
"z" make object! [key: 2]]
sort/all/skip/compare  data 2 func [a b][
     return (join a/1 a/2/key) < (join b/1 b/2/key)
    ]
probe data

oh wait a minute. I see now :-)

I guess I don't after all. I still can't get it to recognize 'b. 
If I try to probe 'b inside the function, sorting just stops. probing 
'a works fine.

I think the compare feature is just a bit underdocumented.

I can use CHANGE/ONLY FIND ...
What do you think, skould I RAMBO it as a wish?

>> regexp "[hello-:-world-:-re]" "[a-z0-9._%-]+@[a-z0-9.-]+\.[a-z]{2,4}"
== true
;))

Anyone else want a regex compiler that generates parse rules? Semantically, 
regex is a (very) limited subset of parse so it shouldn't bee too 
hard.

Still, if you want help, a tester or a second opinion, post your 
code on the Parse group and we will optimize it for you.

Hey... what's the code to prevent  the security from popping up in 
windows?  added -s to a shortcut, but not working?

in 98% of cases I agree with what Brian just said about Parse being 
more powerfull than Regexp.  but in those 2% regexp is SO much more 
powerfull, that it still has its place.   now some of you will ask 
me to say when or why, (I know how we as rebolers think and like 
to challenge each other ;-)  but I cannot give an exact example, 
but having had to implement a few complex systems in both, I remember 
a few times in parse when I'd remember how a single char would replace 
2-3 lines of parse "tricks".

so, having a regexp WITHIN parse would be oh so incredible.  especially 
since Parse allows such an easy way to traverse data at such a fast 
rate.  having regexp to take "decisions" would scale pretty nicely. 
  OTOH having rebol to take decision could be an alternative too.

Parse is good at matching, but sometimes, a simple condition "means" 
something which is hellish to implement as a set of explicitely mutually-excluding 
matches.  regexp is a little easier in this case (note I use easier 
not in style or readability here... just in raw expressiveness).

if we could add a conditional within the dialect of parse directly 
(without using tricks, which I know many will be tempted to demonstrate, 
and many which I already know about) then parse itself would have 
another level of expressiveness IMHO.

one example could be to use the return value of evaluated parens 
as a "matched? or not" in order to continue in a parse rule.

for example, not only the type and shape of data, but its actual 
value?  have I loaded enough of this, for this rule to qualify.  
is a specific attribute set to a mandatory value?  there are many 
such examples.

again, I know most patterns CAN be described using parse, but in 
many occasions, what could have been a simple parens with a decision 
and 2 or 3 very simple rules, ended up being a complex tree of tens 
or more rules, which have non obvious interdenpendencies and things 
like left entry recursions (I hope I make sense here) which are, 
well, slow(er) and hard to map in one's mind.

Max, could you flesh that out as a hypothetical example?

well, I guess the best way would be to use parens within the parse 
block as a means to return if we should continue in this rule, (and 
maybe even based on type, how many items to skip!).

hum... you are asking my mind to shift from cgi and web site writing 
to parse rule generation.. hehe I'm a bit deep in the construction 
of Revault right now... with about 10 files opened and mapped in 
my mind ;-)

What Maxim is describing is the CHECK clause proposal I made a few 
years ago. See here:
http://www.colellachiara.com/soft/Misc/parse-rep.html

I'm working on reducing memory consumption on my little database 
and was wondering if stats is reading out the total memory usage 
correctly or if Windows XP's job list is. I can do a script that 
gradually eats up 100 MB memory and then the memory is recycled, 
when I ask for it. 'stats then prints about 15 MB used, which is 
fine, but the job list reads out about 100 MB still used and it stays 
there. Right now it reads about 104.656 KB used, while stats prints 
15588191 bytes. This is in a stopped console. Recycling more doesn't 
help.


I've even seen the job list memory jump up 10-20 MB once when recycling. 
Which one is reading out the correct number?

I've made a small change to the %filtered-import.r script -- it should 
now properly handle the 'opt modifier:

>> import [][test: opt string!]
== [test none]
>> import [test ""][test: opt string!]
== [test none]
>> import [test ""][test: string!]
== none

This last one could be considered unexpected?

This seems to work:

>> listen: open tcp://localhost:8001

But the docs specifically say:


Notice that you do not supply a host name, only a port number. This 
type of port is called a listen port. The system now accepts connections 
on port number 8001.

Is this a mistake in the docs?

but you can't do
abc/'def/(1 + 2)
so maybe a new repath function:
repath abc/'def/(1 + 2)
?

Louis:   if you:

open tcp://localhost:8001


you do not open port for listening but for reading/writing as for 
example:

p: open tcp://www.rebol.com:80


so if you can open such a port on localhost, you MUST have something 
what listens on such a port

Oh yes, Oldes is right.

Remove "localhost" if you want to be a server. The docs are right.

Your next script, the rebol client that connects to this server, 
*will* specify localhost.

Anton: you mean we need a repath function?

btiffin and sqlab, it turns out that you both suggested the right 
thing. I must have a lot of ports being used already on my computer. 
Since you both thought this is what was wrong. I just kept on trying 
port numbers until finally...it worked! Thank you both very much! 
 Thanks to all of you that helped me, this day has ended pretty good! 
 Having endured the earlier aggravation, the good feels even better 
than usual.


So many ports being open does make me wonder why, however. That seems 
a little dangerous to me.

Louis, here's a windows tool that will let you see what ports are 
open and what application is listening on the ports.

http://www.nirsoft.net/utils/cports.html

Will, a REPATH, AS-PATH, or DO-AS-PATH kind of thing would be very 
handy at times.

This is the client; put it in the folder containing the files you 
want to send.

rebol []
ip: request-text/title/default "IP Address: " "localhost"
port-num: request-text/title/default "Port Number: " "2006"
url: to-url rejoin ["tcp://" ip ":" port-num]
system/options/binary-base: 64  ; best binary encoding


print ["This program SENDS all files in its folder to receive-files-tcp." 
newline]
print "NOTE: receive-files-tcp must be running on the remote"
print ["computer before starting this program." newline]

files: read %. ; Note that 'files is a block of file names.
save %file-names files
server: open url

insert server compress as-binary read/binary %file-names ;send file 
names
file-block: []
foreach file files [
    if not find file "/" [insert file-block file] ;remove folders
]
files: file-block
foreach file files [
    insert server compress as-binary read/binary file
    print ["Successfully sent file: " file]
]
close server

ask [newline "The files transfer is complete. Press <Enter> to close."]

Louis, I think, your problem is, that you only operate with one port. 
When you define a listen port in REBOL, you receive a port from that. 
This new port can be used to receive the file. Something like:

listen-port: open/lines join tcp://: port-num
wait listen-port
talk-port: first listen-port
file: first talk-port
close talk-port
wait listen-port
...


(I haven't tested this code. It's free from a similar program, I 
made in the past.)

Play around with a very simple test, where you do it manually between 
2 REBOL sessions. This way you get a feel for it and can made your 
program to work.

as current path-thru function is not working with queries and port 
numbers, and it's not part of core at all, what do you say about 
using this imporved version (which I'm already using for couple of 
years):

path-thru: func [
    "Return a path relative to the disk cache."
    url /local purl
][
    if file? url [return url]
    if not all [purl: decode-url url purl/host] [return none]

    if all [string? purl/target find purl/target #"?"] [replace purl/target 
    "?" "_query_"]
    rejoin [
        view-root/public slash
        purl/host

        either none? purl/port-id [""] [join "_atport_" purl/port-id] slash
        any [purl/path ""] any [purl/target ""]
    ]
]

so:

>> path-thru http://us.maps2.yimg.com/us.png.maps.yimg.com/png?v=3.52&t=m&x=3&y=0&z=16

== %/D/view/public/us.maps2.yimg.com/us.png.maps.yimg.com/png_query_v=3.52&t=m&x=3&y=0&z=16

How many rebols have written language localization routines?

I'm toggling back and forth between external heaps and in-code strings
I've got a RebGUI widget...


lang-text {en "This is the english" fr "C'est francais" it "Don't 
know any italian"}

meaning a translator will have to get dirty in code edits (or send 
to coder) or use

text (lang "SomekindaKey")

where lang is some func that having read some file, selects the string 
key by lang type...
lang-type being buried somewhere in locale*


How often is a REBOL translator a non-programmer?  I find external 
text to be a pain when
coding.  But...it lets non-coders help with translations.


In particular, I only have about 10 or so screens that could be translated. 
 Ashley's

builtin localization nicely handles all the GUI stuff.  I'm leaning 
toward in-code strings.

Qtask using a huge database for all languages...

On the fly translation?  Or work by coders?  Or a text heap?  :)

I guess I'm just looking for advice, but I'm travelling down the 
in-code multi-language
string path.

and the preprocessor can create a nice .catalog file for you

then, you can use a gui tool to edit .catalog files.

A key consideration is whether you want the translation(s) to be 
static (compile-time) or dynamic (run-time). Advantages and disadvantages 
to both approaches.

some BIND expert, please, help me:

>> act: [add a b]
== [add a b]
>> for a 1 5 1 [for b 1 5 1 [do act]]
** Script Error: a has no value
** Where: do-body
** Near: add a b

There must be something easier than
>> for a 1 5 1 [for b 1 5 1 [do bind bind act 'a 'b]]

Is this simpler and the same effect?
    act: func [a b][add a b]
    for a 1 5 1 [for b 1 5 1 [act a b]]

hm, not exactly what I want because this is simplified, I have more 
variables than just 'a and 'b, but still useful.

May or may not work for your needs

for a 1 5 1 [for b 1 5 1 bind act 'a]

I mean rebol-file from such a url

hm.. parse t: "a//bb/c" [any [to "/" p1: some "/" p2: (p1: change/part 
p1 "/" p2) :p1] to end] t

You mean, to "clean" it, into a legal url ?

I already had own version, but your is a little bit better as correctly 
handles complete url... my was only for the path