AltME groups: search

We weren't ignoring it, trust me. I have been working on integrating 
the new features, which resulted in a redesign of the semantics. 
But the code was intimidating even after the rewrite, and there were 
some repeating awkward code patterns that needed native replacement, 
which I couldn't do. The new module system that Carl and I are working 
on is based on my work, even if the code may not resemble my code 
on a surface level.

Main changes so far, relative to my recent work:

- A different code style which Carl thinks will be easier to read 
and maintain (mostly using CASE/all).

- Some of the more awkward repeating code patterns have been split 
out into functions, which in some cases will be made native.

- The API of LOAD has been simplified, though is more flexible. Its 
behavior with no options is the same, but some options have changed.

- Some functions are renamed, and some code has been moved from function 
to function (this was expected).

- The sys and lib contexts take the role previously planned for the 
exports context and module-tools mixin, though the usage is the same.

- Fewer functions will be exported into lib than I was expecting. 
The short names of the lib and sys contexts enable this.

I have been recently starting to use CASE... funny how we can discover 
new code patterns after over a decade of using a tool.

BrianH, could you show us a before & after example of code modified 
to use CASE ?

Here's a low-level function to parse and process script headers, 
which shows how many features are built into the base script model 
in R3:

load-script: funct [
	"Decode a script into [header-obj script-ref body-ref]"

 source [binary! string!] "Source code (string will be UTF-8 encoded)"
	/header "Return the header object only, no script processing"

 ;/check "Calculate checksum and assign it to the header checksum 
 field"
	/original "Use original source for Content header if possible"
] compose [
	data: either string? source [to-binary source] [
		unless find [0 8] tmp: utf? source [ ; Not UTF-8
			cause-error 'script 'no-decode ajoin ["UTF-" abs tmp]
		]
		source
	]

 ; Checksum all the data, even that before the header or outside the 
 block
	;sum: if check [checksum/secure data]  ; saved for later
	
	if tmp: script? data [data: tmp] ; Find the start of the script
	
	; Check for a REBOL header
	set/any [hdr: rst:] transcode/only data
	unless case [
		:hdr = 'rebol [ ; Possible REBOL header
			set/any [hdr rst] transcode/next/error rst
			block? :hdr ; If true, hdr is header spec
		]
		:hdr = [rebol] [ ; Possible script-in-a-block
			set/any [hdr rst] transcode/next/error rst
			if block? :hdr [ ; Is script-in-a-block
				unless header [ ; Don't decode the rest if /header
					data: first transcode/next data
					rst: skip data 2
				]
				true
			] ; If true, hdr is header spec
		]
	] [ ; No REBOL header, use default
		hdr: [] rst: data
	]
	; hdr is the header spec block, rst the position afterwards

 ;assert/type [hdr block! data [binary! block!] rst [binary! block!]]
	;assert [same? head data head rst]
	
	; Make the header object, or fail if we can't

 unless hdr: attempt [construct/with :hdr system/standard/header] 
 [
		cause-error 'syntax 'no-header data
	]
	; hdr is a correct header object! here, or you don't get here

 ;if check [append hdr 'checksum  hdr/checksum: sum]  ; calculated 
 earlier

 ;assert [sum =? select hdr 'checksum]  ; Should hdr/checksum be reserved?
	

 if header [return hdr] ; If /header, no further processing necessary

 ; Note: Some fields may not be final because post-processing is not 
 done.
	
	; Skip any whitespace after the header

 ws: (charset [1 - 32]) ; For whitespace skipping (DEL not included)
	if binary? rst [parse rst [any ws rst:]] ; Skip any whitespace
	
	; Check for compressed data and decompress if necessary
	case [
		; Magic autodetection of compressed binary
		tmp: attempt [decompress rst] [
			data: rst: tmp  ; Use decompressed data (no header source)
			append hdr 'compressed  hdr/compressed: true ; Just in case
		]
		; Else not directly compressed (without encoding)
		(select hdr 'compressed) != true [] ; Not declared, do nothing
		; Else it's declared to be compressed, thus should be
		binary? rst [ ; Regular script, check for encoded binary
			set/any [tmp rst] transcode/next/error rst
			either tmp: attempt [decompress :tmp] [
				data: rst: tmp  ; Use the decoded binary (no header source)
				hdr/compressed: 'script  ; So it saves the same way
				; Anything after the first binary! is ignored
			] [cause-error 'script 'bad-press -3] ; Else failure
		]
		; Else it's a block, check for script-encoded compressed binary
		tmp: attempt [decompress first rst] [

   data: rst: tmp  hdr/compressed: 'script  ; It's binary again now
		]
		; Else declared compressed but not compressed, so fail
		'else [cause-error 'script 'bad-press -3]
	]
	
	; Save the script content in the header if specified
	if :hdr/content = true [
		hdr/content: either original [source] [copy source]
	]
	

 ;assert/type [hdr object! data [binary! block!] rst [binary! block!]]
	;assert [same? head data head rst]

 reduce [hdr data rst]  ; Header object, start of source, start of 
 body
]


Note all the commented assert statements: they are for testing (when 
uncommented) and documentation. Also, I later removed the checksum 
calculation from this code because it was the wrong place to put 
it, at least as far as modules are concerned. However, Carl didn't 
know this because he was working on it while I was offline for a 
few days.

Here is the corresponding function in the code reorg, renamed. The 
friendly empty lines and comments haven't been added yet.

load-header: funct/with [
	"Loads script header object and body binary (not loaded)."

 source [binary! string!] "Source code (a string! will get UTF-8 encoded)"

 no-decompress [logic!] "Skip decompression of body (because we want 
 to look at header mainly)"
][
	; This function decodes the script header from the script body.

 ; It checks the 'checksum, 'compress and 'content fields of the header.

 ; It will set the 'content field to the binary source if 'content 
 is true.

 ; It will set the 'compress field to 'script for compressed embedded 
 scripts.

 ; If body is compressed, it will be decompressed (header required).

 ; Normally, returns the header object and the body text (as binary).

 ; If no-decompress is false and the script is embedded and not compressed
	; then the body text will be a decoded block instead of binary.
	; Errors are returned as words:
	;    no-header
	;    bad-header
	;    bad-checksum
	;    bad-compress
	; Note: set/any and :var used - prevent malicious code errors.
	case/all [
		binary? source [data: assert-utf8 source]
		string? source [data: to binary! source]
		not data: script? data [return reduce [none data]] ; no header

  set/any [key: rest:] transcode/only data none ; get 'rebol keyword

  set/any [hdr: rest:] transcode/next/error data none ; get header 
  block

  not block? :hdr [return 'no-header] ; header block is incomplete

  not attempt [hdr: construct/with :hdr system/standard/header][return 
  'bad-header]

  :hdr/content = true [hdr/content: data] ; as of start of header (??correct 
  position??)
		:key = 'rebol [ ; regular script

   rest: any [find rest non-ws rest] ; skip whitespace after header

   ;rest: any [find rest #[bitset! [not bits #{7FFFFFFF80}]] rest] ; 
   skip whitespace
			case/all [

    all [:hdr/checksum :hdr/checksum != checksum/secure rest] [return 
    'bad-checksum]

    no-decompress [return reduce [hdr rest]] ; decompress not done

    :hdr/compress = 'script [set/any 'rest first transcode/next rest]
			] ; rest is now suspect, use :rest
		]

  :key = [rebol] [ ; embedded script, only 'script compression supported
			case/all [
				:hdr/checksum [return 'bad-checksum] ; checksum not supported

    no-decompress [return reduce [hdr rest]] ; decompress not done

    rest: skip first transcode/next data 2 none ; decode embedded script

    :hdr/compress [hdr/compress: unbind 'script  set/any 'rest first 
    rest]
			] ; rest is now suspect, use :rest
		]

  :hdr/compress [rest: attempt [decompress :rest]] ; :rest type-checked 
  by decompress

  not :rest [return 'bad-compress] ; only happens if above decompress 
  failed
	]

 ;assert/type [hdr object! rest [binary! block!]] ; just for documentation
	reduce [hdr rest]
][
	non-ws: charset [not 1 - 32]
]

Notes:

- The other half of the CASE/all style is a lot of explicit shortcut 
RETURN statements, whenever the normal flow differs.

- Errors are returned as a word from the error catalog, which is 
later passed to CAUSE-ERROR.

- Carl redid the checksum calculation so that scripts can verify 
against a checksum in their header, to detect corruption.

- The checksum in the header probably can't be used for the module 
checksum because the header itself matters for modules.

- Compressed scripts lost a couple minor, unimportant features that 
we are likely better without. Quiz: What features?

- Part, but not all of the reason the code is shorter is because 
the doc comments haven't been added yet. The CASE/all style helps 
though.

I am not yet sure if using FUNCT/with is OK with the new build process 
(haven't heard back), but serialized values are now OK. This is why 
I have some alternate code with a serialized bitset.

The new code is not much less complex than the original, but it is 
more compact and faster too. And it is easier to maintain, because 
rearranging CASE clauses is easier to do without a full reorg than 
nested conditional code.

to-error doesn't create armed errors anymore which is a bit strange... 

in the least they are not triggering errors when used within an extension's 
init block.

one function which I would really like to see added to R3 is a search 
function which searches the body of all resident code and returns 
paths or full text of every place an occurence of your search is 
found.

I'm building a search function, btw.  so far not bad.  still have 
to solve a little unset! issue

here my simple but effective r3 search function:

;------------------------------------------------------------
search-body: funct [
	data [object! block! function!] "what to search"
	word [word!] "what to find"
	/paths "only returns paths, not their values"
	/indents i "how many tabs when listing?"
	/into blk "Add matches to this block"
	/path pth [lit-path!] "keep track of path"
][
	
	i: any [i 0]
	
	unless into [
		set 'searched-objects copy [] ; will set in "globals"
	]
	
	either block? :data [
		b: data
	][
		b: body-of :data
	]
	
	
	; locals
	item: none
	match?: false
	blk: any [blk copy []]
	pth: any [all [pth copy pth] to-lit-path ""]
	last-set-word: none
	counter: 0
	
	foreach item :b [
		counter: counter + 1
		
		result: switch/default type?/word :item [
			set-word! [
				last-set-word: :item
				false
			]
			object! [
				; prevent endless cycles on self or inter references.
				unless find searched-objects :item [
					append searched-objects :item
					either block? data [

      search-body/indents/into/path :item word i + 1 blk append copy pth 
      counter
					][

      search-body/indents/into/path :item word i + 1 blk append copy pth 
      to-word last-set-word
					]
				]
				true
			]
			function! [
				either word = to-word last-set-word [
					; adds the definition OF the searched item

     append/only blk to-lit-path append/only copy pth last-set-word
					append/only blk mold :item
				][
					if search-body/indents/into/path :item word i + 1 blk pth [
						; adds a function WITH the searched item in it

      append/only blk to-lit-path append/only copy pth last-set-word
						append/only blk mold :item
					]
				]
				true
			]
			integer! tuple! string! [
				if last-set-word [
					if word = to-word last-set-word [

      append/only blk to-lit-path append/only copy pth last-set-word
						append/only blk  :item
					]
				]
				true
			]
			block! [

    search-body/indents/into/path :item word i + 1 blk append copy pth 
    counter
				true
			]
			
			; this is what we search for
			word! [
				either :item = word [
					match?: true
				][
					false
				]
			]
			
		][
			; these types are not specifically managed by the search
			false
		]
	]
	either into [
		match?
	][
		set 'quiet-search? false
		new-line/skip blk true 2
	]
]
;----------------------------------------------

a: [a]

parse [] a

R3 quits. Bug?

henrik, any case where R3 just quits is a bug... no?

in some cases block, might trigger a match but it doesn't seem completely 
functional, but for me, the above is enough.

Maxim, to answer your questions about cause-error: Three arguments, 
the first two being words from system/catalog/errors, the last one 
eiither being a single value of any type or a block of up to three 
values, depending on which error you are generating. All the info 
you need about a particular error is in system/catalog/errors. The 
number of arguments in the argument block is fixed, per error. The 
usage is in the phrasing of the error message. Pick arguments that 
when molded and put in that position in the error message would make 
the error message make sense.

For instance, when I needed to come up with the right error to trigger 
when a function refinement was incompatible with the datatype of 
another argument, there wasn't an explicit error for that. But after 
looking through the catalog, I came up with this:
>> cause-error 'script 'cannot-use [load-module/as block!]
** script error: cannot use load-module/as on block! value

It will do until there is a better error for that situation.

Sounds like a good idea.

yes, that would be a good first clue, since the current docs give 
no indication on how to proceed right now... 


I should have sourced cause-error, and I usually do... but this time 
well... I guess I assumed it was a native  :-)

First thing I do when wondering about a function is HELP it. That 
tells me the basics, and also mentions its datatype.

is there a way to get a bound copy of a function's body?  sometimes, 
its nice to be able to figure out why a sub-function isn't doing 
what its supposed to...


this could be subject to protection schemes... so that a protected 
member cannot be shown via its function body.

No bound copy of a function body, for security reasons. The kind 
of hot-patching that was possible in R2 was always a security hole. 
Plus, it's not task-safe. For that matter, BODY-OF always returns 
a copy or constructed value, never the original, and code that currently 
uses it relies on this.

the copy I don't mind... that's cool, its the fact that it always 
unbinds (which is what you seem to say).  its not a security hole 
if the functions aren't hidden or protected in some way.


I just want to know what a function within a function actually is 
calling... if its unbound... well I can't make any real inspection 
tool or debugger... right now I can go a lot further than R2, except 
this ... unless I didn't properly understand you.

its a bit like a dll, you only have access to the dll within a debugger 
if it was compiled with debugging... I'd like that to still be the 
case within rebol.

There are two ways of hiding values. The tricky way is to use PROTECT/hide 
on a publically visible context. The more common, easier way is to 
use contexts that aren't publically accesible. There is no way that 
a reflector can tell if a bound context is not accessible, but the 
unbind trick prevents that kind of hack. And since inaccesible contexts 
might contain private information like encryption keys that might 
not belong to the person running the R3 process, there is no security 
setting that can make this safe to not do.

I just finished a very powerfull new version of the search tool... 
it now even allows you to search for any value, even unset!. ( I 
just did a search to find all unset words in the whole system and 
it works without a hitch... )


also, if some data in the system is a string, it tries to find a 
formed value of your search value inside of it:-)

Maxim, I felt the same back In time. A tool like anamonitor is not 
anymore possible. (cry on my face)

well, we can, but its severely limits debugging and frankly there 
is a way to do .   any context should have the possibility of having 
a private/public flag on it... its that simple. whenever you try 
to reflect a private context, it returns unset! (unbound) values.

Though debuging the source and a runtime session, is not the same 
thing.

but you cannot resolve the run-time in data. since things are bound 
dynamically.  the name of a func is useless... we need to be able 
to trace it back to its context by getting its value.


this doesn't give us the context, but at least we can see what the 
function really is.

if things really need to be private, then anything which comes from 
a private context could just be flagged as such and inspection routines 
wouldn't return their values.

maybe we could have a FUNC idiom that doesn't protect anything for 
debugging purposes ?

right now, the only thing I can't "anamonitor" is a function body. 
 I can get that function's body, but can't resolve what the functions 
*it* contains really do.  


all the rest works pretty much as is did, actually it works better, 
since we have real inspection routines now.

debugging implies that a tool shows you that information... the name 
of a function or word is irrelevant in REBOL.

If you need to debug your own source, you can do so. You just can't 
use a debugger to get past the protections.

for example, if I want to do a real time debugger which shows me 
the current content of the block which is being stepped through by 
a loop... 

I can't since I can't get access to the block which is being used 
by that function...

it could be simulated with a special mezz. A function which executes 
an indirect DO on one code block instead of building a function.

So that the code block could still be inspected by a debuging tool.

hum... maybe a way to build a context when building a function, with 
the index of each member of the body block mapped out to its original 
source.

Right, and that kind of thing works now. But allowing a debugger 
to get past the protections just isn't going to happen. We don't 
have a way to know at runtime which person needs to be asked for 
permission to break past a protection like that. Since it certainly 
isn't the user.

we could be able to do this by switching to a "debug" version of 
the various func creating mezz

we could even make those functions able to trigger events based on 
"break points"  then listen on those events in a normal event loop.

so the debugger actually adds a line of code where the break should 
happen... then rebuilds the body block it executes when its called.

Are all the "^M"s below because of datatypes not implemented yet 
or is this a bug? I'd expected READ/STRING to not only TO STRING!, 
but to DELINE the data read.

>> sql: read/string clipboard://
== {create table scans (^M
               id     serial primary key,^M
               date   date not null,^M
               time   time(3) not null,^M
               job    integer not null, ^M
               branch integer, ^M
               ean    char(13) not null,^M
               box    integer^M
          )}

IIRC, READ at one point only returned the data read as a binary stream, 
forcing you to DELINE TO STRING! READ ... because of the transition 
to UTF-8, but /STRING was added back later. Found nothing in the 
change log, though.

The clipboard:// type returns a string without line ending conversion, 
at the moment. It hasn't been updated since before the /string refinement 
was added. It's a platform-specific bug.

Does R3 somewhere have the equivalent of
   READ/CUSTOM .... [POST...]
yet?
I'd like to read a URL as a POST rather than a GET.

Thanks Graham....Now, can I have a clue what the equivalent is!?

Thanks Graham.
Initially, that sounds lllogical -- doing a write to read a URL.

But I guess it makes sense -- as POST is supposed to cause an update 
action of some sort at the server end.

Anyway, 'read is a straight 'GET and write allows custom headers

I always thought HTTP GET is incredibly anti-intuitive when used 
to send form data. Mapping write to POST feels a lot more logical

now whether PUT can stream a file off the filesystem

That was a heck of a coding and debugging session we just got through. 
Alpha 108 is going to be really cool :)

have a lot of things changed wrt the host-kit structure and its files?

I don't know about that yet, I just worked on core builds. Afaik 
there are no major changes to the host kit APIs in this release. 
Only major system structure and semantic changes in R3 itself. And 
a new module system with all sorts of fun tricks available. And some 
fun minor changes to some natives and mezzanines, plus some major 
changes to a few other mezzanines.

What are the registered filetypes in the more recent builds? What 
is the consequence of a file type being registered? Is there any 
documentation?

But we are doing a core release first, to let people hammer on it 
to make sure the system changes work. Then we will likely release 
a host kit, or one for the a109 release.

OK, so it's not the plan, just a binary release.

When the release focusses on host kit changes, that makes sense. 
This release completely revamps the R3 internal structure. It might 
not work at all, let alone work in a host kit.

Which is a pity.

Why is it a pity that we would want to isolate testing to that which 
changed?

If nothing changed in the hostkit, then an updated hostkit release 
is mostly a matter of bundling updated r3 core libs.

On the contrary, there are bugs in CC which seem to only occur with 
binaries built from the hostkit. So you have a wider base of testing.

My point is, there's no difference. With a hostkit release where 
nothing changed, you still only test the core.

I will remind you of that when you explain to hostkit developers 
that A109 will be a binary-only release for whatever reason is en 
vogue then.

In the meanwhile, A108 will likely be a break-your-code release, 
so watch out.

Won't be able to test it much, without a hostkit.

You will be able to test the A108 changes quite easily without a 
host kit. Extensions can still be loaded externally, and the rest 
is core.

A host kit release has more code than a core release. Why would we 
delay a core release until the non-core host kit code is adapted? 
We don't have to wait to release everything at once, when we can 
get the core release out early to testers. This is especially important 
for releases that focus on core changes.

speaking for myself...  all of my concerns wrt A108 *are* the changes 
in the core wrt how they affect extensions.


The fact that R3 itself is better is cool, but doesn't really allow 
me to test anything if I can't actually compile the host-kit against 
it.


though I know how preparing working releases takes time, so I'm probably 
a bit more patient than Andreas.  ;-)


I will be starting work on my Custom Gob Renderering A107 R3 host-kit 
sometime this week.  

I still have to *complete* the rendering pipeline architecture from 
end to end (even if its currently very shallow), which is about 80% 
done.

most of what is left is the easy part (the actual OpenGL handling 
itself)

I meant to say that I was starting work on the *release* itself (preping 
and cleanup folders of bogus files, and basically uprooting the current 
compilation setup into new clean folders, while doing a little bit 
of diffing on the released A107 to see if any oddities have crept 
in the code.

If the binary-only is a binary built from the hostkit sources with 
the only difference that the r3 libs are statically linked instead 
of dynamically, that is the same code base.

Not *all* of the codebase in a hostkit is used in a binary-only release. 
There is *more* code in a hostkit.

But still needs to be compatible before a host kit release.

(Besides, I got the impression from the R3 GUI team that they have 
A108 pre-releases with graphics enabled. Might have read a bit too 
much between the lines, though.)

The header changes in A108 will be ignored by A107, and the old settings 
will be ignored by A108, so if you want to have transitional code 
you can use both. The only change that I know is not automatically 
adjusted for is the isolate: true header, so if you need that and 
want it to also be compatible with A108 then also include options: 
[isolate]. And you don't need to use type: 'extension anymore, type: 
'module will do (in either version). Don't use the word 'hidden in 
your modules though, at least not at the top level - hidden will 
be a new keyword in module code, like export.

Right, the RT core binaries are built from a *subset* of the same 
codebase. As I explained before.

Still not preventing hostkit releases, unless the subset is really 
not a strict subset.

No, but it does explain why it is possible to get a core release 
done first. What you are asking for is to *delay* the core release 
until the rest of the hostkit has been updated.

Nope, if you can do a core release from the same codebase, you can 
do a hostkit release.

Sorry, english clearly wasn't being understood so I tried to translate 
to a language I know that you are fluent in, REBOL.

A subset. [[core] rest-of-hostkit]. We can test the core first, even 
if they are built from the same overall codebase. The core doesn't 
include any code that is not in the hostkit, but the converse is 
not true.

So a core-only hostkit release should be trivial, no?

Nope. Internal testing has to pass before any release. And the subset 
of code that I know that has passed internal testing is the subset 
used by the core binary. Even a core hostkit release has more code 
to test than a core binary release.

Even a core hostkit release has more code to test than a core binary 
release.

Good, I was concerned for a moment. Then all of your arguments are 
answered.

If Carl indeed builds from the same codebase, then a binary release 
is identical with a core-only hostkit release except for above polish.

For instance, we only have a Windows build for now. A host-kit release 
includes the sources for all supported platforms, even if it doesn't 
include the binaries. And that source would need to be adapted to 
match the new system model to be included in a release. As of when 
last I checked, we hadn't adapted the Linux-specific stuff yet. No 
changes may be needed, but without test builds, internal testing 
of those builds, and debugging we can't say for sure that the extra 
code in the host kit release is of even alpha release quality, no 
matter if we mark it as experimental.

A host-kit release includes the sources for all supported platforms


Yes, except that there are separate hostkit releases for each platform. 
In a given hostkit release, the sources for all platforms but the 
one matching the release are dysfunctional.

Please redownload the Windows build. An obscure bug and a last-minute 
convenience fix were just added.

Should be easy using a simple custom dialect.

Makes more sense to draft this in a simple function first, imho.

Not likely in direct syntax for two reasons:
- P* is a valid word itself.

- Word lookup is done by pointer comparison, not string comparison.

so I could use a function that traverses the object doing string 
comparisons and returns the full path for me so I don't have to keep 
typing it

Yes, that's a case where the testsuite needs updating.

Yes. Would maybe make sense to keep it around for a while to warn 
that it is now gone.

We did that for more than a year. Now it is gone.

It only had a deprecation notice, no hard error (like e.g. LOAD/next 
has now).

The test on line 51 is a bug. See bug#1471, fixed in a108.