World: r3wp

[ This whole post is mainly in the R2 mindset, but is somewhat relevant 
also to R3. ]


The technique of setting words directly into the global context like 
this:

	context [
		set 'o ...
	]


I do not like, because this code is modifying its environment - it 
has side-effects - thus, it is non-modular, and does not scale.

Being non-modular means large software systems can't be built up 
reliably, because the cumulative effect of all those side-effects 
will eventually cause name-clashes in the global context, so that 
some word exported from one context will be overwritten by the same 
word exported from another context.

For example, suppose I've seen two graphics demos by two different 
authors. They each have an awesome graphics processing routine and 
I decide that I would like to combine both routines into a single 
program. They each wrapped their code up in a context and exported 
a word 'process-image, like so:

	; From author #1
	context [
		set 'process-image does ...
	]

	; From author #2
	context [
		set 'process-image func ...
	]


You can imagine that each of these "modules" also has a large amount 
of supporting code, local variables and functions, and each is in 
a large file on the respective author's website. Somewhere in the 
middle of each of these files, in the CONTEXT body code, are the 
SET lines which export the words into the global context.

When I write my program to combine both of these "modules", I will 
probably write code like:

	; Acquire all dependencies
	do %image-processor.r  ; By author #1
	do %super-gfx.r  ; By author #2

	; Create an image and manipulate it.
	my-image: make image! 400x400
	process-image my-image ...


and here I realise that there is a name-clash with the 'process-image 
word, which is set only to the value exported by the second author.
So what do I do?

Here are some theoretical approaches, which have their own problems.

1) I could reload each file just before use:

	do %image-processor.r  ; By author #1
	process-image my-image ...
	do %super-gfx.r
	process-image my-image ...


  Each "module" is not expecting to be used this way, so this has problems 
  like:

  - "Static" locals which are intended to remain in memory will be 
  lost each time the file is reloaded.

  - Performance could suffer; each file could be large, and many calls 
  to 'process-image might be done.


2) I could set the first imported word to my own chosen word before 
importing the second "module". eg

	do %image-processor.r  ; By author #1

 process-image2: :process-image  ; Create an alias, as 'process-image 
 will be overwritten next line.
	do %super-gfx.r  ; By author #2

	; Now use
	process-image2 my-image ...
	process-image my-image ...


  But this means that a line of code has been created in the dependency 
  acquisition stage which has a complex interdependence between the 
  two "modules". They are not independent, and so individual dependency 
  acquisition lines can't be easily copied from this code and pasted 
  into a new script and expected to work right away. If copy/pasted, 
  the code will have to be examined, probably in great detail, to discover 
  what's going on and how to make it work. This will lead right back 
  into each source file, where the SET lines which export words to 
  the global context must be found. What great fun that will be in 
  a large software system built using many modules.


Another approach could be to try to bind each module code to a new 
context which contains the exported words, so they are isolated from 
each other... but this is complex.


All the above approaches are attempting to work around a single problem: 
that each "module" is exporting words where and when it likes, without 
consideration for the environment (other "modules", other global 
words etc.) This is "global namespace pollution" and the approaches 
above just introduce more problems in trying to work around it.


The solution to all this, is, in my view, for modules to declare, 
in the script header, the words that are intended to be exported, 
but for the module code not to actually perform the exports. This 
should be done by the user code, at its option. If a large module 
provides an API of 10 functions, then those function words should 
not be forced into the global context. The user script should be 
able to choose which, if any, of those words to import, and into 
which context it would like to import them. Additionally, the exported 
word value should be importable to a differently-named word, which 
can solve the name-clash problem we have above.

If modules do not use SET, but instead declare their "export" words 
in the script header, then digging through code to find side-effects 
should no longer be necessary. In R2, this requires that all module 
authors adhere to this type of module system, and declare their "export" 
words in a standard fashion.


In R3, I'm hoping the module system will develop into one which can 
enforce the modularity of modules, so that a user script can be sure 
that there were no side-effects introduced by loading any module.

Some comments about R3:
- R3 modules have the explicit Exports header in the spec

- IMPORT/only doesn't export the words into your current context, 
it just returns a module! reference.

- LOAD module! doesn't DO any of the code in a module, so you can 
examine the spec with SPEC-OF module!.

- If you import a module the first time with IMPORT/isolate then 
all of the words in that module will be local, even the SET 'word 
ones.

So why not do something like this?
REBOL []
[
   process-image: func ....
]
................
image-processor: make object! load %image-processor.r
super-gfx: make object! load %super-gfx.r
image-processor/process-image foo
super-gfx/process-image foo


Static locals stay in memory, there's no extra steps to deal with 
conflicts, no special voodoo..  And, whenever process-image is called, 
it's obvious from where.

BrianH, it looks like R3 module system has the basics really well 
designed.

Try this:
REBOL [type: module]
process-image: func ....

image-processor: import/only %image-processor.r
...

(that was to Izkata)

First I thought that those refinements are not important, because 
- in fact you should specify such things in module header, but am 
I correct thinking, that refinements provide you kind of security? 
Do they override some internal module settings?

Anton, I think it still needs some work, but it is usable as of alpha 
42 (when my DO fix went in). We can make it better later.

anton, slim doesnt expose ANY words in the global context.  module 
interdependency is assured even if they each need each other's code. 
 and the infrastructure is embeded, where its the code LOADING the 
module which decides what to include.  by declaring words in a module, 
that uses another module, you don't even expose those words.

this is available in R2 for 5 years now.

slim/load 'module-name version 


module-name is searched for in search paths and must match header 
name, or its ignored.

slim/load/expose 'module version [a-function another-function]


you can even rename the funcs as you are exposing them to enforce 
code to cooperate:


slim/load/expose 'module version [[module-name my-own-name] [other-module-name 
my-other-name]]

Brian - will modules allow some kind of protect mode? E.g. I want 
to have some func or word value to not be seen from external environment? 
So I don't want it to be accessible even by path notation?

Izkata, that's not a bad approach, but it has these problems:

1) LOADing a module is not quite the same as DOing it. DO sets up 
the current directory and system/script object correctly. LOAD doesn't, 
so the module might not be able to inspect itself and know about 
its location etc.

2) In trying to avoid setting words in the global context, you're 
setting words in the global context. Now you must use paths to get 
to what you want. This should be at the option of the user script. 
Obviously, you're exercising that option in your example. You could 
also do it this way:

 process-image1: get in context load %image-processor.r 'process-image
	process-image2: get in context load %super-gfx.r 'process-image

for me the ONLY advantage of R3 modules is the fact that we seem 
to be able to enforce modules NOT to expose anything... since they 
are closed contexts.  I obviously cannot enforce this in R2, but 
all the management is aready there.

... and also - we now have 'export functionality. Will we allow 'import, 
to import some values into the module?

Pekr, the refinements override module header settings, and are used 
when you need to override default behavior, or when you don't quite 
trust the source.

import/only - nice..  Looks like it's doing some weird things, but 
I'm not used to R3

Access control will be added at the object! level - module! contexts 
are object! contexts.

Yes, import/isolate does look really nice.

slim also has an auto-prefix option when you import words.  so as 
to make "collections" easy to differentiate, without having to rename 
each one separately.

I wanted it by default, but the REBOL standard is to do the easy 
thing by default and the advanced thing with /only.

there are a lot more tweaks like that... all working and being used 
for years by me... but everyone has always complained that REBOL 
has to stay simple... funny now that R3 comes out everyone is crying 
for features that have been available to all of you for years... 
 :-/

Maxim, ok, that's Slim, your software. I was just basically explaining 
to Steeve why I would strongly discourage using SET to export words.

Anton - 1 just takes a do [] within the block, but I see what you 
mean, it is less intuitive

Maxim, I write modular code in R2 as well, though with Gabriele's 
modules, not Slim.

all I'm saying is that we've had a working model which does all of 
proposed R3 tricks for years.

(a part from "enforcing" the closed nature of the context)

And Maxim, the actual software used to import the module isn't so 
important, as how the module is written (in R2, anyway, which has 
no inbuilt module system). If the module makes side-effects, then 
it's not really modular, is it?

And your point is? Of course we could do it before. The question 
is how we *should* be doing it *in the future*.

(Isn't so important - for the point I was trying to make, anyway.)

Brian, I am with you, in wanting the isolating behaviour of module 
import by default, in R3.

my point is.... hehe ... that the community has always suffered from 
the "approved by god" problem... until god gives us the approval, 
nothing seems to take off in rebol... and I'm not talking about only 
my stuff here.

The module code in R3 is in my area, so I will be tweaking it to 
be as close to perfect as I can. And I'll be backporting as much 
of R3's module system to R2 as is possible, as part of the R2-Forward 
project.

Just curious Maxim...  Would SLIM properly catch the following code?

hax: does [ do load {set 'global-leak! "Mwahahaha!"}]

>:)

Lawnmower Man: "I am God here!"

Ok, Ammon, no module system in R2 can catch such code.

no cause it can't do it technically which is what I say is the advantage 
of R3 is...

we could do code inspection, but its pointless cause its easy to 
circumvent anyways.

brian: maybe a lot of the work for F2 forward already exists we could 
work together on this, if you want. ;-)

(*R2 Forward)

You mean, you can't do code inspection, because it's impractical, 
and can't be (achievably) automated.

The word!, LOAD and BIND changes in R3 make it *much* easier to catch 
stuff.

and I second the motion, that modules should be 100% tight closed 
by default.  its just a better habit to have by default... if god 
is going to talk to the pupils.

let the burden on importing stuff, rather than trying to box in the 
words trying to creep out by default.

R2-Forward is mostly done, at least as of R3 circa a month ago.

I have default expose of words... and frankly ,in 5 years and probaly 
about 200 libs, I've never used it once.

(in slim)

cause its impractical

Maxim, do you have a user in R3 chat? Otherwise I can post the original 
debug source of LOAD (not even posted there).

load: func [
	{Loads a file, URL, or string.}

 source [file! url! string! binary! block!] {Source or block of sources}

 /header  {Includes REBOL header object if present. Preempts /all.}

 /next    {Load the next value only. Return block with value and new 
 position.}

;	/library {Force file to be a dynamic library. (Command version)}
;	/markup  {Convert HTML and XML to a block of tags and strings.}
	/all     {Load all values. Does not evaluate REBOL header.}
	/unbound {Do not bind the block.}
	/local data content val rst tmp

][  ; Note: Avoid use of ALL and NEXT funcs, because of /all and 
/next options
	content: val: rst: tmp: none ; In case people call LOAD/local
	
	; Retrieve the script data
	data: case [
		block? source [ ; Load all in block
			return map x source [apply :load [:x header next all unbound]]
		]
		string? source [source] ; Will convert to binary! later
		binary? source [source]
		; Otherwise source is file or url
		'else [
			; See if a codec exists for this file type
			tmp: find find system/catalog/file-types suffix? source word!
			; Get the data, script required if /header
			content: read source  ; Must be a value, not unset
			case [
				binary? :content [content] ; Assumed script or decodable
				string? :content [content] ; Assumed script or decodable
				header [cause-error 'syntax 'no-header source]
				block? :content [content]
				'else [content: reduce [:content]]
			] ; Don't LOAD/header non-script data from urls and files.

  ] ; content is data if content doesn't need copying, or none if it 
  does
	]
	;print [1 "data type?" type? :data 'content true? :content]
	if string? :data [data: to-binary data] ; REBOL script is UTF-8

 assert/type [data [binary! block!] content [binary! string! block! 
 none!]]
	assert [any [binary? :data not header]]
	if tmp [ ; Use a codec if found earlier
		set/any 'data decode first tmp :data

  ; See if we can shortcut return the value, or fake a script if we 
  can't
		case [

   block? :data [if header [insert data val: make system/standard/script 
   []]]

   header [data: reduce [val: make system/standard/script [] :data]]

   (to logic! unbound) and not next [return :data] ; Shortcut return

   any [next any-block? :data any-word? :data] [data: reduce [:data]]
			'else [return :data] ; No binding needed, shortcut return
		]
		assert/type [data block!] ; If we get this far
	]
	;print [2 'data mold to-string :data]
	
	if binary? :data [ ; It's a script
		unless find [0 8] tmp: utf? data [ ; Not UTF-8
			cause-error 'script 'no-decode ajoin ["UTF-" abs tmp]
		]
		; Process the header if necessary
		either any [header not all] [
			if tmp: script? data [data: tmp] ; Load script data
			; Check for a REBOL header
			set/any [val rst] transcode/only data
			unless case [
				:val = [rebol] [ ; Possible script-in-a-block
					set/any [val rst] transcode/next/error rst
					if block? :val [ ; Is script-in-a-block
						data: first transcode/next data
						rst: skip data 2
					] ; If true, val is header spec
				]
				:val = 'rebol [ ; Possible REBOL header
					set/any [val rst] transcode/next/error rst
					block? :val ; If true, val is header spec
				]
			] [ ; No REBOL header, use default
				val: [] rst: data
			]
			; val is the header spec block, rst the position afterwards

   assert/type [val block! rst [binary! block!] data [binary! block!]]
			assert [same? head data head rst]
			; Make the header object

   either val: attempt [construct/with :val system/standard/script] 
   [
				if (select val 'content) = true [
					val/content: any [:content copy source]
				]
			] [cause-error 'syntax 'no-header data]
			; val is correct header object! here, or you don't get here
			; Convert the rest of the data if necessary and not /next
			unless any [next block? data] [data: rst: to block! rst]
			if block? data [ ; Script-in-a-block or not /next
				case [

     header [change/part data val rst] ; Replace the header with the object

     not all [remove/part data rst]	; Remove the header from the data
				]
				rst: none ; Determined later
			]
		] [rst: data] ; /all and not /header
	]

 ; val is the header object or none, rst is the binary position after 
 or none

 assert/type [val [object! none!] rst [binary! none!] data [binary! 
 block!]]

 assert [any [none? rst same? head data head rst] any [val not header]]

 ;print [3 'val mold/all :val 'data mold/all :data "type?" type? :data]
	
	; LOAD/next or convert data to block - block either way
	assert [block? data: case [
		not next [ ; Not /next
			unless any [block? data not binary? rst] [data: to block! rst]
			data
		]
		; Otherwise /next

  block? data [reduce pick [[data] [first+ data data]] empty? data]
		header [reduce [val rst]] ; Already transcoded above
		binary? rst [transcode/next rst]
	]]
	
	; Bind to current global context if not a module
	unless any [ ; Note: NOT ANY instead of ALL because of /all
		unbound
		(select val 'type) = 'module
	][
		bind/new data system/contexts/current
	]
	;print [6 'data mold/all :data 'tmp mold/all :tmp]
	
	; If appropriate and possible, return singular data value
	unless any [
		all header next  ; /all /header /next
		empty? data
		1 < length? data
	][set/any 'data first data]
	;print [7 'data mold/all :data]
	
	:data
]

you can't really catch all possible variations of things like: do 
reduce [ to-set-word "tadam"  44]

You can since R3 words are not bound by default.