r3wp [groups: 83 posts: 189283]
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search
 

World: r3wp

[!REBOL3 Modules] Get help with R3's module system

BrianH
22-Oct-2010
[107x4]
I have also started writing some simple charts to explain the details 
of the design and behavior of the module system. In CSV format. These 
charts helped a lot in the fixing of the problems and implementation 
of the tests. As with the tests, I will try to get the charts published 
somewhere official.
With alpha 109 we got some significant usability revisions to the 
design of the module system, relative to alpha 108:

- The return of unnamed modules. They are now changed to private 
modules (mixins) which aren't stored in the system modules list.

- IMPORT now effectively works a lot like the Needs header in user 
scripts. Most users won't be able to tell the difference.

- The return value of IMPORT block is now a block of the modules 
you imported (but not the modules *they* imported).

- The refinements of IMPORT have been renamed and their behavior 
tweaked to be nicer and more useful - the first API change since 
Carl's original.
  - /no-share: The previous /isolate option. Same behavior.

  - /no-lib: Don't export to the runtime library. Private modules don't 
  do this anyways. Also, don't add to the system modules list.

  - /no-user: Don't export to the user context, even as a private module. 
  When importing to a module, /no-user applies.

- The old /only option was split into /no-lib and /no-user, for more 
control. Specify both if you don't want IMPORT to export anything.

Alpha 110 should bring these changes:

- The above will work properly. With a bunch of specs and charts 
that define what "properly" means. With a full test suite to make 
sure.
For the sake of completeness, here are the highlights of the alpha 
108 changes:

- Script headers can have an options block, a simple block of flag 
words. User extensible.

- The standard script header now has a lot fewer words in it. More 
stuff is optional or in the options block.

- Script compression, either binary and base 64 binary! encoded. 
Automatic, transparent.

- Script checksums, both to verify the script and for IMPORT to compare 
with. Applies to decompressed source.

- An optional script length header field (like http's Content-Length). 
This allows binary script embedding.

- Internal support for getting the end of an embedded script, so 
a multi-loader is possible.

- The 'content and 'isolate header fields are changed to option words. 
The content is still saved to a 'content header field.

- The 'content field, if set, is set to the start position of the 
script proper, even if there is stuff before it.

- The whole system/contexts/system concept is gone, as part of the 
system restructuring. Now we have SYS.

- The system/contexts/exports concept is gone too, replaced by a 
not-module-specific runtime library called LIB.

- The old type: 'extension is now the 'extension header option word. 
The only module type is 'module. And it's optional for most code.

- Mixins are now called "private modules", and are flagged by the 
'private option word. And they can have names.

- Private modules can be added to the system modules list (because 
of the names). This lets them be reused without being reloaded.

- Unnamed modules are now prohibited (until alpha 109, where they 
become private modules that reload every time).

- Delayed modules, which can be partially loaded and then not fully 
made until they are imported. Use the 'delay option word.

- A HIDDEN module source keyword, which applies PROTECT/hide to a 
word or words. Acts like the EXPORT keyword.

- Better errors are triggered when the bad things happen. Including 
new error codes.

- DO and MAKE--MODULE intrinsics are now in sys, as DO* and MAKE-MODULE*. 
No more system/intrinsics.

- DO-NEEDS is no longer exported (it's in sys). IMPORT block is a 
public alias for DO-NEEDS anyways.

- MODULE now makes modules that act more like those in script files. 
And has /mixin support too.

- A whole bunch of changes and fixes to native functions to support 
the above stuff.
Shadwolf's "used by 3 guys around the world" comment brings to mind 
one of the more ironic things about the module system:


Most user code for R3 will be written in "scripts", not "modules". 
This will be even more the case once we get more of concurrency working, 
because "script" code works in the user context, which will be task-local. 
We are going out of our way to make it extremely easy to just use 
"scripts" and not have to bother with "modules".


The ironic part is that "scripts" are just another kind of module, 
one of the three including regular and isolated modules. In particular, 
user scripts are a kind of module that we try to make as non-module-like 
as it is possible to be (given that they run in a module system). 
The entire module system structure is built around the challenge 
of making the module system apparently disappear, or at least be 
something that you can be almost completely ignorant of. The module 
system is built for script programmers, to let people do PITS on 
a systerm that they don't even have to know is capable of the most 
advanced PITL.


So the module system we are discussing here will be used by *everyone 
who programs in R3*, whether they know it or not :)

(I am politely assuming that Shadwolf was not referring to the entire 
REBOL community when he said "3 guys".)
GrahamC
22-Oct-2010
[111x2]
can modules be released as encrypted code ?
Not much point I guess
BrianH
22-Oct-2010
[113]
Not yet, but that is planned. It's even mentioned in the design wiki.
GrahamC
22-Oct-2010
[114]
are modules signed?
BrianH
22-Oct-2010
[115]
Cryptographic signing is also planned, as also mentioned in the wiki. 
Both will work in a similar way to the compression and checksum, 
respectively.
GrahamC
22-Oct-2010
[116]
sha256 ?
BrianH
22-Oct-2010
[117x2]
We plan to do encryption and signing. We aren't far enough along 
in the plan to know how we will do these.
The checksum is CHECKSUM/secure, whatever algorithm that is.
Sunanda
22-Oct-2010
[119]
Checksum/secure is SHA-1
BrianH
22-Oct-2010
[120x2]
In theory, you could generate R3 scripts using R2 and still have 
most of the same features. Loading them would be trickier though.
SAVE is pretty straightforward and doesn't use any R3-specific features.
GrahamC
22-Oct-2010
[122]
Is the cryptography being rehashed from R2 ?
BrianH
22-Oct-2010
[123]
Rehashed :) No, we're using the old hashes :)
GrahamC
22-Oct-2010
[124]
It would be good if certificate use could be more easily enabled
BrianH
22-Oct-2010
[125x4]
I can't say if that will be the case with the encryption or signing 
though.
Certificate use is something R3 doesn't do well yet, afaik (which 
isn't far). We will likely have to do a lot of infrastructure work 
before we can do encryption or signing.
Nonetheless, this is something we want (need?) to do, so the crypto 
infrastructure work will need to be done.
I forgot to mention the LOAD and SAVE option changes in alpha 108. 
Perhaps later after I sleep, if their HELP is insufficient.
Andreas
22-Oct-2010
[129x3]
Most user code for R3 will be written in 'scripts', not 'modules'.

Lets hope that this prediction will turn out to be wrong.
Thanks for the summary, Brian.
Need to find some time to play with it first, but it sounds like 
"private" modules and/or IMPORT/no-lib/no-user will be most useful.
Gregg
22-Oct-2010
[132]
The charts will help. I didn't have a problem with /isolate and /only 
as names. The new ones sound rather clunky at first glance, but I 
may grow to like them. Most important, I think, are small examples 
demonstrating why we have the various options. I hope the charts 
are the first step on that path.

Thanks Brian!
BrianH
22-Oct-2010
[133x2]
The new option names are a good choice because all of the /no-* options 
are a way of breaking modules that otherwise depend on acting the 
way that they were written. The whole group of them needs a warning 
label.
Andreas, for library code you are correct, but most end-developers 
just use library code, they don't write so much of it.
Andreas
22-Oct-2010
[135x2]
My hope is that eventually R3 will be used for more than just small 
and dirty throw-away scripts. And that'll directly induce the need 
for a module system.
My hope is to never, ever come across a "do %..." that "loads" utility 
functions again (in R3).
BrianH
22-Oct-2010
[137]
There are certain advantages to writing user scripts for the front 
end code:

- The user context will be task-local, so a lot of the multitasking 
problems will go away.

- The user context is shared by all scripts running in the same task. 
It acts like a shared isolated context.

- User scripts have fewer settings in their headers. For that matter, 
they can have no header at all. Every command you enter into the 
command line is a separate script, for instance.
Andreas
22-Oct-2010
[138]
None of those sound like an advantage to me. For the "settings in 
their headers" I'll once again have to punt until I look into the 
module system more closely.
BrianH
22-Oct-2010
[139]
Header settings that affect modules only:
- type: module
- name: blah
- version: 1.0.0
- exports: [...]
- options: [isolate]
- options: [private]
- options: [extension]

More than one option is combined in one block. The rest of the header 
stuff affects scripts too.
Andreas
22-Oct-2010
[140x2]
What's the minimal module header?
REBOL [type: module name: foo] ?
BrianH
22-Oct-2010
[142]
REBOL [type: module]. Or if you are loading it with IMPORT or the 
Needs header, REBOL [].
Andreas
22-Oct-2010
[143]
Well, then that's how it should be :)
BrianH
22-Oct-2010
[144]
Regular modules need a name though. Named modules don't get reloaded 
if you import them more than once, they get reused. There are all 
sorts of tricks you can do with that.
Andreas
22-Oct-2010
[145]
system/contexts/user still refers to the task-local user context?
BrianH
22-Oct-2010
[146x7]
Yup. Or in scripts, you can call it self.
That task-local thing is still a plan though. The code is designed 
with that assumption, but it's not yet true.
Technically, you can import modules with no header at all, though 
not DO them. Extension modules require a header though.
Of course, headers let you do all sorts of tricks that you can't 
do without them. In addition to the above stuff, header settings 
let you:

- Embed scripts in text or binary files, even if it's just documentation 
before the script header.
- Aggregate multiple scripts/modules in one file.
- Save and verify a script/module checksum.
- Compress scripts/modules.
Just double-checked. No, you need at least REBOL [] for all modules, 
extension or not.
Andreas, the (dis)advantage to private modules is that they need 
to be explicitly imported into your module for you to get their exports 
(selective import). For regular modules, you import into the runtime 
library once and it is just there to share. Regular modules can take 
advantage of this to support upgrading themselves in place, for instance, 
or more easily managing word overrides on a global basis; private 
modules can't be upgraded. Each approach has advantages and disadvantages 
in different situations. This is why we support both export models.
The differences between the two are less if you are importing to 
user scripts. Since scripts share a context, subsequent scripts can 
benefit from the effects of modules imported by previous scripts, 
whether they were private or not.
Andreas
22-Oct-2010
[153]
For me that's purely an advantage. Explicitly importing the modules/functions 
you need is how it should be.
BrianH
22-Oct-2010
[154]
There are real downsides to explicit import. Choose what works for 
your situation.
Andreas
22-Oct-2010
[155]
I'm also pretty sure that upgrading "private" modules is possible.
BrianH
22-Oct-2010
[156]
For instance, with explicit import you can't upgrade a running process. 
That can be a disadvantage for some apps.