AltME groups: search

The disambiguity in the current rules is simple: Only a version-with-no-name 
at the beginning is treated as a system version.

There is no key-value format, there is a key opt-val-1 opt-val-2 
format. Needs is processed with PARSE.

Yes, poor wording on my part. The point being that there is an identifier 
before the version and other information related to a module, and 
the system version is an implicit exception.

Yup. But it's always at the beginning. There's another exception 
where Needs can just be a tuple, and only the system version is checked.

It turns out that there is no 'system or 'rebol module - it's a chicken-vs-egg 
thing. If it is a module, it can be overriden, so it really does 
need to be special-cased. I don't like that there is a 'core optional 
keyword, because there can also be a 'core module and that isn't 
screened for. But the 'core keyword is there for backwards-compatibility.

I could screen for it in DO-NEEDS, but that wouldn't help with everywhere 
else the modules list is accessed. It would really be better to not 
have any optional keyword at all, but there's that backwards-compatibility 
thing. We'll have to resolve this before we have a real R3 release.

For now, you can have a module named 'core, but you might run into 
gotchas when trying to use it.

If you are adding a module to the module list, and there is an existing 
module of that name, then the new module either overrides it, replaces 
it, or doesn't get added (possibly with an error triggered, but so 
far not). The question is which one to do in the particular circumstances. 
The factors are whether it is the same module, for whatever "same" 
means here considering it might be reloaded or still source; whether 
the versions are the same or greater; whether the existing module 
has already been made or is still source, and the same for the module 
to be added.

So far, my guess is that

- Premade modules can't be delayed since their code blocks and side 
effects have already been executed.

- Delayed modules can't be added unless they have at least the version 
of an existing delayed module, or more than an existing imported 
module.

- Premade modules can't be added unless they have at least the version 
of an existing delayed module, or more than an existing imported 
module.

- If a overriding module is added and the existing module is delayed, 
the existing reference should be replaced, not overriden.

- If a overriding module is added and the existing module is already 
imported, the new module overrides the old but the old reference 
is still there.

- If a delayed module is a mixin, the module is made and returned, 
but the stage-two delayed source is kept in the list.
- I'm missing something.

The questions I have are:

1. What do we do when a module is not added due to a policy issue? 
Currently the add accessor returns none if it is a version issue, 
and triggers an error for a checksum violation.

2. How do we determine (officially) that two modules are to be considered 
the same? Name and version?

2. Can we safely LOAD-EXTENSION more than once with the same extension?

3. Does LOAD-EXTENSION on an embedded extension have any side-effects 
beyond creating an object?

4. ... return the same source each time, or different copies of the 
same source? Testable by SAME?

5. Is is safe to delay the object returned by LOAD-EXTENSION instead 
of the source?

6. Should the checksum of an extension include the extension-specific 
source added?

7. Should the version in the header of a module be set to 0.0.0 if 
not a tuple? Currently it is.

8. If so, should module checksums be done after the version field 
is fixed?

I haven't added documentation for the individual functions yet, since 
they are not yet done. Well, most are, but there might be changes 
required to implement the rest. That is a guru-level doc, but it 
could be used as source material for user-level docs.

I'll be done before then. Yeah, a lot of the proposed object enhancements 
have been done to modules instead. Objects will likely still get 
enhanced in some ways, but modules can better handle a lot of what 
objects were previously used for.

Bundle the modules as what Carl now calls "optionally included". 
Also keep a list of scheme prefix to module name, and just auto-import 
the module from this list when a scheme is used in on of the scheme 
action functions (READ, OPEN, ...).

A regular, non-delayed module could include a delayed module that 
does the real work. And there are other tricks that can be done.

Well, I personally see not cluttering the "global" namespace with 
those names as a great advantage.

A lot of those names won't need to be exported at all. Internal use 
words can stay internal.

OK, the third rewrite of the module system has a few simplifications 
as related to the (unreleased) second rewrite, but it is mostly the 
same design. I will be revising the wiki above with the changes.

That is a separate discussion.

so much of a thing used by 3 guys around the world. That impress 
me.

hum that was the cas before too no? You will need luck that's a sure 
thing.

It wasn't a request for assistance. If you want to help, it will 
be welcomed. If not, we are doing well now. Either way you are free 
to use what we create, or not. Enjoy!

Over the last day I have been fleshing out the test suite for the 
new module system. Almost all features have corresponding tests now. 
As soon as we have complete coverage I will try to get the tests 
published somewhere official. FYI, alpha 109 fails a lot of the tests, 
but also in the last day I fixed all of the errors that were causing 
test failures. In theory - I need a new build with the fixes to be 
sure, but code tracing says the behavior matches the tests. In any 
case, the test suite will have full feature coverage before alpha 
110 comes out.

I have also started writing some simple charts to explain the details 
of the design and behavior of the module system. In CSV format. These 
charts helped a lot in the fixing of the problems and implementation 
of the tests. As with the tests, I will try to get the charts published 
somewhere official.

With alpha 109 we got some significant usability revisions to the 
design of the module system, relative to alpha 108:

- The return of unnamed modules. They are now changed to private 
modules (mixins) which aren't stored in the system modules list.

- IMPORT now effectively works a lot like the Needs header in user 
scripts. Most users won't be able to tell the difference.

- The return value of IMPORT block is now a block of the modules 
you imported (but not the modules *they* imported).

- The refinements of IMPORT have been renamed and their behavior 
tweaked to be nicer and more useful - the first API change since 
Carl's original.
  - /no-share: The previous /isolate option. Same behavior.

  - /no-lib: Don't export to the runtime library. Private modules don't 
  do this anyways. Also, don't add to the system modules list.

  - /no-user: Don't export to the user context, even as a private module. 
  When importing to a module, /no-user applies.

- The old /only option was split into /no-lib and /no-user, for more 
control. Specify both if you don't want IMPORT to export anything.

Alpha 110 should bring these changes:

- The above will work properly. With a bunch of specs and charts 
that define what "properly" means. With a full test suite to make 
sure.

For the sake of completeness, here are the highlights of the alpha 
108 changes:

- Script headers can have an options block, a simple block of flag 
words. User extensible.

- The standard script header now has a lot fewer words in it. More 
stuff is optional or in the options block.

- Script compression, either binary and base 64 binary! encoded. 
Automatic, transparent.

- Script checksums, both to verify the script and for IMPORT to compare 
with. Applies to decompressed source.

- An optional script length header field (like http's Content-Length). 
This allows binary script embedding.

- Internal support for getting the end of an embedded script, so 
a multi-loader is possible.

- The 'content and 'isolate header fields are changed to option words. 
The content is still saved to a 'content header field.

- The 'content field, if set, is set to the start position of the 
script proper, even if there is stuff before it.

- The whole system/contexts/system concept is gone, as part of the 
system restructuring. Now we have SYS.

- The system/contexts/exports concept is gone too, replaced by a 
not-module-specific runtime library called LIB.

- The old type: 'extension is now the 'extension header option word. 
The only module type is 'module. And it's optional for most code.

- Mixins are now called "private modules", and are flagged by the 
'private option word. And they can have names.

- Private modules can be added to the system modules list (because 
of the names). This lets them be reused without being reloaded.

- Unnamed modules are now prohibited (until alpha 109, where they 
become private modules that reload every time).

- Delayed modules, which can be partially loaded and then not fully 
made until they are imported. Use the 'delay option word.

- A HIDDEN module source keyword, which applies PROTECT/hide to a 
word or words. Acts like the EXPORT keyword.

- Better errors are triggered when the bad things happen. Including 
new error codes.

- DO and MAKE--MODULE intrinsics are now in sys, as DO* and MAKE-MODULE*. 
No more system/intrinsics.

- DO-NEEDS is no longer exported (it's in sys). IMPORT block is a 
public alias for DO-NEEDS anyways.

- MODULE now makes modules that act more like those in script files. 
And has /mixin support too.

- A whole bunch of changes and fixes to native functions to support 
the above stuff.

Shadwolf's "used by 3 guys around the world" comment brings to mind 
one of the more ironic things about the module system:


Most user code for R3 will be written in "scripts", not "modules". 
This will be even more the case once we get more of concurrency working, 
because "script" code works in the user context, which will be task-local. 
We are going out of our way to make it extremely easy to just use 
"scripts" and not have to bother with "modules".


The ironic part is that "scripts" are just another kind of module, 
one of the three including regular and isolated modules. In particular, 
user scripts are a kind of module that we try to make as non-module-like 
as it is possible to be (given that they run in a module system). 
The entire module system structure is built around the challenge 
of making the module system apparently disappear, or at least be 
something that you can be almost completely ignorant of. The module 
system is built for script programmers, to let people do PITS on 
a systerm that they don't even have to know is capable of the most 
advanced PITL.


So the module system we are discussing here will be used by *everyone 
who programs in R3*, whether they know it or not :)

(I am politely assuming that Shadwolf was not referring to the entire 
REBOL community when he said "3 guys".)

Cryptographic signing is also planned, as also mentioned in the wiki. 
Both will work in a similar way to the compression and checksum, 
respectively.

Certificate use is something R3 doesn't do well yet, afaik (which 
isn't far). We will likely have to do a lot of infrastructure work 
before we can do encryption or signing.

The charts will help. I didn't have a problem with /isolate and /only 
as names. The new ones sound rather clunky at first glance, but I 
may grow to like them. Most important, I think, are small examples 
demonstrating why we have the various options. I hope the charts 
are the first step on that path.

Thanks Brian!

The new option names are a good choice because all of the /no-* options 
are a way of breaking modules that otherwise depend on acting the 
way that they were written. The whole group of them needs a warning 
label.

My hope is that eventually R3 will be used for more than just small 
and dirty throw-away scripts. And that'll directly induce the need 
for a module system.

My hope is to never, ever come across a "do %..." that "loads" utility 
functions again (in R3).

There are certain advantages to writing user scripts for the front 
end code:

- The user context will be task-local, so a lot of the multitasking 
problems will go away.

- The user context is shared by all scripts running in the same task. 
It acts like a shared isolated context.

- User scripts have fewer settings in their headers. For that matter, 
they can have no header at all. Every command you enter into the 
command line is a separate script, for instance.

Regular modules need a name though. Named modules don't get reloaded 
if you import them more than once, they get reused. There are all 
sorts of tricks you can do with that.

That task-local thing is still a plan though. The code is designed 
with that assumption, but it's not yet true.

Technically, you can import modules with no header at all, though 
not DO them. Extension modules require a header though.

Of course, headers let you do all sorts of tricks that you can't 
do without them. In addition to the above stuff, header settings 
let you:

- Embed scripts in text or binary files, even if it's just documentation 
before the script header.
- Aggregate multiple scripts/modules in one file.
- Save and verify a script/module checksum.
- Compress scripts/modules.

Andreas, the (dis)advantage to private modules is that they need 
to be explicitly imported into your module for you to get their exports 
(selective import). For regular modules, you import into the runtime 
library once and it is just there to share. Regular modules can take 
advantage of this to support upgrading themselves in place, for instance, 
or more easily managing word overrides on a global basis; private 
modules can't be upgraded. Each approach has advantages and disadvantages 
in different situations. This is why we support both export models.

The differences between the two are less if you are importing to 
user scripts. Since scripts share a context, subsequent scripts can 
benefit from the effects of modules imported by previous scripts, 
whether they were private or not.

For instance, with explicit import you can't upgrade a running process. 
That can be a disadvantage for some apps.

Part of exporting is copying the values to another context, where 
it is used. You don't normally get any references to the original 
module words. And part of importing is copying those words again 
to your own context (for isolated modules and for scripts), or binding 
to the runtime library. So in practice, the only known contexts that 
you can update the values in are your own, the runtime library, and 
the current task's user context. To upgrade other contexts they would 
need to register with you, and you would have to do them one at a 
time.

REBOL processes tend to run for years, if they don't have bugs and 
don't use a buggy REBOL. Do you remember the first mailing list outage?

Too good a feature to call it a workaround.

The first REBOL mailing list outage happened because the server rebooted. 
The REBOL process that was running the mailing list had been started 
manually, and just kept running. They forgot to add the startup code 
for the mailing list to the init code for the server. But noone knew 
this because the process just kept running for more than 3 years. 
It took a UPS failure to stop the list.

OK, most of the discussion of the new module system so far has been 
in a private world. And the final working system hasn't been released 
yet. And the documentation is outdated. So pardon me if I say that 
you should reserve judgement. But still, being to remove the module 
system with a boot level is a pretty cool feature :)

Be shocked, I guess. Actually, I was just shocked: The mod-test.r 
file only has 696 lines in it at the moment. I guess this is a testament 
to how compact we can make REBOL scripts with the new system :)

Note that -b base is not useful for you (it's for me) because schemes 
are not yet init'd.  It's a bit like booting an OS without the file 
system.

Also, -b sys is somewhat useful, because there is a very minimal 
LOAD that should work.

Next week, as time allows, I will be reformatting the module system 
into a loadable script that can work at lower boot levels. This will 
both be good documentation and allow better testing, for the module 
system and the boot levels too.

Doing a plain import %module.r from the console only works as expected 
if the module is named and not private. Unnamed, private+named, private+unnamed 
modules seem to not work.

Despite what the blog says, unnamed modules are a bit of an advanced 
feature in the R3 module system. Normally modules should be named. 
But yes, the bug was in the criteria that determined that private 
import should be done, and all unnamed modules do the private import 
method (in alpha 110+ at least).

I write a module in a file named %module.r. I don't want to repeat 
that name in a header unless necessary.

In order to be fully imported into the system, a module needs a name 
that the system can use to refer to it by in the modules list. If 
it doesn't have a name then it can't be reused or referred to later, 
so subsequent imports will reload the module source and create a 
new module. And all unnamed modules import privately, meaning that 
they import into the local context only, not into the system runtime 
library. This means no variable sharing.

I don't care how someone wants to write their modules or import statements. 
We have tried to make unnamed modules work as well as we can, given 
their limitations. However, there is a real difference between the 
behavior of named and unnamed modules. For most code it won't matter, 
but if your code depends on that difference then you better make 
sure it loads the way you want it to.

Let's try again.


You wrote "Carl really should stop pushing unnamed modules in those 
blogs of his." I presume this also refers to:
http://www.rebol.net/r3blogs/0344.html

Considering this blog post, would your suggestion amount to:
- A: using import 'simple instead of import %simple.reb
- B: adding name: 'simple to the REBOL header
- C: both, A and B
- D: neither of the above

A, B or C. Or he can continue to gloss over the difference until 
he decides to write a blog about it.

There are real consequences to whether a module is named or not, 
but both styles are appropriate in different circumstances.

Well, then I simply observe that you don't want to give a straight 
answer to above question.

Or let's add, E: "Exactly one of A, B, or C", assume you chose E 
and be done with it.

I don't write blogs. If his purpose is to make the module system 
seem simpler than it is, cool. It can be that simple in practice. 
When called from user scripts there is very little difference between 
regular, unnamed or private modules. The context of user scripts 
is isolated, so all the values are copied into it afterall. It doesn't 
really become a big deal unless you are concerned about when words 
are added to the user context, or until we get concurrency going.

It matters a *lot* for modules that are imported into other modules, 
but no blog has mentioned that situation yet.

I've not read the entire discussion... but let's roll back a little.


Andreas, simple things should be simple. A REBOL rule. So some points 
on modules:


1. We've used objects as "a type of module" for many years. Pretty 
easy.


2. The first thing you do is give them a new datatype, calling it 
module!  But, still basically an object. Easy.


3. Next, you make it clear what is exported... with the EXPORT word 
or EXPORTS block in the spec. Still easy.


4. Next, you want the runtime system to help keep track of the module. 
To do that, the module needs at least a name to identify it. Not 
difficult.


From there, you can imagine many other features you might want: versions, 
checksums, compression, dependencies (needs). You can add quite a 
lot. But, the more you add, the more likely it's going to get complicated... 
and few users will understand it, etc. So, for R3, Brian and I agree 
to a design that provided quite a few features without too much code, 
but also kept simple things simple.

The word "sufficient" there isn't quite true.  Explicit naming is 
more powerful... and provides a map as well from name to filename.

The rest of the above banter was just me getting lost in a particular 
question.

Brian and I have both used quite a few... but not really cared much 
for most of them.

But it's basically impossible to have a sensible discourse about 
this module system without being able to try it out.

I like the idea of not needing to repeat a name = name the module 
automatically upon the filename. "However, there is a real difference 
between the behavior of named and unnamed modules." - why? Because 
someone said there should be a difference? So just not explicitly 
naming the module means it gets treated the different way? Why? And 
what was the technical reason to decide so?

if the module has a name and you rename the file, it should fail, 
which is probably what it does already.

in slim I also added an extension filename, in order to allow import 
by name to work with alternate extensions which is very usefull when 
you want to use the module system as a plugin mechanism for an application.

Technical reason = because one has a name and the other doesn't. 
I'm not dumbing it down, it really is that simple.


Say you are a module, and I want to import you. It's rather straightforward, 
I just add your exported words to my collection (how I do that depends 
on what I am, but that's a story for another time). And then I can 
use those words, no problem.


But what if I don't know whether you have been imported already? 
Or what if I know you have been imported by someone else, but I want 
to use you in particular instead of someone who just looks like you? 
Or what if you have data that you want to share, or resources that 
can't be used more than once at the same time? Or what if you want 
to know if a previous version of you was imported already, so you 
can get that guy's data or resources and take over for him?


To do all of these things, you need a way for others to refer to 
you, a name. If you have a name, I can put you in a collection with 
other modules and then others can look in that collection for a module 
of that name and if they find one they can know that it's you. Simply 
having some way to find you in a crowd makes all of that stuff possible. 
It really is that simple.

Another trick that I can do if I have a name for you is to just put 
you in a box and then import you later: delayed modules. If you don't 
have a name, I can't find you in that box, you look just like all 
the other delayed modules.

It turns out that every little feature or quirk of the module system 
is there for a reason that is as simple as that. It's just that there 
are a lot of these little situations that pop up when you are writing 
a module system if you want to make it work properly. Especially 
if you want to hide a lot of that complexity from the user, to deal 
with the complexity in the module system itself so the programmer 
using it doesn't have to think about any of that. Simple on the outside 
requires some complexity on the inside.

Ok, so the subject of the discussion is how to move the host-kit 
build and its related processes to a more manageable method.

Andreas has some suggestions, and if we can figure out a good way 
to put this together, it would benefit most developers (at this level.)

Where I was going previously: for now, the only part that would need 
your direct involvement is the automation of an "export" from your 
"internal" sources to a set of "external" sources.

So, do we have a good place to put this? Google code?

Unless you want to adopt a new version control system for your sources 
alltogether.

Ok, second step is to use a single repository for all your builds.

BTW, I don't type a dot without version control.

I.e. a 'Linux" repository and a "Win32" repository where you manually 
do merges (or copy/paste) in between will only lead to trouble.

Back in a while. Go ahead and post your notes, then I'll reply. Thanks!

Third step is to write a post-commit hook which exports from internal 
to external.

You'll get the notification for free on Github (I don't know about 
Google Code). But this could also be updating a simple .txt on a 
webserver somewhere which lists all available exports. Heck, it even 
the directory listing features of a webserver would suffice (as long 
as the exports are named consistently).

And from there on, we can have build bots which pick up any new export 
and build it for their platform. Build results are reported back 
somewhere (email, static website on the bots which gets aggregated 
elsewhere, a simple CGI, R3 chat, ...).

The details of those latter parts are not particularly exciting, 
as they are easy to do in a variety of ways.

Which is why it's also important to realise that those build bots 
need not necessarily be public. They could well be Carl's own machines 
standing at home which put the final binary in a central place for 
Carl to use.

A few comments...

Next, is there any conern about flooding. On a good day, I make about 
10-20 commits.

That's why I asked earlier about where we want to put it.... because 
that seems to be a bigger question.

I think using Git is the way to god for a new project.   every big 
project I see that is changing VCS is goint Git.  it seems to be 
the most powerfull VCS right now.  do you agree Andreas?

(Meaning in a day or so.)

What do you think about soliciting a few inputs from other developers 
regarding choice of rev control and related issues... because we'll 
want them to use it?

Fine with me, but not really a necessity.

Well, I want to make sure that Brian, Cyphre, Henrik, Maxim, and 
a few others have a chance to say something... since they're going 
to be some of the primary users.

There are a few side issues around it too. Eg. do we need web access.

I'll post a blog to make sure everyone is tuned in on it.

Note that I already maintain a mirror of the R3 hostkit sources on 
Github:
http://github.com/rebolsource/r3-hostkit

Maybe a good place to start if you want to play around a bit.

Also, can you point us to a concise summary of Git usage?  I've used 
CSV and SVN, but not Git.