AltME groups: search

I'm not so far... I'm just a C newbie:)

I.e. if you have an RXIARG of of a series type (RXT_BINARY, RXT_STRING, 
etc), this arg also holds the index.

And a pointer to the series data.

You can import 'file for %file.r (or whatever system/options/default-suffix 
is), but when specifying a file you are assumed to know what name 
it is. And you can use %.rx if you want a cross-platform file extension 
for your extension.

On Windows at least this is not a problem - libraries can have any 
extension, even .exe.

How would you know ahead of time which program a user wants to start?

How come? Computer is started. User may start a program on Monday 
that needs to load cURL extension. May decide to start a program 
on Tuesday that needs to load 0MQ extension

Earlier in the session of the call to the interpreter. When the program 
starts it loads the extensions and modules it needs. If you need 
to load things from specific filenames, do that before you load the 
other code. The module system is designed to help you organize and 
manage the code in a program.

I really don't want to load all extensions in the system on all days 
of the week. Just like I don't load the thousand libraries in a Linux 
system into every program (well, almost, but that's the gruesome 
consequence of this going wrong)

Is the module system meant to manage the code in a program, or the 
code in an entire operating system?

Just in a program, not the OS. Each program loads its own modules 
and extensions. Unless R3 *is* the operating system.

Then the program has a problem with portability due to different 
file extensions on different operating systems

You can use different IMPORT blocks depending on the OS, in the same 
code. The highest-level script is usually a not a module, so you 
can call IMPORT directly. Then your modules can just do their requirements 
by (word) name instead of specifying filenames.

For one thing: system/options/default-suffix. For another thing, 
platform-specific stuff doesn't go in the mezzanines, it goes in 
the host code, as a (really strict) rule. For most code that needs 
extensions there is assumed to be a bit of platform-specificity, 
due to the nature of extensions. Nonetheless, this is why LOAD-EXTENSION 
is implemented in the host code, and why we can reference functions 
by word name.

Also, look at system/options/file-types. You can find the file extensions 
that are loaded as R3 extensions, and then go through them, adding 
them to your file's base name to find your file. Or you can add a 
one or two line platform-specific wrapper for your module code.

default-suffix is .reb and that is meant for a mezzanine module, 
not for an extension

You don't want IMPORT to do automatic lookup for more than one file 
extension because it's a potential security hole. For that matter, 
you should know if you're loading an extension instead of a module, 
because extensions could break the rules that REBOL source modules 
can't. And system/options/default-suffix can be set by your program. 
For that matter, the host code for LOAD-EXTENSION could translate 
.rx to .so on platforms that require .so and won't load .rx - that 
is a platform-specific restriction.

That seems to me to be a much bigger security hole

Not .dll - it would only be required on platforms with a restriction 
on filenames that can be loaded as libraries. On windows there are 
dozens of different suffixes that are all basically DLLs, including 
.exe, .ocx, .dpl, .cpl, etc. Not many platforms restrict the file 
suffixes of libraries.

But in general extensions are considered to be unsafe and platform-specific 
unless they are embedded in the host. For that matter, they aren't 
even loadable by default, as a security setting. Cross-platform external 
extensions are expected to be rare, but just in case we have the 
.rx suffix if you want to use them in cross-platform code.

It's my own tool, so it's not a big deal, but people will run into 
tools that they can't easily modify

Do you have a problem with making an embedded delayed extension that 
loads cURL dynamically when the extension is imported? This is supposed 
to be possible, and is the main reason for delayed modules and extensions. 
Then only programs with Needs: [curl] would initialize curl.

R3 external extensions don't pretend to be cross-platform; if they 
are, it's a bonus. Only embedded extensions pretend to be cross-platform.

That's a disaster, then. But it's really very simple to avoid it

I haven't touched Windows in years, and I never wanted to be a Windows 
C programmer, but now I am, just because Oldes and Andreas compiled 
my extension on it

R3 is cross-platform, and cURL is cross-platform, but both require 
a little work to make the platform distinctions go away. POSIX helps 
with this on POSIX platforms, but that's not everything. You can 
do a little extra work in your extension to make it work the same 
externally, so R3 code can't tell the difference, but if you want 
that to extend to filenames then you have to pick a cross-platform 
filename standard. Once the extension is loaded, embedded or delayed 
it can be referred to by module name. We don't want IMPORT to grab 
extensions in its module-paths lookup because that would make it 
possible to load an extension when you were looking for a safe module, 
so .rx and .so files aren't going to be in the search list unless 
your program sets system/options/default-suffix to .rx or .so, with 
the same security implications.

I am fine with explicitly specifying an extension, just not with 
a forced file extension. How about IMPORT/EXTENSION ?

Do you have a problem with making an embedded delayed extension that 
loads cURL dynamically when the extension is imported? This is supposed 
to be possible, and is the main reason for delayed modules and extensions. 
Then only programs with Needs: [curl] would initialize curl.

Yes, I have a problem with that. It's way too much work for making 
a file extension dynamic, and it breaks most of my use cases

There are many platforms that use the same file extensions, so you'll 
have to find a way to put the BSD .so files in another directory 
from the Linux .so files. The .rx extension just adds Windows and 
OSX to that.

That's merely a problem of organising download directories

The common use case is a modular operating system. You have a standard 
R3 installed. During the life of the system, you install new extensions 
and programs. R3 makes the extensions be operating system dynamic 
libraries and lets the OS look for them, so they must be installed 
in the OS. You don't want to also have to embed each new extension 
in a new R3 binary. You want to install and write separate REBOL 
scripts that load these extensions, and you want those scripts to 
be cross-platform, because they easily can be

So, you want to make a modular OS that has all dynamic libraries 
be R3 extensions natively - LOAD-EXTENSION won't load arbitrary libraries, 
just extensions - and you want the library filename suffix to be 
something other than .rx. OK.

LOAD-EXTENSION doesn't do lookup beyond the current directory, so 
if you want to have it use some library path you have to add that 
in the host code, or %rebol.r. Either way you can add extensions 
and modules programmatically to the R3 runtime before the script 
you're running starts. If you want your scripts to be cross-platform 
then you probably shouldn't use a platform-specific file extension 
for the files, but if they are loaded or delayed in the host code 
or %rebol.r then it wouldn't matter.

For that matter, code like that can go in %rebol.r and then scripts 
won't have to even know that an extension isn't a regular module, 
as long as the API matches what they expect.

Oldes, R3 has decided to let the OS find an extension as a native 
library. I really don't want to override that and write my own loader 
in every script

I'm a newbie.. I expect that when I build DLL against windows version 
of the lib, it will not work on Linux.. or maybe I'm wrong.

True. You would use a different file for Linux, though it could also 
be called %zlib.rx :)

You would then import %zlib.rx instead of %zlib.r3. It would work 
on Linux if you made a Linux build of zlib as a R3 extension and 
also called it zlib.rx.

never mind... this is just a minor detail. Form me is important I 
can decompress zlib data as I could with R2. That's all.

I have no time to wai for built in zlibt. But for me is every small 
extension a good C exercise

BrianH: isn't having just one name going to be a problem for us? 
I like the .rx extension very much. But think about distros. You 
can't have one storage of extensions for various platforms. Dunno 
if such use case would exist. Something like big directory of extensions 
somewhere on the fileshare. In the same path. If the suffixes would 
be different, then those files could co-exist. Hmm, maybe not, because 
.so works for Linux, other Unixes, and those might require different 
extension binary anyway? How would you solve such a case? Subdirs 
per system? e.g. /amiga, /win32, /win64, /linux, /syllable?

Alien binaries shouldn't be installed, so you need separate packages, 
or a package with an extension installer. If you really don't want 
to do that, you need subdirectories

A system-wide loader like that is a feature of an OS that starts 
once, not of a language platform that launches many times

Subdirs per platform is one way to do this (if you are calling your 
app from a network share, or building it from source for instance).

Have them download a zip file containing the Windows binaries. That's 
how SQLite does it.

Kaj: No need. You can import the extensions by filename. We want 
the initial import of an extension to be really clear, distinct from 
the import of a module. If you say that LOAD-EXTENSION looks up the 
file in the system library path, that shoul;d be all the searching 
we would need.

I am not clever enough to suggest a solution, but I think that some 
solution could be at least thought about ....

Maybe we could CC it as a wish - how to free rebol resources for 
more complex stuff as a gui?

Kaj, you do realize that you won't be able to just load any system 
library, right? Unless that library has an R3 extension API, it can't 
be loaded as an extension. And if you can only load R3 extensions 
anyways, why not name them as R3 extensions? The .so filename suffix 
isn't even portable to all POSIX platforms. If you want your code 
to be portable and load filenames, use a portable filename. That 
is why we have the .rx convention. And we have %rebol.r for anything 
specific to your personal platform - you don't even have to touch 
host code. For that matter, CGI scripts could call a R3 interpreter 
that is in a different directory than the user R3 interpreter so 
they would use different %rebol.r settings.

Still, the only way to get a proposal submitted around here is to 
go to CureCode

Oldes, Pekr, I don't know whether extensions can be unloaded. Last 
time I checked it was considered unsafe to do so, because the import 
process doesn't isolate things. In theory you could clean up after 
a module, remove it from the system modules list, remove all references 
to it from all other modules (that you have tracked), and then let 
the module be collected by the garbage collector, but I don't know 
if extensions are unloaded when they are collected.

OK, after reading all the above, I agree on .rx being a suffix for 
an extension. The reasons? Even dll or so are not necessarily crossplatform, 
plus having an extension even as .dll is confusing - normally it 
might not be clean .dll, which you can load in some other environment 
- there might be packed rebol code inside. But I don't want to complicate 
Kaj's aproach, and more important - I never coded any extension, 
so regard it being just an opinion ...

OTOH it would be imediatelly obvious, that the code is some module, 
not a plain script, which necessarily does something ...

You can distinguish R3 modules from scripts with LOAD/header or some 
such. But you can call R3 scripts anything you like. Scripts are 
just a kind of module in R3 anyways (one of 3 or 5, depending on 
how you count it).

I thought about user, downloading a distro. If those are not distinguished, 
I would try to DO each script, just to demo the REBOL. And doing 
a module usually does nothing more than loading the module = nothing 
visually usefull. So I would not mind having .r3, .rm, .rx

If you want to call your modules .rm or .r3m or whatever, go for 
it. We don't have a standard suffix for scripts, but feel free to 
make one. Actually, the default suffix for R3 module files is .reb 
:)

The .rex extension is taken on Windows by a third-party app, but 
that may not matter. The main reason the default extension is .reb 
is because Carl likes it, but the programmer is expected to change 
it, and even Carl doesn't use it. I like .r3 for R3 scripts and modules 
both, because there's really not enough of a difference between them 
to justify a different name, but YMMV. Maybe we can get Github to 
treat .r3, .r2 and .reb as REBOL scripts.

well, there was even one other suffix, for a compressed scripts, 
was it .rip? I like .r3, .rm, .rx, and those I will use (maybe really 
not necessary to do a separate suffix, I will see). .reb just says 
- nothing ...

I think that is the real reason the default is set to .reb: Noone 
uses it. People either use .r or .r3, and you are expected to change 
system/options/default-suffix to what you need it to be. That is 
the reason it's a system option.

No separate extension for compressed scripts is needed, as all scripts 
can be compressed. For that matter, script-encoded compressed scripts 
can even be colorized, though the data is just a big binary. Again, 
YMMV, you might have another opinion about this :)

if people are distributing their files in a single dir, they will 
have .so, .dll  in the same dir.


can't we just build a little .r file which imports the proper lib, 
no?

At which point it makes a _lot_ of sense to not artificially hide 
an R3 extension's nature as dynamic library and just use the platform-specific 
conventions.

Now for most extensions they: a) won't be delayed extensions built 
into the host, and b) will easily be cross-platform across at least 
some platforms.

Having a simple cross-platform way to load those extensions while 
not imposing artificial naming constraints would be _extremely_ useful.

I've used a few applications which have their own extension names 
simply because there is a lot of overlap in names and clashing with 
the OS is very easy at some point.

I'd prefer using system names, only not the system load path management. 
  Rebol, should only look for R3 extensions withing its path configs... 
whatever they are... the R3 extensions *will* use the OS loading 
path, which is normal, since the extension is in fact a bridge between 
the OS and rebol.

And name clashes is what a proper loader is there to take care of.

so if a rebol zlib interface is installed, I want the r3 module to 
be loadable using   IMPORT zlib     ... not r3zlib.

Doesn't necessarily have to be IMPORT. I agree with Brian that we 
probably want to have extension loads "stand out" a bit.

it goes beyond name clashes... R3 extensions have a specific duty, 
we are not loading OS libs directly within R3... import must not 
try to access these.


I would prefer to have a specific loader for extensions something 
like.

EXTEND my-extension

That's just a matter of definition.

Maxim, R3 does a simple `dlopen`.

well how do you install zlib.so on a system that already has this 
?

knowing that the zlib.so actually is only a stub for the real one.

yes, I know... what I want is for R3 to control the lib search path 
not the user ... I'm sure all OSes allow this.  R3 would only load 
extensions in a single place it expects.

Btw, there are two places where a "user" could control the search 
path. From within R3 or from the outside.

windows actually has a registry of name/version numbers when dlls 
are installed properly, which helps a lot.

I want to be able to setup a R3 config file that says:

extensions-dir:  %/some/path/


and then R3 will only ever load extensions from there.    the extensions 
can do whatever they want, but we have a controled point of entry 
within REBOL.

afaik, systems allow paths on dlopen.    rebol would simply always 
ask for libs with paths... 


in fact, on windows, that is how you are supposed to do it, probing 
the registry first to get proper paths for libs.

no... only.  I really don't want rebol to start playing around in 
the OS, looking for extensions, if I set up a unique path.  really.

REBOL could very well setup things like local paths if the extension-dir 
is setup as a block with fallbacks...  which make the loading of 
application libs portable.

This allows distributions with a strong library management to properly 
and fully integrate R3 into the OS.

a lot of things in R2 are frustrating, one of them is the ability 
to "clamp down" an environment in which you *know absolutely* where 
files are being searched and stored relative to the running app. 
 distributing apps in a multi-user setup where apps are installed 
on network disks, is impossible to properly clamp down unless you 
use absolute paths in the application itself.


I also want to be able to tell it not to fallback.  though, by default, 
defaulting is usefull, I agree.


for example, I want to be able to make sure that whatever happens, 
a local machines libs will never be accessed, cause that can lead 
to people overiding an install which has serious security considerations.

though the install path may change from one user to the next, so 
absolute paths in the application might not make sense.


I just providing these examples, cause they have actually hapened 
to me in a 100+ employe setup where we had applications being setup 
on the fly on login.

with rebol, one could always break the install easily, since you 
can't easily force it into a corner.

If you don't want your loader to look into the system paths for a 
specific app, tell it not to do so.

But leaving that aside, this specific item is a very simple flag 
that can be exposed from within R3 as well.

myself I have about 10 different rebol installs, they are all setup 
differently.   some I'd like to lock down, but I can't unless I build 
a special version of R3.


If we want REBOL to allow strong security on all platforms, it has 
to be *able* to manage the search paths on its own.   cause all platforms 
will have different working models.


another example is for a web plugin that allows extensions.  we don't 
want it to look for its extensions anywhere else than in a path that 
its managing.


I don't want to have to invent this system everytime I build a new 
system built with R3

I'd say that the chances that a R3 extension you forgot to bundle 
with your app happens to linger around in the Win32 system paths 
are pretty slim :)

Roughly summarising: an extension search path (preferred) + os loader 
search (fallback) + a cross-platform mechanism to load extensions 
via abstract names.


Could be as simple as reusing system/options/module-paths and load-extension, 
allowing the latter to take a word! parameter and also search the 
former. The possible extensions themselves are already in place (system/options/file-types). 
Loading is already done via dlopen, so loading a lib only by name 
(w/o any path component) will use the OS loader's search mechanisms.


And alternative would be to add a system/options/extension-paths 
(block!) option, and a dedicated loading primitive, say, import-extension, 
which searches the extension-paths and tries all possible platform-specific 
extensions (as per system/options/file-types).

While there wouldn't be a point to the OS managing REBOL modules, 
because it doesn't know what they are

Exactly, and in a portable way

If you want to fully integrate R3 into an OS, that is a job for the 
host code. For most OSes you want as much isolation between the OS 
and R3 as you can get. Unless you have a locked-down OS (most don't) 
then what constitutes a "safe" extension would vary widely between 
programs. That is why the module-paths option and others like it 
are just set by default by the startup code, to be changeable by 
the program code as needed.

We do have a way to have extension search managed in a portable way 
by OS-specific means: LOAD-EXTENSION is implemented in the host code.

Yup. But keep in mind that for most potential R3 users (who like 
most users are running Windows, OSX or some stock Linux) the OS mechanisms 
for managing libraries shared between programs are really poor, and 
they are encouraging program-specific storage. I'm really interested 
in how you will experiment with integrating R3 with a properly managed 
OS, which (I hope) Syllable is.

Even though Syllable has its own dynamic loader in the kernel, we're 
not managing user space libraries vastly differently than Linux, 
which has a user space loader. The difference is in kernel drivers, 
which are standard ELF libraries just like user space libraries, 
which isn't possible in Linux. I consider this similar to the nice 
circumstance that R3 extensions are native OS libraries, which has 
similar integration benefits - if you take advantage of them

The idea of a "safe" extension is tenuous at best as it is. Don't 
add the possibility of having some system-specific libraries polluting 
the file name space. For instance, I would consider one of the minimum 
requirements of a system-installed "safe" extension to be that it 
be guaranteed delayable.

If you want safe binary extensions, rewrite REBOL on Genode with 
a GPL license. Anything less won't do

One interesting thing you haven't considered: sys/load-module calls 
lib/load-extension without checking whether it is native. You can 
override lib/load-extension with a REBOL code wrapper, in %rebol.r 
even. Then you can do any safe lookups you like without messing with 
host code, and even translate .rx suffixes to .so if you like.

This is a great way to experiment with platform integration, and 
what you learn can be adapted to the host code.