UHURU/RSL/? summary

Hi, My apologies for the length, but IMO it's important this is done.. Before we can go into much more detail with this system, I think we need to clear up all the discussion so far and summarise the key ideas otherwise me may start to lose new contributors to the thread or confuse everyone. I'll attempt to distil the key points form the discussion so far, please make corrections and additions or raise objections where you see fit. I will use "the system" here rather than one of the acronyms (mainly because I type it faster than UHURU ;)) I will try to describe things in a more "task based" way than the concepts discussed in Joel's introduction (reference 1 below) and I've fleshed some parts out with some additional descriptions of my own, feel free to rip them to pieces :) Intro: Intent of the system The aim of this system is to provide: - A centralised, keyword searchable repository of packages (from individual functions through to applications). See part 1. - A REBOL script or package which can obtain specific packages from the repository, based on package name (or keywords ?). This must handle dependancies automagically. See part 2 for more on this. - A standard for package implementation, including a specification of required metadata and coding and documentation standards. This is discussed in part 3. Part 1. The repository. Some would call this a Library (which is the less "hackeresque" term). The core of this system is a hierarcical structure of packages which can either be browsed or searched via a keyword search engine on the web. In this respect the repositry would be similar to the script libraries on rebol.com or rebol.org, with a search engine added for assisted lookups. A possible refinement would be that when selecting package, rather than being presented with the code itself a summary is shown. For example, selecting delim-ascii-html would bring up: Author: Joel Neely Latest version: 2.1 Last update: 13:32, 25 May 2001 Synopsis: Delimited ASCII to HTML conversions Keywords: ASCII HTML comma tab convert text conversion Dependancies: delimited-ascii Version Date Source Documentation 1.0 1 May 2001 15k (mirrors) 40k (mirrors) ... 2.1 25 May 2001 28k (mirrors) 64k (mirrors) In this scenario, clicking on the author name would bring up all the scripts the author had written (profile as well?), clicking on one of the keywords would search for that keyword, clicking on the dependancies would show the summary for that file. The table at the end lists the available versions. Clicking on the size would download the file, clicking on the mirrors word would bring up a list of mirrors of that file. A key requirement of the repository is that it must be easily mirrored to ensure fast access from any location in the world. The problem with this is ensuring coherancy across the mirrors. Mirroring would be best achieved via a script which obtained all new or updated packages from a central repository on an automatic basis (cron triggered for example). This means that a coordinating repository with a recent additions facility is required. Something to consider: REBOL has built in compression, if the repository is purely cgi powered then it would be possible to store the files on the server in compressed form. When a file is requested, the script could decompress it and sent it to the the user's browser. This does not gain anything for the web version - bit for the remote access program this would be a bit help. Note that I do not mean "store all the files for a package in one archive". Dynamically processing this would be clumbersome, and it would cause even more problems when obtaining packages with the remote access program. I mean that each file should be stored seperately on the server but in a compressed form. Part 2. The remote access program. Using this package developers can obtain packages from the repository without a web browser, or scripts can automagically obtain sources from the repository when they are setting up or checking for updates. What it needs to be able to do: - Given a package name it must check a designated repository (see below) for that package. If found, and the version on the local machine is not up to date, the package is downloaded to the local machine. - (possible, this could be tricky) Given keywords this will search the repository for suitable files. This is more likely to be a front-end to the search engine on the repository which processes the search result webpage to produce a list of matching packages to display in the console - All dependancies must be resolved during the download operation. If delim-ascii-html depends on delimited-ascii then the script must detect this, check whether a suitable version is available locally and, if not, automatically obtain the delimited-ascii package. - By default the script only downloads the source file(s). By specifying an additional argument, the script will download both the sources and associated documentation. As a tie-in to the compression discussion in step 1, if files on the repository are compressed then the remote access program would be able to work considerably more efficiently than a web browser for obtaining sources: the archive could be downloaded and unpacked locally (unlike the web situation where it would have to be unpacked on the server). (note that this reduces somewhat the need for "low comment" code.) Specifying a repository could be tricky - the most efficient way to do it is to choose the fastest, closest mirror. How this could be done automatically I have no idea (I don't even know if it could be done automatically). I would suggest that the central repositroy holds a list of mirrors giving their geographical location, when the script is run for the first time the user is asked to choose a local mirror.. something we need to clear up. Part 3. This mail is already too big, so I'll do this in a second one. Coding standards are always a nasty area as they need to be flexible enough to allow programmers to keep their personal style while ensuring code can be used and understood by all. There are some big questions still left in here, particularly the issues of docuentation content, extensions to the REBOL style guide and the specification of metadata required to make the system possible. A. References These are not in chronological order, I have rearranged them on a subject basis. I have also removed mails not directly related to the specification of the system. 1. http://www.escribe.com/internet/rebol/m9432.html - Joel Neely Introduction to the concepts behind the system. 2. http://www.escribe.com/internet/rebol/m9437.html - Chris Page Discussion of REBOL header v XML meta information (and dissing Joel's acronym ;)) Ladislav Mecir backs up Chris' view in m9468.html 3. http://www.escribe.com/internet/rebol/m9447.html - Joel Neely Explanation of rationale behind XML suggestion, start of comments discussion. 4. http://www.escribe.com/internet/rebol/m9458.html - Chris Page Comments on XML v REBOL, followup to Joel's comment statement. 5. http://www.escribe.com/internet/rebol/m9461.html - GS Jones Comments on obtaining commented or uncommented versions of packages. 6. http://www.escribe.com/internet/rebol/m9463.html - Chris Page Expansion on m9461 with discussion of possible mechanisms for comment/documentation selection. 7. http://www.escribe.com/internet/rebol/m9467.html - Joel Neely Further discussion of the commenting/documentation distinction. 8. http://www.escribe.com/internet/rebol/m9422.html - GS Jones Discussion of versioning, packages, namespace and operation. 9. http://www.escribe.com/internet/rebol/m9449.html - Chris Page Elaboration of m9422, discussion of mertis of REBOL over cvs 10 http://www.escribe.com/internet/rebol/m9479.html - Volker Nitsch Discussing methods. Partially the reason I started this as there seems ot have been confusion about the final system which Volker was trying to address here.

Hi, Chris, I just wanted to express a high-decibel ***** * * *** * * * * * * *** * * * * * * * * ** * * * * * * * * * * * ***** ***** * * * * * *** * * * * * * * * * * * ** ** * * * * * * * * * * * * * * * * *** **** * For your work in summarizing and organizing the material on this topic! -jn- Chris wrote:

> Hi, > My apologies for the length, but IMO it's important > far and summarise the key ideas otherwise we may start to > lose new contributors to the thread or confuse...

#25-May-01# Message from *Joel Neely*: Hi Joel,

> Hi, Chris, > I just wanted to express a high-decibel <snip> > For your work in summarizing and organizing the material on > this topic!

#25-May-01# Message from *Chris*: Hi Chris, Please refer to http://www.escribe.com/internet/rebol/m9502.html for a full copy of the summary and references.

> - A standard for package implementation, including a specification > of required metadata and coding and documentation standards. > This is discussed in part 3.

Hi Chris, I think I start believing in the RSL again ;-) Some suggestions from my side: 1) Make the repository database'd. We can use MySQL for that, as bindings for both Core and /Pro /Command exist. Should simplify the keyword search part. 2) Style guides shouldn't be that hard. Formatting is most of the time what make a script readable: that can be automated. Undocumented functions/objects can be detected and a reason for replying to the author the original script with automatically inserted hints, such as <Insert documentation here> My point with 2) is that if you automate, people can code in whatever style they wish as long as they use Rebols built-in doc facilities. This is a realistic approach (IMHO) to get a library/repository up and running. 3) Namespace placement (on submission) can be automated too, I think. One other option is some distributed script network. Thsi can be easily built with the next release of Rugby, as that will feature oneway, deffered and redirect messages. The latter allow you to do something like rexec/oneway/redirect [message] [destination-list] which simplifies P2P messages flying around. --Maarten

From: "Chris"

<snip> > 1_4_2_93_2Pre3B1 or something equally ridiculous, ...

> A side effect of the latter is that dependancy checks can obtain the

> version without the need for two dependancy blocks: > > depends-on [/text/conversion/delimited-ascii-2

> as opposed to > > depends-on [/text/conversion/delimited-ascii /foo/bar/thingy] > depends-versions [2 1]

#25-May-01# Message from *Maarten Koopmans*: Hi Maarten,

> 1) Make the repository database'd. We can use MySQL for that, as bindings > for both Core and /Pro /Command exist. Should simplify the keyword search > part.