script: skimp-my-altme.r
      title: Index and search Altme worlds
    purpose: Easy and quick way of finding things in Altme worlds
     author: Sunanda
       date: 21-may-2007
    Version: 0.0.2
 

doc updated needed
 callback
 -- what's new section
 

1. purpose

If you ever use Altme to access one or more worlds, this REBOL application could be of use to you.

It allows you to search the contents of all your Altme Worlds. It differs from the search function built into Altme in these important ways:

• fast -- once the index is built, searches take only seconds
• multiple worlds -- one search can search across all of your worlds
• number of search results -- not limited to your settings/message limit (which often has to be set low to avoid suspected garbage collection bugs in REBOL)
• html output is a HTML file, so can be viewed or saved outside of Altme

2. gui to come

A GUI is planned and the coding for it has been started. But it is unlikely to be available this side of DevCon 2007.

This command line version is being released now for several reasons:

• it is a useful utility in its own right
• to help debug parts of the API (the GUI will use exactly the same API as skimp-my-altme.r uses)
• to ask the REBOL community for help in tuning and refining some of the algorithms used.

The command line version requires you to do a little more typing than the GUI will. But the results are worth it!

3. installation

3.1. download files

skimp-my-altme.r is a package that consists of four other library scripts and a couple of other files from the package.

If you use the package downloader....

   do http://www.rebol.org/library/public/repack.r
 

....You should end up with these files:

• extend-an-object.r
• skimp.r
• skimp-my-altme.r
• skimp-my-altme-html.r
• sma-css.css
• sma-worlds.r
• rse-ids.r
• tsn.r

You need to edit the sma-worlds.r file to add the worlds you want indexed. See immediately below.

3.2. edit sma-worlds.r

This file is a list of the Altme worlds you want indexed.

The downloaded version has two worlds as an example -- but they are from my test bed, so will not work for you.

The format of the files is a block with one or more objects. Each object is the details of one Altme world that you want indexed. There are four required fields in each object, and one optional one:

• name (string!): the name of the world, eg REBOL3 or rebol-world . The name is used on all reports. It must be unique for all the worlds in your sma-worlds.r file. The name is also used as part of the file name for the index we build: so the name must form part of a legal file name on your platform.
• user-name (string!): the name of the user who owns this world. If you have multiple logons for one world (eg your-name and guest pick one of those names. The name you pick affects what groups and messages skimp-my-altme can see.
• path (file!): the path to the world (ie to the folder containing the users.set folder for the world). This must be a folder (ie end in a /). skimp-my-altme does not update any files in the Altme world, so you can safely point to your live data
• index-folder (file!): the path to a folder where skimp-my-altme can save the index-files. If this folder does nor exist, it will be created. It is best if it is not a subfolder of the path -- you never know when an update of Altme will change or delete folders. The entries for several worlds can point to the same folder
• index-signatures (block!): this optional field allows you to specify some signatures (identifying strings). If there are any index-signatures, skimp-my-altme will index only the groups whose descriptions contain those strings. Example: index-signatures: ["[web-public]" carl ] will limit the index to groups whose contains (anywhere in the description) either of those two phrases.

3.3. multiple indexes on the same world

You can do this by having multiple entries in sma-worlds. For example, for three distinct indexes on the world my-world you could have this:

 make object! [
           name: "mw-me"
      user-name: "me"
           path: %"/c/program files/altme/worlds/my-world/"
   index-folder: %/c/skimp-indexes/
 ]
 make object! [
           name: "mw-public"
      user-name: "guest"
           path: %"/c/program files/altme/worlds/my-world/"
   index-folder: %/c/skimp-indexes/

 ]
 make object! [
              name: "mw-me-music"
         user-name: "me"
              path: %"/c/program files/altme/worlds/my-world/"
  index-signatures: ["jazz" "garage" "baroque"]
      index-folder: %/c/skimp-indexes/
 ]
 

The three indexes are:

• my-mw -- a complete index of my-world for user me. This will include private messages to user me
• my-public -- a complete index of my-world for user guest
• my-music -- an index of my-world for user me for groups whose descriptions contain any of the three strings in the index-signatures.

4. preparing for searching

As skimp-my-altme.r is a command line utility, you need to type (or cut'n'paste) some commands as below.

4.1. starting up

    do %skimp-my-altme.r
 

4.2. preparing your worlds

    world-list: sma-api/get-worlds
 

This creates a block, world-list (or whatever you want to call it), that contains your sma-world.r entries (with some extra fields added).

As a sanity check, please check that all objects in world-list have a field active: true:

    probe world-list
 

If you see active: false we have failed to open the world for some reason (perhaps the path was wrong). There may be a useful status message in the last-action block. Correct the problem, and try again.

4.3. building the indexes

Please run these three commands:

    sma-api/aaw/get-groups-list world-list
    sma-api/aaw/get-updates-needed world-list
    sma-api/aaw/update-index/new-groups/old-groups/dead-groups world-list
 

Note that each command takes your world-list as an argument.nhey:

• 1: populate a data structure with details of the each world
• 2: analyse the state of the index and Altme world and work out which groups need updating (as this is the first time you have run time, that will be all groups)
• 3: update the index for all Altme groups for all worlds in your world-list

If your world-list includes the REBOL Community's main flagship world, REBOL3, expect the update to take between 10 minutes and an hour, depending on the speed of your machine. (There are some 120,000 messages in that world: your numbers may vary depending on the number of private messages you have).

If you have Graham's alternative world (REBOL-World), it'll take about a tenth as long: it's a smaller world.

The update process is not (in this version) interruptible, so please let it run to end.

The AAW in the path means action all worlds. There is a lower level API that works on individual worlds -- but, for now, that is undocumented.

Sanity check: given various garbage collection problems REBOL has, it is a good idea at this point to close your console and start another one.

    do %skimp-my-altme.r
    world-list: sma-api/get-worlds
 

5. run some searches

5.1. first search

You are now ready to search your worlds. Try this:

    sma-api/aaw/search-and-display world-list "happy"
 

With a bit of luck, you see something like this....

    >> sma-api/aaw/search-and-display world-list "happy"
    1-May-2007/08:06:50.203+1:00 searching....
    1-May-2007/08:06:50.203+1:00 converting....
    1-May-2007/08:06:50.656+1:00 generating html....
    1-May-2007/08:06:53.765+1:00 done
    == true
 

....And a browser window opens with the results. If not, please browse the file sma-results.html (in the same folder as skimp-my-altme.r).

There is a summary table listing each world and the number of hits (matches). Followed by a table for each world.

5.2. search results in context

It can help to see the messages before and after a match. Use the /context refinement for that. This example shows one post before and three posts after each match. In the HTML file, the matching lines are bolded. The "context" lines are not:

     sma-api/aaw/search-and-display/context world-list "happy" 1x3
 

5.3. results window

If you search for "happy" on the REBOL3 world, you may see a summary table like this:

World	Status	Search string	Window	Total hits	Hits in this results page
REBOL3	Active/searched	["happy"]	1x200	370	200

The key information here is that there were 370 hits on happy but only 200 are shown in the search results.

That's because the default window is to show results 1 to 200.

You can see more results in one of two ways:

First, have a very wide window -- here we've asked for results 1 through 9,999.

     sma-api/aaw/search-and-display/context/window world-list "happy" 1x3 1x9999
 

Second, slide the window -- here we've asked for results 201 through 370.

     sma-api/aaw/search-and-display/context/window world-list "happy" 1x3 201x370
 

• The first method works fine for up to 500 or so results (your definition of fine will vary, and it depends on the size of your context too), thereafter, the HTML gets very large and your browser may be struggling.
• The second method lets you see subsets of the matches on different pages.

5.4. searching for multiple words

The default operator is AND, so this searches for the two words happy and bug

     sma-api/aaw/search-and-display world-list "happy bug"
 

5.5. not operator

Use the tilde ( ~ ) to search for posts that do not contain the word. In this example, carl but not sassenrath

     sma-api/aaw/search-and-display world-list "carl ~sassenrath"
 

5.6. other search examples

     sma-api/aaw/search-and-display world-list "http://www.rebol.org"  ;; url
     sma-api/aaw/search-and-display world-list "first system/words"    ;; path
     sma-api/aaw/search-and-display world-list "18-aug-2005"           ;; date
     sma-api/aaw/search-and-display world-list "1x1"                   ;; pair
 

5.7. keeping your index up-to-date

Simply repeat the three commands in building the index above. Any new posts since the last update will be added to the indexes for each world.

5.8. excluding worlds from a search

If you want to search on only some of your worlds, the easier and safest way is to set the world's index as inactive in your world-list, eg to set the third world in your list as inactive:

   sma-api/set-world-status world-list/3 "inactive"
 

Searches (and index update operations) will ignore your third world until you reactivate it:

   sma-api/set-world-status world-list/3 "active"
 

6. notes for a gui api

A GUI front end is being prepared, but we have no date for that as yet.

If you fancy writing one -- perhaps to showcase your own GUI or GUI skins -- please feel free.

These notes are intended to help you if you want to do so. They are rather sketchy, so please ask me if you want to know more.

6.1. dynamically adding your world-list

You can add new worlds to your world-list block:

    append world-list extend-an-object
                 make object! [
                       name:
                       path:
                       user-name:
                       index-folder:
                  ]
            sma-api/world-template

    sma-api/set-world-status last world-list "active"
 

• append line -- creates a new world object with the public fields, and extends it to include all the fields skimp-my-altme needs to access the world
• set-world-status -- setting the status to active triggers some validation and setup code for your new entry

6.2. including / excluding worlds from a search

You could physically remove them from your world-list before callina any aaw API functions.

Or you can set individual entries as active or inactive:

    sma-api/set-world-status world-list/4 "active"
    sma-api/set-world-status world-list/2 "inactive"
 

Only active worlds are searched.

6.3. saving the updated world-list

This will update sma-worlds.r. The file will include all internal fields for each world. When reloaded later, all internal status settings will be restored:

   sma-api/save-worlds world-list
 

6.4. Callbacks for status reporting

By default, skimp-my-altme prints some progress information on the console. You can replace that with your own function like this:

    sma-api/sma-callback: func [
     world [object!]
     info [block!]
 ][
     print "CALL-BACK"
     probe info
     return true
 ]
 

• world -- the world that is being processed
• info -- a block with either a string or two integers -- see below
• return true -- asks skimp-my-altme to continue processing
• return false -- asks skimp-my-altme to safely stop processing....Use this rather than halt or throw, otherwise indexes may be left in an erratic state. You may get some additional callbacks even after setting a false return -- skimp-my-altme will be informing you of the stages of saving the indexes.

The info block contains either:

• a string -- you are being informed of a major status point: such as starting to index a new group, or writing back the indexes
• two integers (eg [120 6747] meaning you are at a minor status point, and 120 posts out of 6747 have been indexed for the current major status point

7. work to do

This is an unfinished version of skimp-my-altme.r There is stuff that still needs doing. Including:

• output filters (so results can be trimmed to remove unwanted groups or posters or other such filters)
• some algorithm tweaks to speed things up (we have a couple of hot spots in the performance)
• use the post colors (the present CSS/HTML presents posts in the color of the forum (ie the user's current color). They may be better in the color of the post (using both foreground and background color)
• other HTML and CSS improvements

Feedback is welcome!