Documention for: skimp-my-altme.r Created by: sunanda on: 1-May-2007 Last updated by: sunanda on: 21-May-2007 Format: text/editable Downloaded on: 28-Mar-2024 [div [numbering-on [asis/style/font-size:small;overflow:auto;background-color:#ddffee script: skimp-my-altme.r title: Index and search Altme worlds purpose: Easy and quick way of finding things in Altme worlds author: Sunanda date: 21-may-2007 Version: 0.0.2 asis] [contents [asis doc updated needed callback -- what's new section asis] [div/style/border:thin,blue,solid;margin:1em;padding:1em [h2 purpose [p If you ever use **Altme** to access one or more worlds, this REBOL application could be of use to you. [p It allows you to search the contents of **all** your Altme Worlds. It differs from the search function built into Altme in these important ways: [li **fast** -- once the index is built, searches take only seconds [li **multiple worlds** -- one search can search across **all** of your worlds [li **number of search results** -- not limited to your **settings/message limit** (which often has to be set low to avoid suspected garbage collection bugs in REBOL) [li **html** output is a HTML file, so can be viewed or saved outside of Altme list] div] [div/style/border:thin,green,solid;margin:1em;padding:1em [h2 gui to come [p A GUI is planned and the coding for it has been started. But it is unlikely to be available this side of DevCon 2007. [p This **command line** version is being released now for several reasons: [li it is a useful utility in its own right [li to help debug parts of the API (the GUI will use exactly the same API as **skimp-my-altme.r** uses) [li to ask the REBOL community for help in tuning and refining some of the algorithms used. list] [p The **command line version** requires you to do a **little** more typing than the GUI will. But the results are worth it! div] [div/style/border:thin,red,solid;margin:1em;padding:1em [h2 installation [h3 download files [p **skimp-my-altme.r** is a **package** that consists of four other library scripts and a couple of other files from the package. [p If you use the **package downloader**.... [asis/style/* do http://www.rebol.org/library/public/repack.r asis] [p ....You should end up with these files: [li extend-an-object.r [li skimp.r [li skimp-my-altme.r [li skimp-my-altme-html.r [li sma-css.css [li sma-worlds.r [li rse-ids.r [li tsn.r list] [p You need to edit the **sma-worlds.r** file to add the worlds you want indexed. See immediately below. [h3 edit sma-worlds.r [p This file is a list of the Altme worlds you want indexed. [p The downloaded version has two worlds as an example -- but they are from my test bed, so will not work for you. [p The format of the files is a **block** with one or more **objects**. Each object is the details of one Altme world that you want indexed. There are **four** required fields in each object, and **one** optional one: [li **name** (string!): the name of the world, eg "REBOL3" or "rebol-world". The name is used on all reports. It must be unique for all the worlds in your **sma-worlds.r** file. The name is also used as part of the file name for the index we build: so the name must form part of a legal file name on your platform. [li **user-name** (string!): the name of the user who owns this world. If you have multiple logons for one world (eg **your-name** and **guest** pick one of those names. The name you pick affects what groups and messages **skimp-my-altme** can see. [li **path** (file!): the path to the world (ie to the folder containing the users.set folder for the world). This must be a **folder** (ie end in a **/**). **skimp-my-altme** does not update any files in the Altme world, so you can safely point to your live data [li **index-folder** (file!): the path to a **folder** where **skimp-my-altme** can save the index-files. If this folder does nor exist, it will be created. It is best if it is **not** a subfolder of the **path** -- you never know when an update of Altme will change or delete folders. The entries for several worlds can point to the same folder [li **index-signatures** (block!): this **optional** field allows you to specify some **signatures** (identifying strings). If there are any index-signatures, **skimp-my-altme** will index only the groups whose descriptions contain those strings. **Example:** index-signatures: **["[web-public]" "carl"]** will limit the index to groups whose contains (anywhere in the description) either of those two phrases. list] [h3 multiple indexes on the same world [p You can do this by having multiple entries in **sma-worlds**. For example, for three distinct indexes on the world **my-world** you could have this: [asis make object! [ name: "mw-me" user-name: "me" path: %"/c/program files/altme/worlds/my-world/" index-folder: %/c/skimp-indexes/ ] make object! [ name: "mw-public" user-name: "guest" path: %"/c/program files/altme/worlds/my-world/" index-folder: %/c/skimp-indexes/ ] make object! [ name: "mw-me-music" user-name: "me" path: %"/c/program files/altme/worlds/my-world/" index-signatures: ["jazz" "garage" "baroque"] index-folder: %/c/skimp-indexes/ ] asis] [p The three indexes are: [li **my-mw** -- a complete index of **my-world** for user **me**. This will include private messages to user **me** [li **my-public** -- a complete index of **my-world** for user **guest** [li **my-music** -- an index of **my-world** for user **me** for groups whose descriptions contain any of the three strings in the index-signatures. list] div] [div/style/*-2 [h2 preparing for searching [p As **skimp-my-altme.r** is a command line utility, you need to type (or cut'n'paste) some commands as below. [h3 starting up [asis/style/* do %skimp-my-altme.r asis] [h3 preparing your worlds [asis/style/* world-list: sma-api/get-worlds asis] [p This creates a block, **world-list** (or whatever you want to call it), that contains your **sma-world.r** entries (with some extra fields added). [p As a sanity check, please check that all objects in **world-list** have a field **active: true**: [asis/style/* probe world-list asis] If you see **active: false** we have failed to open the world for some reason (perhaps the path was wrong). There may be a useful status message in the **last-action** block. Correct the problem, and try again. [h3 building the indexes [p Please run these three commands: [asis/style/* sma-api/aaw/get-groups-list world-list sma-api/aaw/get-updates-needed world-list sma-api/aaw/update-index/new-groups/old-groups/dead-groups world-list asis] [P Note that each command takes your **world-list** as an argument.nhey: [li 1: populate a data structure with details of the each world [li 2: analyse the state of the index and Altme world and work out which groups need updating (as this is the first time you have run time, that will be all groups) [li 3: update the index for all Altme groups for all worlds in your world-list list] [p If your world-list includes the REBOL Community's main flagship world, **REBOL3**, expect the update to take between 10 minutes and an hour, depending on the speed of your machine. (There are some 120,000 messages in that world: your numbers may vary depending on the number of private messages you have). [p If you have Graham's alternative world (REBOL-World), it'll take about a tenth as long: it's a smaller world. [p The update process is not (in this version) interruptible, so please let it run to end. [p The **AAW** in the path means **action all worlds**. There is a lower level API that works on individual worlds -- but, for now, that is undocumented. [p Sanity check: given various garbage collection problems REBOL has, it is a good idea at this point to close your console and start another one. [asis/style/* do %skimp-my-altme.r world-list: sma-api/get-worlds asis] div] [div/style/*-2 [h2 run some searches [h3 first search [p You are now ready to search your worlds. Try this: [asis/style/* sma-api/aaw/search-and-display world-list "happy" asis] [p With a bit of luck, you see something like this.... [asis/style/* >> sma-api/aaw/search-and-display world-list "happy" 1-May-2007/08:06:50.203+1:00 searching.... 1-May-2007/08:06:50.203+1:00 converting.... 1-May-2007/08:06:50.656+1:00 generating html.... 1-May-2007/08:06:53.765+1:00 done == true asis] [p ....And a browser window opens with the results. If not, please browse the file **sma-results.html** (in the same folder as **skimp-my-altme.r**). [p There is a **summary table** listing each world and the number of hits (matches). Followed by a table for each world. [h3 search results in context [p It can help to see the messages before and after a match. Use the **/context** refinement for that. This example shows **one** post before and **three** posts after each match. In the HTML file, the matching lines are bolded. The ""context"" lines are not: [asis/style/* sma-api/aaw/search-and-display/context world-list "happy" 1x3 asis] [h3 results window [p If you search for ""happy"" on the REBOL3 world, you may see a summary table like this: [table/att/border="1" [cell World [cell Status [cell Search string [cell Window [cell Total hits [cell Hits in this results page [row [cell REBOL3 [cell Active/searched [cell ["happy"] [cell 1x200 [cell 370 [cell 200 table] [p The key information here is that there were **370** hits on **happy** but only **200** are shown in the search results. [p That's because the default **window** is to show results 1 to 200. [p You can see more results in one of two ways: [p First, have a very wide window -- here we've asked for results 1 through 9,999. [asis/style/* sma-api/aaw/search-and-display/context/window world-list "happy" 1x3 1x9999 asis] [p Second, slide the window -- here we've asked for results 201 through 370. [asis/style/* sma-api/aaw/search-and-display/context/window world-list "happy" 1x3 201x370 asis] [li The first method works fine for up to 500 or so results (your definition of fine will vary, and it depends on the size of your **context** too), thereafter, the HTML gets very large and your browser may be struggling. [li The second method lets you see subsets of the matches on different pages. list] [h3 searching for multiple words [p The default operator is **AND**, so this searches for the two words **happy** and **bug** [asis/style/* sma-api/aaw/search-and-display world-list "happy bug" asis] [h3 not operator [p Use the tilde ( **~** ) to search for posts that do not contain the word. In this example, **carl** but not **sassenrath** [asis/style/* sma-api/aaw/search-and-display world-list "carl ~sassenrath" asis] [h3 other search examples [asis/style/* sma-api/aaw/search-and-display world-list "http://www.rebol.org" ;; url sma-api/aaw/search-and-display world-list "first system/words" ;; path sma-api/aaw/search-and-display world-list "18-aug-2005" ;; date sma-api/aaw/search-and-display world-list "1x1" ;; pair asis] [h3 keeping your index up-to-date [p Simply repeat the three commands in **building the index** above. Any new posts since the last update will be added to the indexes for each world. [h3 excluding worlds from a search [p If you want to search on only some of your worlds, the easier and safest way is to set the world's index as **inactive** in your **world-list**, eg to set the third world in your list as inactive: [asis/style/* sma-api/set-world-status world-list/3 "inactive" asis] [p Searches (and index update operations) will ignore your third world until you reactivate it: [asis/style/* sma-api/set-world-status world-list/3 "active" asis] div] [div/style/*-2 [h2 notes for a gui api [p A GUI front end is being prepared, but we have no date for that as yet. [p If you fancy writing one -- perhaps to showcase your own GUI or GUI skins -- please feel free. [p These notes are intended to help you if you want to do so. They are rather sketchy, so please ask me if you want to know more. [h3 dynamically adding your world-list [p You can add new worlds to your **world-list** block: [asis append world-list extend-an-object make object! [ name: path: user-name: index-folder: ] sma-api/world-template sma-api/set-world-status last world-list "active" asis] [li **append line** -- creates a new world object with the public fields, and extends it to include all the fields **skimp-my-altme** needs to access the world [li **set-world-status** -- setting the status to **active** triggers some validation and setup code for your new entry list] [h3 including / excluding worlds from a search [p You could physically remove them from your world-list before callina any **aaw** API functions. [p Or you can set individual entries as **active** or **inactive**: [asis sma-api/set-world-status world-list/4 "active" sma-api/set-world-status world-list/2 "inactive" asis] [p Only **active** worlds are searched. [h3 saving the updated world-list [p This will update **sma-worlds.r**. The file will include all internal fields for each world. When reloaded later, all internal status settings will be restored: [asis sma-api/save-worlds world-list asis] [h3 Callbacks for status reporting [p By default, **skimp-my-altme** prints some progress information on the console. You can replace that with your own function like this: [asis sma-api/sma-callback: func [ world [object!] info [block!] ][ print "CALL-BACK" probe info return true ] asis] [li **world** -- the world that is being processed [li **info** -- a block with either a **string** or **two integers** -- see below [li **return true** -- asks **skimp-my-altme** to continue processing [li **return false** -- asks **skimp-my-altme** to safely stop processing....Use this rather than **halt** or **throw**, otherwise indexes may be left in an erratic state. You may get some additional callbacks even after setting a **false** return -- **skimp-my-altme** will be informing you of the stages of saving the indexes. list] [p The **info** block contains either: [li **a string** -- you are being informed of a major status point: such as starting to index a new group, or writing back the indexes [li two integers (eg **[120 6747]** meaning you are at a minor status point, and 120 posts out of 6747 have been indexed for the current major status point list] list] div] [div/style/*-2 [h2 work to do [p This is an unfinished version of **skimp-my-altme.r** There is stuff that still needs doing. Including: [li output filters (so results can be trimmed to remove unwanted groups or posters or other such filters) [li some algorithm tweaks to speed things up (we have a couple of hot spots in the performance) [li use the post colors (the present CSS/HTML presents posts in the color of the forum (ie the user's current color). They may be better in the color of the post (using both foreground and background color) [li other HTML and CSS improvements list] [p Feedback is welcome! div] div]