AltME groups: search

Too bad I thought it was a hardware issue and I sent a reboot from 
the control panel, just a few seconds before realizing that the server 
plan has expired. The server was running since 670 days uninterrupted.

You could also make your own syslog server with 0MQ and send log 
messages to it from RSP scripts. That will offload the writing to 
a different process and 0MQ will take care of serialisation

Endo, thanks for the code. I will need something similar for sqlite. 
I just got a first db is locked error yesterday with it at UsrJoy.


What I'm trying to log is side-info (like usage info) so I don't 
want to inpact the speed of response by having aditional disk write 
in the request-response process (it has to be async).


Doc: I used debug functions for various info logging too now (and 
I do diff on trace in cron and send myself email if there is any 
difference), but I am trying to log more things and I would like 
to leave trace.log for errors only. I was interested if the existing 
functionality that serializes debug to trace.log could be used to 
log to some other file. like info.log . That would complicate the 
app-code the least.. otherwise I was thinking of what  Kaj  proposed, 
to have some queue to send data over via tcp to and it will write 
it to disk in intervals. That would bring another dependancy into 
app-code.. something like redis could automatically work like this 
I think.

The probably best option would be for Cheyenne to spawn a new process 
that would handle all the log files serialization (both for Cheyenne 
internal use and for web apps). The code for that is already bundled 
in Cheyenne main process, so it would not be a big work to extract 
it and spawn a new process. (but would require at least a couple 
of days, including testing).

0MQ is already heavily async, and you can make the request/response 
pattern not wait

Btw, I am currently working on making Cheyenne websocket support 
conform to the latest RFC specification. The current Cheyenne support 
is obsolete and won't work anymore with latest browsers.

The newer websocket RFC is much better written and more exhaustive 
than the previous versions. The protocol has also nicely improved 
fixing the remaining security issues.

It makes heavy requirements on the file locking of the operating 
system for that, and it does have a document section that explains 
how operating systems are buggy and badly documented, so that doesn't 
exactly instill confidence

Reliable and efficient file locking is hard to achieve, I agree with 
that. That's why I went for a syslog-like solution for Cheyenne.

When any process wants to write, it must lock the entire database 
file for the duration of its update


However, client/server database engines (such as PostgreSQL, MySQL, 
or Oracle) usually support a higher level of concurrency and allow 
multiple processes to be writing to the same database at the same 
time. This is possible in a client/server database because there 
is always a single well-controlled server process available to coordinate 
access. If your application has a need for a lot of concurrency, 
then you should consider using a client/server database.

Bad news for websocket support in REBOL: the new RFC requires that 
client encodes data sent to server using a basic XOR encryption algorithm:


http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-10#section-4.3


This is a bad news for us, because it requires to process all bytes 
received, one by one to decode the message. REBOL is very slow at 
processing big data in loops, so the overhead can be very significant 
for data frames of a few dozen KB and more. It could affect Cheyenne 
global performances drastically.


However, it could have been worse, this encryption scheme is not 
required for data sent by server. So, as long as clients are sending 
small messages (up to a few KB), the overhead should be low. Fortunately, 
the usual client messages are queries to obtain data, so usually 
small. But if you have to move big amouts of data (like XML documents) 
back and forth through websockets, Cheyenne won't be able to cop 
with the load and it will most probably be a show-stopper.

Make the C lib open source and let people, who want that functionality, 
maintain the lib. It shouldn't necessarily be your job.

In the end, the burden will fall on my shoulders if I want fixes 
and updates to be done in time (as usual). If someone makes such 
lib (just 3 lines of C, btw), maintains it (means provide binaries 
for target platforms), I can add an optional loader in Cheyenne to 
use it when present. As for myself, I prefer to switch to Red asap.

I have pushed a preliminary implementation of the latest websocket 
RFC to Cheyenne SVN repo. It works only for text messages of size 
< 126 bytes. I will get back to it in the next day and complete it.

I think that the cleanest way would be then to extend the existing 
INCLUDE function in RSP.r by adding a refinement (/with or /options) 
and use it to pass additional custom data to the INCLUDE function 
(like a 'decrypt word, followed by a key).

You could add a property to ENGINE object for controlling how files 
are loaded, and set/unset it depending on INCLUDE refinement.

Loading and compilation of RSP scripts are done in ENGINE object, 
not by INCLUDE.

about encap: I'll try to find the problem and if it's not difficult 
I try to fix it as well. I wanted to ask before spending time on 
it. Thank you. Doc: if you already done / test it please let me know. 
I'm planning to use it in a production.

Some were very interested by a simple, small and easily deployed 
web server that could run PHP code efficiently.

The first parameter (a URL) is mandatory and should point to the 
resource where login happens (it's a pass-thru, every other URL will 
be redirected or will return a custom HTTP code, if specified).

websocket test server application

: you already have it, just run cheyenne.exe and start writing your 
websocket app.

I don't understand what you mean by 

external" there?" when I copy embed-demo.exe alone to outside cheyenne 
directory, it gives an error like "cannot find the file mods/mod-embed.r". 
So even encapped it requires that file and probably the other modules 
(.r) files.


What I was thinking, when I use embed mode and encap the cheyenne, 
it already includes necessary mod files into encapped exe.

Here is the error:
make object! [
    code: 500
    type: 'access
    id: 'cannot-open
    arg1: "/C/Documents and Settings/endo/mods/mod-embed.r"
    arg2: none
    arg3: none
    near: [do any [get-cache file file]]
    where: 'do-cache
]

Works great. Thank you!

the embedded mode is for providing an HTTP server to an existing 
app, not a full-featured Cheyenne
 But I can use RSP, websockets and webapps I think, right?

If you want a full-featured Cheyenne and integrate your own GUI app, 
you would have to make it the other way around, which is include 
your app in %cheyenne.r.

Oh but there a "testapp" in embed-demo.r? and its not plain HTML.

No, it's the embed API, see 'publish-site and 'testapp specs in %embed-demo.r

Thank you. I'll try.

What I'm working on is, to poll a database table and send some changes 
to a client on a web socket. polling should not block Cheyenne of 
course.

My plan is, adding database to Cheyenne config, poll in an aysnc 
way (jobs, task-master, etc.) and send something to clients over 
web socket.

Web socket + database query: it won't block if you transfer the query 
job to an RSP script. See the ws demo app: %www/ws-app/ws-test-app.r 
and the back-end RSP script: %www/ws.rsp

Just something to keep in mind when working on websockets: the transer 
mode used by Cheyenne to reply to clients is "text" mode. This mode 
requires UTF-8 encoding and IIRC, the browser is allowed to reject 
your response and close the connection if the response is wrongly 
encoded.

yep I just downloaded yesterday.

But actulally the donwloaded program downloads another one, so it 
may be different for Home and Pro.

Chat demo: in fact, it should work ok in all cases, because the UTF-8 
encoding is done by the browser and the chat back-end just broadcast 
it as is to others.

It looks like I can use on-timer event to do my async polling and 
then sending the results for each client.

You can just pass a block! value as a job to the scheduler and call 
your code from there, it would be cleaner than hacking in on-timer.

One last thing: in ws.html there is a button to close the connection:

<button onClick=" alert('button closed');conn.close();"> Disconnect 
</button>


But I never get onClose event and don't see a connection close on 
server logs:
conn.onclose = function(evt) { alert("Conn closed"); }

And "tick"s appear even if I click on Disconnect.

And one more thing: I think I found the reason of "Warning in [boot] 
: Premature exit from event loop" message in on-timer event, and 
fixed:

client removed AFTER the on-disconnect event, so CLIENTS are not 
EMPTY? while the last client disconnecting.

I changed the 43. line on ws-test-app.r file from:
  if empty? clients [
to 
if 1 = length? clients [

And problem solved.

When I remove AUTH line from WEBAPP section in httpd.cfg file, the 
browser always returns :ERR_EMPTY_RESPONSE error.
Here is my part of httpd.cfg
	webapp [
		virtual-root "/testapp"
		root-dir %www/testapp/
;		auth "/testapp/login.rsp"
		;debug
	]


in %testapp/  I have test.html and test.rsp they both very simple 
files, I have app-init.r also. But I can never access those files.
In Cheyenne log file I see following error:


10/12-3:11:39.3120-## Error in [uniserve] : On-received call failed 
with error: make object! [
    code: 303
    type: 'script
    id: 'expect-arg
    arg1: 'second
    arg2: 'series

    arg3: [series! pair! event! money! date! object! port! time! tuple! 
    any-function! struct! event!]
    near: [either url: second pos: find]
    where: 'process-webapp
] !


As a work around I put auth line to a rsp file that just do session/content/login?: 
yes and redirect. 
Do you have any idea? I tested with 0920 and r164.

Doc: "You can just pass a block! value as a job to the scheduler 
and call your code from there, it would be cleaner than hacking in 
on-timer."

How do I interact with "clients" connected via ws inside a "job" 
block?

I have "clients" block in on-timer event and I able to send messages 
to them. Is it possible from jobs?

2) in ws-test-app.r file it says do-task/on-done takes an argument 
that can be anything.

But when call do-task/on-done with any non-string argument I get 
an error, am I doing something wrong?

I tried with a port (connected ws client) and a integer, both failed:


11/12-6:17:04.1090-## Error in [mod-socket] : On-timer call failed 
with error: make object! [
    code: 303
    type: 'script
    id: 'expect-arg
    arg1: 'do-task
    arg2: 'data
    arg3: [string!]
    near: [do-task/on-done 5 func [client data] [
            send client join "Job end: " reform [now/time tab i]
        ]]
    where: :action
] !

Also, the ws-test-app test application has been fixed too and now 
works correctly (in Chrome at least, I should add support for FF 
too).

OK. I saw it as a REBOL process was suddenly racing at 100% CPU. 
Someone accessed my site, which posted an entry in the default-access.log 
with an HTTP 1.0 request:


74.52.168.98 - - [25/Dec/2011:10:30:29 +0100] "GET / HTTP/1.0" 200 
9


Then 5 minutes later, the none-access.log appears and I'm flooded 
with requests until that log is 45 MB in size.

The file starts like this:

.168.98 - - [25/Dec/2011:10:35:54 +0100] "" 405 -
74.52.168.98 - - [25/Dec/2011:10:35:54 +0100] "" 405 -
74.52.168.98 - - [25/Dec/2011:10:35:54 +0100] "" 405 -
74.52.168.98 - - [25/Dec/2011:10:35:54 +0100] "" 405 -
.... 45 MB of this

perhaps we need some kind of test server set up, which we can throw 
garbage and flood requests at? it could be useful for testing such 
things.

I've tried a lot of different hand-crafted request to try to crash 
Cheyenne and reproduce your issue, but Cheyenne is keeping responding 
right.

btw: I yesterday used Ladislav's excellent!! closure with cheyenne 
and it didn't work. Then I found it's because of do which it uses 
.. It worked once I changed it to *do

It can cause some problem only when you're changing Cheyenne version 
or when you're upgrading your app and changing the data structure 
format stored in sessions. You can enable or disable it using the 
PERSIST config keyword described here: http://cheyenne-server.org/wiki/Config/Persist

The main point of making sessions persist on disk, is to allow Cheyenne 
to be restarted without loosing user's sessions (and force them to 
log in again).

There is not much that can be done at HTTP level to protect against 
that, unless you get back to HTTP/1.0 and forbid keep-alive states. 
It is also worth mentioning that multithreaded servers (one client 
connection = one thread), will suffer much more from such attacks 
than event-based single threaded ones.

Does cheyenne websockets work with socket.io (http://socket.io/) 
or some other third party client library that allows websocketes 
to work on mobile and non websocket browsers?

The plink example didn't work for me. I got the following message:


>> win-call/output "plink.exe -l user -pw password [checkall-:-host] 
 df "  str: make stri
ng! 102

4
== {CreateProcess failed!: The system cannot find the file specified.}

Though I got a similar message from the console:

C:\Documents and Settings\Peter>plink.exe -l user -pw password [checkall-:-host] 
 df

'plink.exe' is not recognized as an internal or external command,
operable 
program or batch file.

I was using an older version (not sure which) so I upgraded to newest 
version and email works! but now I have some .rsp scripts failing 
: #[object! [ code: 311 type: script id: invalid-path arg1: MTA arg2: 
#[none] arg3: #[none] near: [port-id: any [ all [ value? 'servers-port 
block? servers-port servers-port/MTA ] 9803 ]] where: context ]]

I converted the script to .rsp and it still gives an error. Must 
be something in the script but the same script runs on previous version 
of cheyenne and the new version does run other .rsp scripts.

I found out recently that Cheyenne has trouble with the PHP-based 
DokuWiki. Using FastCGI, it will work until you open the Media Manager 
and then Cheyenne will "hang". You have to restart Cheyenne. I haven't 
had time to troubleshoot yet.

My Win2003 issue is still not solved, but I have one additional question,

RSPs can read/write anywhere in the server, even they are in a webapp. 
Is it not a security hole? If someone able to upload an rsp file 
and execute it, he can delete a vital file from the system, or read 
whole directory structure.
Is there a way to run Cheyenne (or a webapp) in a sandbox?

I have another issue, Cheyenne hangs if do-sql/flat used and the 
result is empty. if no /flat it works, returns empty block.
result: do-sql/flat 'test "select top 0 * from mytable"

this should also work  (and should theoretically be faster:

 until [not if data: pick port 1 [append out data]]

Gregg, I know your take on optimisation... ;-)


<start rant  ;-) > If I had the same opinion, liquid would still 
be 10 times slower than it is now.   each little part of the changes 
add up and after years it really adds up.  I have some new changes 
which will probably shave off another 5-10% when they are done.  
 It requires several changes (some probably removing  less than a 
%).  its been like that since the begining.  the relative impact 
of any optimisation is always bigger the more you do it.  


the first 1% looks like nothing when you compare it to the original 
100% but after you've removed 25% its now 1.33%... but when your 
app is 10 times faster, that 1% is now 10 % of the new speed.


btw, I'm not trying to justify this specific optimisation, but I'm 
trying to balance the general REBOLer consensus that optimisation 
(in speed, code size and RAM use) isn't important when compared to 
other Reboling tasks... 
 

Have you (using "you" in the general sense, not gregg specifically) 
ever looked at Carl's Code?  its some of the most optimised and dense 
code out there... it hurts the brain... and its very enlightening 
too.  all the little variations in the series handlers often are 
there by  design and all balanced out against the others.  Carl uses 
all of those little variations profusely... to me its in the very 
essence of REBOL to be optimal and constantly be refined, improved, 
and this usually means by shrinking it in all vectors.  

<end of rant  ;-) >

I agree with you Maxim.

But on the other hand, I never allow my team to sacrifice the readability 
unless IF there is a really good reason to do that AND IF they put 
enough comment WHAT and WHY they do that. Otherwise it wastes our 
time to "solve" the optimisation a few months later.

Optimisations are good / necessary if they are (can be) in loops 
(mostly), but they are dangerous especially if you are writting lower 
level code (mostly C, even Java) because compilers use code-patterns 
to optimize your code, so your hand-optimized code prevents compiler 
optimizations on modern compilers.

And worst, if you are using assembly, your optimizations may prevent 
to use CPU caches and for example when it works faster on one CPU 
it can be even slower than the original version on another CPU.

By the way, I tested on 3 different PC with R2, while loop was always 
faster then the until loop in the above case. Of course I didn't 
test the real case (using do-sql with millions of rows)

And of course Carl should write very optimized code in REBOL because 
everything else is depend on it. 
Hmm I think we should move to another group :)

If I had the same opinion, liquid would still be 10 times slower 
than it is now.


That's big enough to matter, not just measure. I think, that's what 
Gregg pointed out. And remember, there is not a final truth about 
this, as what matters to one person, might not to another.

My thoughts, too. There's also the matter of why REBOL and AltME 
are such slow memory hogs if Carl's code is so good. The reasons 
are found elsewhere. REBOL systems have aspects of penny wise, pound 
foolish. The actual code needs to be optimised to limit the damage 
a bit

I believe in optimizing on a case by case basis, as most do. And 
I believe in optimizing different things in any given case. Size, 
speed, felxibility, and readability are all fair game for optimization.


As far as AltME and other slow REBOL UIs, I remember Carl saying 
once that View is a miser, saving pennies, while VID is the government 
and spends millions. I think whoever designed the list model used 
in AltMe and other apps (e.g. IOS conference and messenger) chose 
to make the implementation small and quick to write, knowing that 
they might not be fast. They may also not have imagined how far they 
would be pushed.

View is a miser...

 That certainly depends on how you use VID. VIew's SHOW function is 
 a big bottleneck and the most important function of all to optimize 
 for. But yes, AltME lists are rather heavy.

Altme is a slow UI? Just don't make me laugh, those who claim so. 
Have you tried Azureus? That's Java. Altme just runs ok. We all know, 
that the UI is not OS compatible, so there are things to be desired. 
And altme server can run for months without a restart.

Altme is consistent speed wise. I have 3 years not reinstalled my 
Vista, so once my ntb boots, it takes 5-7 minutes to be usable.I 
click on many apps to start, and Altme definitely starts first. Stuff 
as Outlook, Firefox, etc., take usually tens of seconds!

Maxim: you are mostly talking about the overall performance gain 
in the whole system. But the case above (Cheyenne, do-sql bug) we 
gain 2-3% performance just in do-sql function, IF it's used in a 
millions loop (or returns a few millions of rows) AND only IF you 
use native drivers AND only IF you use /flat refinement. which leads 
very rare cases. So the overall perfornance difference is  very very 
low. (well, still WHILE faster than UNTIL in above case anyway..)

You are right, giving a link is better than including the whole dll.

What do you think about making a setup package for Windows? Including 
that dll and some configuration options during the setup, service 
mode default, etc.
If do you think its worth to do, I can do it.

@Endo, you can do it, and Doc can link to it :)

so far I haven't had any problems with the R3 side of things including 
compiling my own extension with a few new funcs in it...


But now I'm starting to implement the GL specifics and just hit a 
snag.

MILESTONE !!!  I've got an OpenGL window up and running... called 
from within REBOL  3  :-)

yes Geomol did, but the limitations of R2 can't make it native... 
meaning, you don't have access to the binary part of the data and 
use it directly within normal datatypes.

(and in a few KBs)

I need to wait for callback and/or image! specific enhancements to 
API to make it REBOL controlable

I also have a client which needs to visualize genomic data and millions 
of nodes in space.... so this is the first step to getting that contract 
 :-)

Open GL gfx source is mostly  on the C side for now because I'm still 
playing around with the extension and I can't interact with GLut 
or OGL until the extension API is expanded a little bit.

And this comes in between:
to pair! 2

And as-pair [2 2] is significantly slower than either.

Nice tuning experiment! There will be other surprises too, I'm sure.

I created this group, cause ever so often these experiments come 
around and we loose them... this should become a quick repository 
of tests & disussion for profiling discoveries we do.


any test should provide the full test command-line code, so its easier 
for other to copy-paste & compare without having to re-type.


ever so often, a compilation should be done, highlighted in a different 
color for easy referral...

probably should add Rebol version and platform in any further profiling 
compilations... to make them even more usefull as a reference.

AS-PAIR and TO-PAIR are mezzanine. Carl has mentioned wanting to 
make AS-PAIR native - sounds like a good candidate.

my framework for tests and benchmarks will be released... eventually 
:)

Ah, you're requesting that the math operators apply on blocks of 
scalar (vectors).
Old request.
Never done in R2 and not yet in R3

Moreover, maximum-of and minimum-of are probably the only ones functions 
faster with R2 than R3.

In R3, they have been turned back into mezzanines (never understood 
why)

btw


'PROFILE is a handy function I built which accepts ANY function with 
ANY args and repeats the test until it takes longer than one second, 


you can adjust its loop scaling by varying amplitude and magnitude 
of loops at each iteration.


'REPORT-TEST simply dumps human-readable and easy to compare stats 
of calls to profile (which returns a block of info on the test).

If you are trying to find the largest in a series of not-strictly 
comparable items, then be aware that R2 behaves differently to R3:
     b: reduce [1 none 12-jan-2005 unset 'a copy []]
     last sort b       ;; r2 and r3 agree
     maximum-of  b   ;; r3 has a headache
== [[]]

I'm doing an in-depth analysis of various looping funcs... and discovering 
some VERY unexpected results amongst the various tests... will report 
in a while when I'm done with the various loop use cases.

and remove-each is 90 times faster if it always return true rather 
than false !

wow I'm already at 7kb of output text with notes and proper header 
... I haven't done half the tests yet!

Yeah, there's often a huge difference between a mezzanine function 
and a native. In R2, FOR is mezz, REPEAT is native.

but I'm discovering a lot of discrepancies in things like string 
vs block speed of certain loops... 
and a lot of other neat things like:

pick series 1   

is  15% faster  than

not tail? series

and its not because of the paren... I checked that....

profiling almost done... my machine has been looping series and indexes 
non-stop for about 8 hours now  :-)


be ready for the most in-depth analysis on loops ever done for R2 
 ;-)

See who is the overall winner in this REBOL iterator slug fest analysis!!!
   

over 8 hours of practically non-stop cpu cycling over a wide variety 
of exit conditions, datasets and ALL iterators in rebol 2 

(loop, repeat, for, forever, foreach, remove-each, forskip, forall, 
while, until )

20 kb of data, statistics, comments and test details.

INVALUABLE data for people wanting to optimize their REBOL code.


http://www.pointillistic.com/open-REBOL/moa/scream/rebol-iterator-comparison.txt

A thing should be noted.
repeat and foreach do a bind/copy of the evaluated block.

Even if they are the fastest loops, they should be not used too intensivly 
because they will polluate the memory.

It's particularly sensitive for graphics applications or services 
that linger in memory. 


So, that's why  I advise to use only LOOP, WHILE and UNTIL for intensive 
repeated loopings, if you don't want to blow up the memory used by 
your app.

thanks steeve, I'm accumulating all comments

First revision of the benchmarks will include:
	-RAM stats

 -empty vs filled-up loops.  many words and a single func with the 
 same content called from the loop
	-GC de-activated tests + recycle time stats

as noted in the document test notes:

I specifically didn't do any GC control, cause I wanted, at this 
point, to see how the loops react under normal rebol execution.  

the GC normally is pretty aggressive and when you look at the tests, 
most loops roll for several hundred thousands times, so the GC will 
have kicked-in... if it can.

I did note, that there is a HUGE memory leak which occured probably 
in the actual benchmark procedure itself.


although I keep no reference to any of the data or transient test 
blocks and funcs, they are kept somewhere, and my rebol.exe process 
keeps growing and growing.... I caught it at 500MB !! but it didn't 
do any difference in actual speeds... after a few tests.... cause 
i was a bit scared.

and if you activate recycle/on, does that make any difference ?

I think R2 GC can't determine co-dependent unused references... in 
some situations.
ex:
blk: reduce [ a: context [b: none] b: context [c: a] a/b: b ]
blk: none


in this case both a and b point to each other, and clearing blk doesn't 
tell a or b that they aren't used anymore... that is my guess.

I reduce a block which is the test... and since foreach copy/deep, 
and there is NO word ever refering to the content of the refered 
block, I think the contents of the blocks prevent the blocks and 
the data they contain from being collected... 


the block contains words which are not GC counted as zero reference, 
so nothing gets de-allocated...

that's just my guess.

in any case I want to build a single script which does all the tests, 
statistics, and eventually graphics and html pages of all results 
in one (VERY) long process.

so I can better control how the tests are done and prevent automated 
test creation as I am doing now.