AltME groups: search

BrianH: "and leave the periferal stuff to other people" - I know, 
what I am trying to point out though is, that - it does not work 
(as can be seen with networking). The GUI would not be here, if it 
would not be sponsored by Robert. So I just asked, how much is eventually 
needed, for someone taking the DLL bounty? I surely am not able to 
write it myself, nor are other ppl, but we might be able to collect 
a sponsorhip fee :-)

HTTP protocol is one of the simpliest Internet protocol to support, 
nothing that a skilled developer like you can't handle. :-) Anyway, 
I would be pleased to answer questions, as my free time permits.

Cool, thanks. And I expect that the Q&A will likely be asynchronous 
:)

That's standard CGI, but what would be needed to give R3 a Fast CGI 
interface?

A FastCGI server protocol implementation in R3.

Are requests serialised to a FastCGI server, or is the server supposed 
to multi-task them?

FastCGI requests are serialized by Cheyenne, but without tasking, 
R3 will still process one at a time, so any blocking call or long 
processing will block the whole server.

Yup. The downside to app pools is that you have to coordinate access 
to the data amongst multiple interpreter instances (less cache, more 
disk access), but that's not as big a problem for mostly-read apps.

Cheyenne has some experimental support for FastCGI multiple server 
instances (multiprocessing) but this has never been really tested. 
The balancing is very simple, distributing requests using a simple 
round-robin method. Pool managment is minimalistic, starting all 
the instances and killing them on Cheyenne's quitting, no restarting 
or failover handling.

Interesting. So unlike UniServe, there's a fixed number of instances, 
and requests to one instance are serialised? That would be quite 
workable

UniServe worker processes don't support multiplexed requests (multiplexing 
is the right word used in FastCGI specs IIRC). The FastCGI multiplexing 
mode requires a form of multitasking support to be able to handle 
all the incoming requests in a multiplexed way. Without that, you'll 
end up with just multiprocessing, which is what UniServe+Taskmaster 
are doing for CGI and RSP scripts.

For example, PHP in FastCGI mode supports multiplexed requests over 
a single FastCGI socket, while dispatching the load internally between 
several PHP processes (or thread under Windows). In that mode, the 
main PHP process manages the thread/process pool alone, and starts 
new instances when required.

There's no need for session affinity internally in Cheyenne, the 
session context is carried with the request to any worker process.


CGI/RSP requests are dispatched to any available worker process. 
If all are busy, a new one is forked up to the maximum number (8 
by default). If the max number of workers is reached and all are 
busy, the request is queued (the queue is global). Once a worker 
becomes available, it gets a new request assigned at once.

Btw, worker processes are not equal wrt the load. The first in the 
list gets the more jobs to do, especially if requests can be processed 
fast (which should be the case in most of well-coded scripts). So, 
you get a natural "affinity" to the first worker (having the most 
code and data cached there) for all new incoming requests. 

So, in the long run, you can look at workers memory usage to see 
how the workload is spread, the first one having the biggest memory 
usage, the last one having the lowest. Looking at the last one is 
a good indicator if the number of workers needs to be raised or not, 
if his memory footprint hasn't changed since the last restart, your 
server is doing fine with the load. I should expose some workers 
and job queue usage stats to help fine tune workers numbers.

The worker number doesn't decrease unless you're using the -w command 
line option (or unless your code or a native bug crash some of them 
badly). Having more than 8 workers is possible if some of them are 
blocked (in a endless loop or waiting something forever). If you 
quit Cheyenne in that case, they'll remain there and will need manual 
killing. Cheyenne could do a better job at handling those non-responding 
workers in future versions.

R3 is more suited for such.
secure [memory integer!] and secure [eval integer!]
Allow to quit from forever loops.
(Not from a forever loop, which does nothing, though)

yeah perhaps... is there a ticket for that request ?

I don't know if there's a ticket. I could live with relatively coarse 
granularity, which would be much better than nothing, if that at 
least made it possible.

At least a memory constraint is a good one. That should prevent memory 
leakage. I personally don't like eval at all, as my brain is not 
mature enough to be able to guess, how many cycles will my script 
ideally need. I would welcome time constraint as well, and it was 
proposed, but not accepted.

Here's what does not work yet:


# If the program quits, the exit code is set to 101, the same as 
any security termination; however, we may want to use special codes 
to indicate the reason for the quit.

# Some types of loops are not yet checked, but we will add them. 
For example, PARSE cycles are not yet counted.

# Time limits are not yet supported, but may be added in the future. 
However, the cycle limit is better for most cases, because it is 
CPU speed independent.

You can also do it in a try statement and check the error codes.

I saw the news yesterday for FF4. The badly written RFC for ws was 
alarming since the beginning, this kind of design defect doesn't 
surprise me much, but it's such a waste...I hope the new CONNECT-based 
workaround will be adopted rapidly.

There's a possibility that users might require certificates to access 
my app ... can that be done with Cheyenne and something else?

Sorry, that was a question, not a suggestion. Do you actually want 
to authenticate based on client certificates?

Use SSL: http://it.toolbox.com/blogs/securitymonkey/howto-securing-a-website-with-client-ssl-certificates-11500

Or use Nginx as a reverse proxy to Cheyenne.

Looks like I could still use stunnel .. and put all the certificates 
in a directory and disconnect users who don't present a certificate

I might have a few hundred users ... so hopefully this can cope.

I'm looking to implement a nationwide registry for patients taking 
biologics

Shouldn't be much of a problem performance wise

How many requests would they make within a minute?

60 Concurrent users is a lot - except if they're idling for five 
minutes between every request

I guess I can just get a bigger VM if it comes down to it ... :) 
 Yes, I expect they'll be idling most of the time

If it's a Unix box, you use tools such as uptime, top and htop

There's QM from ChrisRG but not sure it reached a official release.

There's a Guest account

Now if a certain big pharma company goes ahead to pay for this as 
they have said .. I'll put "powered by Rebol" banners everywhere 
:)

RSP is a template system. Magic! generates pages from REBOL dialects

Graham, if you're concerned about performances and scalability, I 
don't understand why you're considering a CGI-based solution? AFAICT, 
Cheyenne/RSP is the most scalable solution for running REBOL code 
on server-side. If a single Cheyenne instance is not enough, you 
can use a nginx front-end to server static content and dispatch the 
load over several Cheyenne instances.


Our biggest online web-app has ~800 users, with several hundreds 
logging in each day, accessing a 400MB RDBMS with millions of records 
served by a single Cheyenne instance, with 8 workers (each taking 
15 to 25MB of RAM) running on a 2.2Ghz DualCore CPU with 2GB of RAM. 
Average Cheyenne load is 2 req/s with peaks up to 100 req/s (usually 
caused by web scanners or attacks). The webapp is still performing 
well under these conditions, and we have plenty of space to improve 
performances when it will be required (with faster hw and more instances).


Also, a key factor is optimizing webapp's code, especially for the 
most used RSP pages. Btw, a profiler would be a good addition to 
Cheyenne/RSP framework.

This is a result of not overriding the default toString() method 
in your pain, gs, fn, rapid3, fatigue, ros and ems objects.

..but I'm surprised that they are trying to use objects in such a 
way.

If you can send me the sample, I'll try to take a look for you later 
on.

crap .. need a compose

In the "I hope this group thinks there are no stupid questions" mode, 
if I get new View 2.7.8.3.1 from 1-Jan-2011, how do I run that with 
Cheyenne?  I tried an .rsp with Funct and it flunks.  So my Cheyenne 
version (a win exe) is not using new R2.

2.7.7 has FUNCT too, so what you have is a 2.7.6 build.

I haven't used 2.7.7 due to a few issues that made it look less stable 
than 2.7.6. I can't afford to use a new version with regressions. 
Hopefully, SDK 2.7.8 looks more polished and it runs Cheyenne flawlessly 
on Linux platform. If the Windows SDK is ok too, I'll upgrade my 
Cheyenne build chain to 2.7.8.

MikeL: 

- Download Cheyenne revision 122 source package from here: http://cheyenne-server.org/download.shtml
- Open a 2.7.8 console
- Type: 
>> cd %<path-to-cheyenne-folder>
>> do %cheyenne.r

Thanks I had tried that combination a little earlier and after running 
in Windows Auth mode, I get  "Can not open server Rconsole on port 
9801" and "Can not open server Logger on port 9802" as Uniserve errors.

Cheyenne is working fine from sources with 2.7.8 (XP)... Check what 
is using the port.

You can use for example PortQry util: http://support.microsoft.com/kb/832919
Which can show you something like that: 
...
Process ID: 1376 (rebol-view-278-3-1.exe)

Process doesn't appear to be a service

PID	Port		Local IP	State		 Remote IP:Port
1376	TCP 8080  	0.0.0.0 	LISTENING	 0.0.0.0:22712
1376	TCP 9799  	0.0.0.0 	LISTENING	 0.0.0.0:41163
1376	TCP 9801  	0.0.0.0 	LISTENING	 0.0.0.0:43253
1376	TCP 9802  	0.0.0.0 	LISTENING	 0.0.0.0:22573
1376	TCP 9803  	0.0.0.0 	LISTENING	 0.0.0.0:40994
1376	TCP 9799  	127.0.0.1 	ESTABLISHED	 127.0.0.1:1116
1376	TCP 9799  	127.0.0.1 	ESTABLISHED	 127.0.0.1:1117
1376	TCP 9799  	127.0.0.1 	ESTABLISHED	 127.0.0.1:1118
1376	TCP 9799  	127.0.0.1 	ESTABLISHED	 127.0.0.1:1119

Thanks for the help.  R2 is listening on 9801 and 9802 now.   Next 
step is to get it to server a page.

I don't know what he did to make Cheyenne better, except ACCESS-OS. 
There were other native fixes to Linux builds mentioned, but I can't 
test on anything other than XP at the moment so I didn't ask for 
a full list. Given the "coordinated with DocKimbel" comment I figured 
you knew better, but that might just be the ACCESS-OS thing, which 
currently does nothing on Windows, and doesn't require an SDK. We'll 
know when the changes doc is updated.

It was in the planning since a year

I was also a bit surprize by the inclusion of those OS natives instead 
of just freeing /Library on Linux, but I guess it's business related.

I'm also surprised by the bad design. ACCESS-OS is extremly ugly. 
To kill a process:

access-os/set 'pid 29110

Library wasn't in Core at all. It was in View, so that was just a 
matter of tweaking the licensing code. For Core it would mean either 
adding the licensing and library code, or just renaming Pro as Core. 
View did get Library in 2.7.7. It's not just a business thing, it's 
also a matter of time and effort.

Andreas: I share your feeling. I guess that it's easier and safer 
to add a new native in R2 than integrate all that stuff into system:// 
port (using set-modes/get-modes as interfaces).

I am in favor of releasing Pro as Core and just disabling the licensing 
code; that would give us Library on Core a lot quicker than trying 
to retrofit things into Core. R2 is quite an old codebase, so a lot 
of changes would require a huge amount of work - R3 was rewritten 
for good reasons.

I can't get cheyenne to serve to an ajax json request.


I can get it to read the array as a local file but it does not seem 
to work through url request.


I played with content-type: application/json which I read was needed 
but I don't know if I'm on the right track.

This doesn't look like a valid URL: 'http://localhost.jsontest.cgi'

Alan, I'm logged in to AltME from Ubuntu - so many non-ascii characters 
get displayed incorrectly. In your script the closing double-quote 
after /jsontest.cgi doesn't display properly. Perhaps you could check 
that it really is a double-quote and not a "smart-quote" in the actual 
source.

Amacleod: Your CGI script headers looks very wrong:
- What are those "%s" on the shebang line?

- /cgi-bin/rebol.exe: this doesn't look like a valid filesystem path

- Why the Content-Type header isn't emitted as required by CGI specification?


Maybe you should read again documentation about REBOL CGI usage on 
rebol.com site and also have a look at CGI sample scripts provided 
with Cheyenne source package. Understanding what a shebang line is 
might also help: http://en.wikipedia.org/wiki/Shebang_(Unix)

No, you can't use remote server .. this is a security feature.

Additional Notes:

Due to browser security restrictions, most "Ajax" requests are subject 
to the same origin policy; the request can not successfully retrieve 
data from a different domain, subdomain, or protocol.

Script and JSONP requests are not subject to the same origin policy 
restrictions.

Sorry, I thought it might be a cheyenne issue...i'll move to javascript 
group

Longshot here .. but has anyone written a code formatter for RSP 
pages .. so that it correctly indents HTML, JS, and Rebol all on 
the same page??

there's a php pretty print here http://beta.phpformatter.com/

Cheyenne reached a new milestone, as promised during the ReBorCon, 
0.9.20 is out! 

Read the announcement here: http://cheyenne-server.org/blog.rsp?view=26

All the new binaries are built using REBOL 2.7.8, enjoy!

Complicate expressions to tell you about a request blocking new feature 
with pattern recognition (see Changelog, look for "block").

My RSP scripts do a lot of SOAP and REST calls which use my mods

Still looking forward to a virtual drive so we can encap all the 
sources for our rsp scripts with cheyenne!

You're using a literal string as response, you should try to add 
a COPY in front of it to avoid side-effects: response/buffer: copy 
{<svg></svg>}. Even better, as you're in a templating system, make 
it a template:

<% response/set-header 'Content-type "image/svg+xml" %>
<svg></svg>

Have you put the LOCALES-DIR directive in a webapp definition?

I tried a lot of combinations .... %catalogs %www/catalogs with the 
.cat files at all levels

I don't remember exactly right now but I think when I used cheyenne 
in multilingual I had some problems with this too, and I hacked RSP.r 
(I think) I hardcoded some path to make it work .. it's a while

MikeL: I'll write a small test using the example from the doc to 
see if there's something wrong there. I got localization working 
correctly in CureCode, but that's the only webapp where I used it, 
so unknows issues with paths are not excluded.

Jobs are quite experimental, but should be working in the general 
case (I'm using it on cheyenne-server.org to run a batch script every 
hour).

this will seem like a dumb question but....


anyone know how I can get worker threads in cheyenne to actually 
open a console and allow on-screen tracing?


right now I've got 4 workers and no one opens up a console screen... 
I really need this real-time tracing, since the server is being monitored 
in real-time using remote-desktop or vnc.

there's a single logging process I think

my client had one console for each process on his cheyenne setup... 
I need to see what is happening independently from each other.  trying 
to follow 5 high-speed traces in a single file, is like using altme 
without any groups  it would be impossible.  ;-)


if one worker has a fuck up i need to be able to see why *it* is 
failing.

my client
  (as in a customer who was using cheyenne)

but that is a single file for several thread... its impossible to 
use when there is traffic.

maybe a system/words/print or somethng like that

I don't want logging... I need to look at it live... these services 
actually do network calls and wait on other servers... I need to 
look at them run each on their own... if one blocks... I don't want 
all the other logs to push of the view.


also, the worker process might be having mutually exclusive or causal 
side-effects.   with a different console view opened side-by-side 
its easy to see all these effects.  (one window jams when another 
runs... continues when the other is done... etc.

To open a console window for each worker process, you need to change 
the CALL command line from %UniServe/uni-engine.r:

    call join form to-local-file system/options/boot [" -qws " cmd]

Replace:
* CALL with CALL/SHOW
* -qws with -s


Also, in that case it is also more practical to reduce the worker 
number to 1 using Cheyenne command line option: -w 1

guys, how can i get rid of these chey-pid-*.log files? or at least 
i would like to put them under a %log/ dir

As a quick fix, I can patch %cheyenne.r to redirect %chey-pid* files 
to %log/ folder and upload a new Windows/Linux binary (OS X would 
take more time). Would that suit your needs?

Will probably do that in a couple of hours (after lunch).

I would be pleased, but unfortunately, Phuket is a bit far from Paris. 
:-)

half a day... think about it ;)

Dockimbel: i sent u an email with some background story which can 
help to think about a good solution

i would help making these patches but i can't run cheyenne (on a 
mac) from source. can u, maxim?

(running rebview -qws cheyenne.r even with -w 0 is really ugly under 
osx, because of the 2 black app icons which doesn't even have a window 
associated.... :/)

but it works with rebview. would be nice to still have a /core only 
version

so, i can't run it from source on a headless linux server, right?

Yes, it can since 0.9.20, thanks to 'access-os native, and it will 
in a few minutes on OS X too (requires /Core 2.7.8)

doc, having issues with a handler I am building.  


my mod is using send-msg (taken from the RSP mod) in the make-response().


the handler actually pops-up and it runs the proper code, I can see 
the printing in the console.

when my god gets its task-done called, it has the proper code in 
the result....
 all seems good..   :-)

except:

   the browser doesn't get the response and it hangs (telling me that 
   its waiting for the server).

any clues?

and u only call it when there is a POST request?

I have included the READ-CGI mezz in CGI.r to make CGI work on /Core 
2.7.8 on OS X. Testing, will push it in a couple of minutes.

Script: "Encap virtual filesystem" (21-Sep-2009)
REBOL - Security Check:

Script requests permission to execute a system shell command: netstat 
-f inet -p tcp -na
Yes, allow all, no, or quit? (Y/A/N/Q)

Answer: A

It is a Cheyenne thing to check for available TCP ports for listening.