AltME groups: search

I used your latest svn trunk and encapped it with encmd

encapped with encmd and not encmdface?

I want to try to plug R3 into Cheyenne to test CGI under Windows. 
I can't understand, how Cheyenne handles CGI though. I can find handlers/CGI, 
where it does some strange stuff. First and foremost - the shebang 
line is ignored, yet CGU it works - you can entirely delete it. I 
can get shebang section evaluated only if I delete REBOL header. 
This is confusing for me ...

CGI handling: two different strategies are applied :


- if the target script is a REBOL script, the process is already 
a REBOL session, so no need to start a new one (and avoid the startup 
cost, so you get FastCGI speed). Shebang line is ignored in that 
case.


- if the target script is not a REBOL script (no REBOL header), classic 
approach is used: setting of environnemental variables, CALLing executable 
from shebang line, sending input, catching output,...

I wanted to make it easier for CGI coders to switch from Cheyenne<=>Apache. 
Using your solution would force users to add/remove the shebang line 
each time you change server (like debugging locally on Cheyenne and 
putting in production on Apache).

I chosed the simplier and most efficient solution for the user. Having 
to support REBOL scripts using same  header than R2 but with a R2-non-compatible 
kernel was not something planned.

btw - it seems that Carl's beta plan might be a bit larger, if he 
is about to accept many additional user requests. There are still 
important things missing, and without them it imo does not make sense 
to call R3 a beta ...

In my book "beta stage" is reached when the product is (mostly) feature-complete 
and all the known bugs are fixed.

Or ... better yet, use NAT to reroute port 80 to something above 
1024 and run Cheyenne from user space listening on the non-priv'ed 
port??  Safer (I think) if anyone ever assumes control of your server 
internals.  Or do you need root powers for something other than listening 
on port 80?

it was quite simple , I just joined two examples from their tutorials 
.. the SSL and the reverse proxy one .. I can give you the config 
file

how I use it: nginx get all the requests, if a file is not found 
it will proxy-reverse the request to cheyenne wich is configured 
with a on-status-code [404 trap.r] . trap.r check the url request 
and either load and respond with an rsp or send a 404

That requires a virtual filesystem and deep patching Cheyenne to 
use it. I've tried to make one a few months ago, but hadn't enough 
time to finish it. It's a feature I need too (but low pri currently).

SVN r17 : big update, lot of code added mostly for the new embedded 
async Mail Transfer Agent (MTA)

FEAT: email async sending sub-system (MTA) added . 

FEAT: added two new functions for email support in RSP/CGI scripts: 
'send-email and 'email-info?
FEAT: added email sending demo form %www/email.rsp.

FEAT: Cheyenne's main process ID is now exported in /tmp/cheyenne.pid 
on start and deleted on quit. (All platforms except Windows).

FIX: fixed broken global words protection in RSP.r.
FIX: HTTP Date headers are now in UTC format.

FIX: "Reset Workers" menu wasn't working in service mode. Fixed now. 
(Thanks Will)

FIX: SVNr4 regression bug on system port for UNIX fixed. (Thanks 
Will)

FIX: multipart file uploading code refactored. Fixes many bugs in 
file uploading.

A few words about the new MTA and email capabilities :


- Emails are written to disk before sending in async mode. Upper 
email size limit is 2GB. Concurrent emails sending supported (limited 
only by machine resources).

- Only the higher priority MX is used (fallback on others MX should 
be added soon).

- The MTA currently supports only 8BITMIME compliant SMTP servers 
(fallback to 7bit mode will be added if it still make sense to support 
it).

- MTA will  try to send an email once. If the server is busy, it 
will report a failure. Deferred retries will be added using the scheduler 
module (if MX fallback is not enough).

- Emails queue saving should be added in next update (the queue will 
survive to server restart, but not server crash).

- Automatic failure reports are sent back to emitter (From: field).
- Reports can be fully customized.

- Still needs a lot of testing for tracking small errors or issues 
that can trigger antispam filters (e.g. gmail is very picky about 
correct emails format).


You can test it using the provided test script : http://localhost/email.rsp

Write up your patches and we can see about getting them integrated 
into the next version of R2 :)

stick the email in a SQS queue ... and remove from the queue when 
the send is successful.

Me too, that's why I'm reluctant to implement it in the old fashion 
way (retrying every 8 or 12 hours seems way too long now). I would 
support trying without delays alternative MX servers and if all fail, 
try 1 or 2 minutes after that. That would be by default, but could 
be extended by user if required.

Emails sending and MX querying are all fully async. Emails are streamed 
from disk, no memory overhead whatever the email size. You could 
send hundred (maybe even thousands on fastest machines) of email 
per second using Cheyenne if needed.

Guess we need to field test .. and see how it works

If you're not using your ISP's smtp server, won't the recipient notice 
your domain and the email from address don't match and flag your 
email as spam??

they don't delay for 5 mins and retry .. as it is too expensive

I was quite surprised to see that Gmail and others accepted emails 
sent directly from a MTA on home computer...(I thought that I would 
need to make all testing from remote servers).

and analyse the email instead ...

might as well wear none and dodge things instead  ;-D

Update cheyenne via Svn, ran it .. and it locks up consistently.

Max: I want the web control panel and the one-click webapp deployement 
feature for Cheyenne to make it more attractive to non-rebol users. 
Still a lot of work to make it close to what I have in mind.

it tries to obtain the DNS servers using 2 strategies depending on 
the REBOL kernel used. If GET-REG is available, it tries to get the 
info from the registry, else, it calls "ipconfig" and parses output.

ok encap and the issue goes away then ?  :)

yeahj... tried

call/wait/output "dir" out

and that locks up too

what about vista and xp ??

using localhost I sent an email to gmail.  I turned on wireshark 
and captured all traffic for a couple of minutes.  no smtp traffic.

less than a second in fact with gmail, it takes 1-3 sec here to send 
it and get it signaled by Thunderbird.

Btw, don't forget to check your spam box if email is sent and you 
don't see it.

While testing now, I've noticed a few issue with send-email/attach: 
header and multipart/form-data encoding. I'll fix them tonight, need 
to go now.

Can I use virtual hosts to serve multiple document roots on the same 
site without having separate domain names? I I would like to avoid 
"creeping" between multiple sites that don't share code and also 
use it as a tool to avoid the browser accessing code directories 
for the site.

suppose I have:

somewhere/site1
somewhere/site2

and I don't want

somewhere/site2/../site1 == somewhere/site1

Like so:

1. create dir test/ in the doc root folder.
2. create empty file index.html in test/
3. try to display in browser: no go
4. insert char "a" in index.html
5. "a" is displayed in browser.
6. remove "a" again so index.html is empty
7."a" is still displayed in browser.


If I navigate away and then back to that page using the Back button, 
it still faithfully displays the "a".


If I directly enter the url for the index.html file, I get bounced 
back to the previous URL.

I'm not entirely clear at all from your description what the issue 
is.  But I presume doc is working on fixing the email issue and if 
there are others ... he'll get on to them.

Sorry, if I'm rambling a little bit. It's what I'm observing that 
is straight reposted here. Moving a few milimeters forward after 
a whole day's work is a bit frustrating, particularly when it's REBOL. 
:-)


Right now I just see that app-init.r is not run and that the loading 
of it is wrapped in an ATTEMPT in Cheyenne's source, so I basically 
can't see what's going on. From the docs it appears that the app-init.r 
file is required for a web-app, but request/web-app returns a web-app 
name. So am I running a real web-app or what's going on?

and log the error instead

So, which encapped version is faulty and which one works?

if you have uniserve and cheyenne.r ... of course they can fiddle

Two other issues in the 0.9.19 source:


1. INCLUDE and INCLUDE-FILE do not process paths identically. This 
is not mentioned in the docs.

2. INCLUDE won't include files in absolute paths. It simply changes 
the path to the current path which stores the RSP file that is currently 
being executed. I can't say that I have a fixed include directory 
somewhere at an absolute path. I don't see the purpose of altering 
the path like this.

I use include and include-file only to include files inside the web 
app ... never tried it otherwise

used for footers, headers, and Javascript libraries

Quick warn on current email lib in Cheyenne: it was reported that 
my patchs to REBOL internal functions for RFC compliance are breaking 
some builtin protocols (HTTP e.g.). That's caused by other LF (instead 
of CRLF) deep buried in most protocols implementations conflicting 
with my patches. I'll remove my patches in next SVN build and recode 
all the stuff.

My app reports errors back to me ... 

I do a set-net [ [myemail-:-gmail-:-com] gmail-smtp-in.l.google.com ]

and then send the message to me.

Why don't you use it more and then report back ?

or I could wait for some comments and more documentation by DocKimbel.

Henrik, trying to answer your questions/issues :


to serve multiple document roots on the same site without having 
separate domain names

 => Use sub-domains for such isolation. Everything that's under one 
 domain can be accessed with /.. parent syntax. I think that you can 
 hack it around with ALIAS, custom webapps on-page-start event  handler, 
 but there aren't clean solutions. Use sub-domains for such isolation.

I think webapps require a bit more than static pages?

 => Just to make it clear, webapps are REBOL applications interfaced 
 with external world using RSP scripts. Webapp are not meant to be 
 container for *only* static pages (HTML/CSS/JS/images).


attempt [load join request/config/root-dir %/app-init.r] ; TBD: report 
errors !!! [...] 
RSP: error in events from %app-init.r now logged. 
That's from the change log. That's not correct.

 => Yes it is. What's being logged so far is the errors caught at 
 runtime in event functions declared in app-init. What need to be 
 logged is the LOAD %app-init.r process (syntax errors at boot time).


after a lot of experimentation, the latest encapped version was the 
only one that worked properly.

 Both encap and sources versions works well on Win/Mac/Unix. The issues 
 you have are related to running a rebol app as daemon in console 
 mode on a remote Unix server (without a UI desktop). Cheyenne can 
 work in source mode on such server, but it's much easier and pratical 
 to use it in binary form in such case (typical remote linux server 
 case).

Btw, the usual way to not have to "restart workers" on each app-init 
or other memory cached code is to run cheyenne with -w 0 option (use 
only 1 worker process and kill it after each request) and some verbose 
output for debugging :

 ./cheyenne -vv -w 0 &

Graham: I'll release a new SVN version in a few hours after fixing 
known bugs/issues and rewritten netutils and file attachement stuff 
to get rid of Rebol non-conforming legacy code.

Correcting myself about my claim regarding CRLF not sent by REBOL 
built-in protocols : REBOL does send CRLF even if 'newline if used 
as line terminator. A conversion is applied and it becomes CRLF once 
inserted in the port. This automatic conversion (LF=>CRLF) is OS 
independent and is triggered by /lines refinement used on OPEN native 
in root-protocol/open-proto. AFAICT, this behaviour is not documented 
explicitely anywhere. I can just find a few mentions about such conversion 
here : http://www.rebol.com/docs/core23/rebolcore-12.html, but it 
looks like it's related to FTP scheme only (while in fact being the 
default for all schemes relying on OPEN-PROTO).

darn, a single little thing goes unnoticed and this ends up causing 
so much grief!!  :-)

<sigh!>  I wish I'd known about /lines hehe... I had to go through 
countless hours trying to fix the headers and when doing multipart... 
god... it was horrible to fix.  ;-)

and the windows 7 issue?

If you're still having issues with the MTA, please active maximum 
log level : -vvvvv and send me the output on private channel or by 
email.

Ok, sent to myself .. and worked.

very odd... I just send email using rebol and there's no smtp traffic 
recorded!

and if not ?

The DNS server you're using are either under heavy load or just not 
very reliable (happens sometime). Anyway, my MTA should at least 
retry 5-10 times before giving up and reporting an error. I'll add 
that feature tomorrow.

SVN r23 :


FEAT: improved logging and error handling for dig/SMTP/MTA modules.

FEAT: integration of deferred retries in MTA for temp errors recovering 
and greylisting passing.

FEAT: new config keyword in global section: dns-server. (see changelog.txt)

FIX: in email generation, added a missing CRLF between headers and 
body.
FIX: logger sub-second times output padding fixed.
FIX: minor code cleanups.

actually I think line 174 should read

			switch strategy [

and lose the "pick dns-server"

SVN r24

FIX: more conforming handling of 4xx SMTP return codes
FIX: typo and coding error fixed in dig.r (Thanks Graham)

Thanks Graham - what should I do with the first run of Tortoise? 
I suppose I better don't do Check-out? Should I create a "working 
copy"? I don't remember how I did it for Rebgui, as I am long time 
issuing Update item from the context menu. I need to do first sync 
of Cheyenne now, and would not like to screw something on the server 
side :-)

I create Cheyenne local dir, and there is no SVN Update in the context 
menu. I first need to somehow create copy of the SVN. OK, there was 
Check-out in the context menu, hopefully Doc does not kill me that 
I have checked-out sources to myself, blocking others from submissions 
:-)

Btw, it doesn't accepting other mail clients than Cheyenne's CGI 
and RSP scripts.

It's for Cheyenne only usage. Sending email is often required in 
web applications, setting up and maintaining a third party MTA like 
sendmail is costly and complicated. Having a MTA built in Cheyenne 
will save everyone a lot of time and troubles.

Still need some work for debugging and fine-tuning as said by Graham.

I finally got gmail to accept mail from cheyenne.  I changed the 
from address in emai..rsp to one that matched the sending domain, 
and changed the greylisting delay retry period to 5 mins from 15 
mins.  It went thru after the 2nd attempt.

Ohloh is crawling in Google Code's and probably Sourceforge's repositories. 
Volker has a few projects in Sourceforge.

You have to sign up, confirm your email, and then confirm your email 
subscriptions before you can add cheyenne-server!

I'm need to get the name I assigned a face when I alt-click on it. 
I can get its style, offset, size etc but I do not see a way to get 
the face's name. More specificly I have multiple scroll-panels and 
I need to know the one I'm clicking in...

And this works ...


myname: "Graham Chiu"
myemail: [gchiu-:-compkarori-:-com]
toemail: [dontspamme-:-gmail-:-com]
subject: "testing Cheyenne"
msg: "Yes, this i a message"
file: "encap-paths.r"
path-to-file: %/c/chesvn/Cheyenne/encap-paths.r



result: read/custom http://localhost:7900/email.rsp reduce compose/deep 
[ 'POST
	rejoin [
	"from=" (myname)
	"&from=" (myemail)
	"&to=" (toemail)
	"&subject=" (subject)
	"&msg=" (msg)
	"&file=" (file)
	"&file=" (path-to-file)
	]
]

because I use duplicate cgi names .. ie. from and file, they are 
turned into a block when processed

And this means order is important

in email.rsp we have the following ..

	;-- check attach file and format it for send-email
	ten-mega: 10 * (2 ** 20)
	if block? file: param/file [
		file/1: to-file file/1
		

  either file? file/2 [	; check case where uploaded file has been streamed 
  on disk
			if any [not exists? file/2 ten-mega < size? file/2][
				print "Attached file not found or size above 10MB limit!"
				response/end
			]
			file/2: read/binary file/2
		]
		param/file: reduce [param/file]
	]

the either file? is missing the second conditional block .. I presume 
the first block checks to see if file/2 is of file type and the other 
type is binary

Also in chrome when you select a file, it just shows the filename 
... I wonder if and where it is storing the full path.

How can I stop the browser from caching the response/buffer ?  I 
set the response butter and content type, but the browser seems to 
be caching the buffer.

SVN r30 :


FEAT: MTA engine refactored, jobs are split in sub-tasks allowing 
more reliable and cleaner code.
FEAT: Default SMTP delay reduced to 5mn.

FEAT: MTA queue can now be saved on disk when server is restarted 
(persist [mail-queue])

FEAT: added native DNS async wrapper for UniServe (protocols/DNS.r)

FEAT: replaced blocking DNS calls for MX domain resolution by async 
DNS calls.

FEAT: 'on-mx event in DIG protocol now returns a block of all the 
declared MX.

FEAT: now all MX will be tried in right order by the MTA if connexion 
fails.
FEAT: improved SMTP logs by prefixing with a SMTP session ID.

FEAT: upload data in HTTPd limited to 2GB (max supported by integer! 
type)

FEAT: 'email-info? RSP function return data format improved (see 
email.rsp header)

FIX: email.rsp minor bugs fixed and From: field added.
FIX: minor fixes in email.r library.

FIX: now uni-engine's logging level is also set by 'set-verbose function.

What I'm getting however is this


request/content is [file "%/C/Cheyenne/incoming/oixaqcdg.tmp^M^/"]

and the file contains all the post variables and data

I assumed it only put file data there and still captured the non 
file data into request/content

Yes. It's probably to cut down on interprocess communication between 
the host and worker processes. The RSP is processed in a different 
process than the web server process.

I assumed it only put file data there and still captured the non 
file data into request/content

 => it requires to send data using "multipart/form-data" encoding.

Jarnal is an open source Java app that can ink on PDFs and save them 
back to the server by posting the PDF with the annotations to a cgi 
script.

It can also save the first page of the PDF as a Jpg, and interestingly 
the PDFs are about 12kb in size but the Jpegs are about 150kbs

Adding support for that shouldn't be difficult, but how to efficiently 
and securely manage the associations between user/pass and access 
rights? I'm not a big fan of .htaccess files scattered everywhere, 
and I'm not sure that cluttering the httpd.cfg file is much better. 
Maybe a separated centralized config file for user accesses? (I need 
to dig more on this topic before starting any implementation)

Dunno .. but at present I am putting files into public to allow 3rd 
party apps to access them ... and using very long filenames to obscure 
them

I'm about to build a user login system, but I would like to know 
if it's a good idea to make this a public project so others can follow 
and contribute? Would it be a good idea to include it in Cheyenne? 
I don't intend to use MySQL for it as the rest of my site isn't DB 
driven.


I've not done anything yet, so I also need to know, what Cheyenne 
can already do in this area, to avoid duplicating functionality. 
Can I completely rely on the session context, or are there crucial 
parts missing?

Well it depends on what you mean precisely with "user login system". 
Sessions associate a context (in the general sense) holding data 
with a remote web browser session. The AUTH config keyword controls 
whether access to webapps resources requires a "login" process or 
not. The login state is reflected in a built-in special word in RSP's 
sessions (session/content/login?). LOGIN? is also settable, so you 
can avoid using AUTH and build your own login system (you'll have 
to overload the 'on-page-start handler to restrict accesses).

Doc, thanks.

What I need to do is basically the whole package:

Sessions, which Cheyenne can do.
Login process with authentication. AUTH config, perhaps.
Log out process, also possible.
User adding and removal process. This needs to be added.
Privilege usage and management. Also added separately.

I'll have a closer look at the built-in AUTH config keyword.

It's a uniserve service with an rsp interfaces, works pretty well, 
if something takes more than 1 millisecond to execute, better cache 
and reuse 8)
interface is simple:

.cache 'pool-name 'variable-name 'valid-time-in-seconds [code to 
execute]
one refinement, /set , will force update cache

How is a session managed? When is it created and when is it destoryed?

It seems that I somehow get back an old session/id from someone else 
even I just opened my browser and loaded a page.


This problem shows in my REST shopping cart in that I get a shopping 
cart from someone else.

When is session created?
 On first request to a webapp resource.

When is session destroyed?

 When you invoke session/end or once the session timeout period has 
 expired and the garbage collector has suppressed it. Not sure what 
 happens if you try to access it after the timeout before GC passing, 
 need to check it.

Ok, trying to access an expired session still in memory, will force 
it to be destroyed and a new one will be created instead (with a 
new ID).

But it seems that you're not using webapps, just plain RSP and manual 
session handling. You should know that invoking session/start container 
will create a new session context but you won't get a new ID at once 
(session/id will be set to none until the RSP page ends). We already 
discussed this point previously and I have an entry in my todo list 
to improve that. (it requires a change in the session ID creation 
process)

I think I can just include the S3 libraries in the Cheyenne binary, 
and do a S3 download as a rsp form where I submit the filename, bucketname, 
and access keys

just thought I'd drop a little note that serious remark module work 
has begun.  I was trying some stuff before, but starting too wide 
and I wasn't able to get traction on the project.  now I'm just integrating 
the v2 remark parser into a mod.  


One cool thing that it will do out of the box, is handle statically 
parsed files. basically, you build .html files using remark dynamic 
tags, they are saved out in a cache dir and then the url-file function 
will redirect to the parsed file, if done, or will run the parser 
on it and then cache it.