AltME groups: search

I was thinking about making a low-level dialect allowing to simulate 
user actions in a web browser. Then use that dialect to build a higher 
level dialect (specific for each webapp) allowing to navigate inside 
a webapp, make complex actions and check the result, using a minimal 
set of keywords. I haven't wrote down, nor refined those ideas, but 
it should be able to give you (at the higher level) something like 
:  
[
	go site-assistant
	signup 
		login: "[user-:-test-:-com]"
		pass: "pass"
	check-page default-page
	add-new-website
		website: "rebol.com"
		registrar: "registrar"		
	check-result site-created "rebol.com"
	check-details website-form "rebol.com" "registrar"
	... 
]

In this example, test data is hardcoded, but the dialect should be 
able to read data from scenarii blocks allowing to loop test a sequence 
of features using several data sets.

I'd like to have a test dialect being it's own meta-dialect, so that 
you can build higher level dialect one on top of the other, to be 
able to test you webapp at *any* level (from basic HTTP request, 
to full simulation of a working day).

yes, dialect based solution would be very interesting ... I was thinking 
about dialects a little too (not concrete yet) ... currently my testing 
engine is made so that it uses a proxy and it records your what you 
do via browser and then it can repeat the same and comparte the output 
(it already figures out that it needs to set different cookie and 
some basic stuff to login for example , and I have idea to make it 
scriptable for other dynamic data but I haven't come that far yet 
) . It also doesn't do any smart comparisson of the outputs, but 
more of a report for human to view for now

The recording proxy can be a very efficient approach for non-regression 
testing.

A bit OT but Lad and I started a test-dialect for RebGUI. You could 
send CLICK to GUI objects etc. and compare results and state, make 
screenshots of something was wrong etc.

I think I will pic-up the project in a couple of weeks again and 
push a bit forward and than it get useable.

Yes, I think we could come up with a compatible reporting format.

Here is a snippet of a test script:

new_company: [
	dt: "Eingabe/Firmeninformationen"

	with company [
		; reset data form
		press-right a

		check note = ""

		name: 		"Test Suite Company 1"
		address:	"Teststra§e. 10"

		; should bring up an ALERT
		check-window [press a][press ok]

		; country:	"Deutschland"
		; press a

		; select sector
		check-window [press req-sector][
			sector-table: ["27"]
		]
	]
]

company_tbl_test: [

	company_tbl: [61 "GFA-Motoren"]

	check company_tbl = [61 "GFA-Motoren"]
]


First line sets a line in a table, second line checks if the selected 
value matches.

Robert, maybe it should be the right time to switch to a "Testing" 
group (surprinsingly, there's no such group yet), we're going too 
OT.

Max, that sounds too hard, a ringing bell would be more accurate. 
;-)

I was doing a search on cheyenne the other day and came accross this. 
 Is she a relative of yours doc?   ;-)

yess.... a glorious cheyenne built 404 page not found in my browser 
using firefox and my new VPS with shiny 6 letter domain name  :-)

now the fun part... building a new web site from scratch, using brand 
new technology  ;-)

Cheyenne's 404 : sounds like a victory scream. :-)

doc, trying to understand the mod  'WORDS  dialect...   I am reading 
its parser, and I still don't get what the 'IN token does.  or what 
options I can use, or even why its needed.

can you give me a few pointers on how and why it should be used?

Q2:  within the mod, requests have a locals value of none, does any 
of the uniserve/cheyenne internals touch this or can I use this at 
will within any module, creating an object and appending/changing 
 attributes as needed?

Q3:  Are there any time-based call backs we can implement through 
mods?  sort of like a rate on a view face.

this would be very usefull:

-it could allow me to keep statistics on mod internals at regular 
intervals ex:  average load, high/low peaks, 

-perform cash cleanup/buildup, ping remote tools for "I am alive" 
monitors, etc.

I hope there is a simple trick I don't see at the moment:

As a result of an RSP script I want to return a new web-page that's 
on the file-system. I do it like this:

print read %payment/index.html


So far this works. What I need to do is, to insert some dynamic content 
into the read HTML file.

For this I see several options:

1. Somehow load this file as an RSP file and have it processed.

2. Somehow load this file as a SHTML file and have it processed as 
SSI file.

3. Add some marks and inser the stuff on the fly before delivering 
it.

Q1 - WORDS dialect : this dialect allow defining new config keywords 
that can be used in the %httpd.cfg file for your mod. IN defines 
the config file sections where it can apply. Possible values are 
:
- globals : global config block used for server-wide options
- main : applies to a domain or webapp context
- location / folder : reserved for future use.

Q3 : Not yet. It's a planned feature that I need also to add a simple 
CRON-like task scheduler inside Cheyenne. Feel free to add your own 
in your mod, I don't that I'll have time to work on it before middle 
summer (low priority task).

I'd like to not do everything by myself, but it's not that easy. 
I have some deep concerns for Cheyenne core part such as speed, memory 
usage, stability and security. Cheyenne has become a *critical* part 
of our business, I have to garantee that a new version won't break 
our webapps in production, nor make them instable, insecure or noticeably 
slower. My responsibility also extends to other companies that are 
selling products or services based on Cheyenne.


I've already accepted small patches on Cheyenne core in the past, 
but it takes me a lot of time to study and test each line of code 
an rewrite them if required. If your code has only a local impact, 
I might use it, if it needs to patch a lot of parts of Cheyenne/UniServe, 
I probably won't. Anyway, you can send it to me, it's always a good 
inspiration to see how other developers solved some specific problem.

I can't disclose much right now, one of our Cheyenne based product 
is getting a lot of attention recently and in a couple of weeks, 
we will know if we need to hire a few more peoples at Softinnov. 
;-)

I thought he worked for a company .. didn't realise Rebol was important 
in that way.

A few small intranet apps in a couple of TOP 5 french company are 
using Cheyenne and RSP.

I did release my Hylafax web portal which I know a few people are 
using :)

And we have also a big product using Cheyenne/RSP with already a 
few dozens customers.

I prefer to keep it "under the radar" for a few more weeks. ;-)

I did build a web portal to my medical database .. but too busy to 
keep that going.  Have to learn a lot more jQuery ....

Still, I have a few users of that as well.

I see Cheyenne as a web-window to REBOL apps.

the inability to run more than one Cheyenne server at the same time 
has been a problem too.

I don't see it anymore because now I use a Rebol client to access 
Cheyenne and not a web browser.

UniServe is a thin framework layer other the raw port! stuff. It 
provides an event-oriented framework for implementing server or client 
side protocols. Some UniServe events are same as the lower async 
ones : on-connect, on-close. Other are higher level such as : on-receive 
(trigger when a given amount of data or a given sequence is received).

Did I mention I'd like to see a zope clone one day :)

response/Forward: This looks good. from the docs I see that it's 
possible to forward to a new RSP page. Will this work with a SHTML 
page as well?

graham, you can run MANY cheyenne servers on the same system .  and 
they can be handling several thousand requests / hour each without 
failure.  


at my client cheyenne is probably the most stable server application 
they have, a part from apache.

UniServe still has a purpose in R3, but it implementation will be 
much lighter and it will run much faster. Btw, one of UniServe's 
plugin, Task-master, is in charge of running and exchanging data 
with external processes given true multitasking abilities to UniServe's 
based products (RSP scripts are evaluated in such helper processes). 
R3 multithreading will make multitasking much simpler and way much 
faster.

the mod for access refusal is finished btw.  it works really well, 
I ended up doing it in a mod and doing a few invisible actions within 
the make-response and task-done callbacks.

Graham: qwikitodo has close to 100 todos so it's "finished" although 
it will keep evolving, but it's a small project

that said, if R3 provides the capability of producing a really good 
webserver in 5 kb, I might use that instead.

henrik: "that said, if R3 provides the capability of producing a 
really good webserver in 5 kb, I might use that instead."
what do you mean?


a webserver is not just about tcp/ip.... its all the framework it 
provides.  supporting the full range of reply errors, plugins, proper 
headers, etc , etc.  you can't really make that kind of thing in 
5kb.

Well, maybe you can't. I haven't given any thought to what it takes. 
I was only thinking of the basics like large file transfer and proper 
working async ports and threading. some basics that a good webserver 
can do.

right now, I can tell you that cheyenne, from the client's point 
of view, does all of that.  R3 will just allow it to be a bit faster, 
probably a bit more robust at the seams, and definitely easier to 
support, since some of the workarounds will now be implemented directly, 
and whatever is missing, doc can add/fix directly in binary.

apache coders would already be jealous at how easy it really is to 
build a mod right now... even the configs can be specified within 
the mod with a few words.

Having the TCP/IP part open-sourced in R3 will be great. It will 
allow to use much faster OS hooks for file transfers, extend the 
port! API to bind only on selected interfaces, etc...I wonder if 
the main event loop will be there also, so we can replace the not-scalable 
Select( ) call by other faster ones or even integrate libevent. That 
would definitely make Cheyenne able to handle a much higher number 
of connections.

for my part, when R3 goes open, I'll be integrating liquid-paint 
with OpenGL asap... which will completely alleviate my need for /view. 
 we will have a 3D native GUI  :-)

Not just the TCP code will be open in R3 - the HTTP port code will 
also be open. One of the goals is to make something like UniServe 
completely unnecessary for R3. This is not a criticism of UniServe, 
but of R2. If R2's networking infrastructure were good enough we 
wouldn't need UniServe.


Hopefully by the time that Cheyenne is ready to be rewritten for 
R3 it will be able to be written right on R3's HTTP ports.

libevent was suggested in the past, along with links to liboop etc. 
Not sure the licence is OK. Anyway - I wonder where do we go such 
a way? I can already imagine complete mess and tens of versions of 
custom R3s, if such low level things as main event loop are open-sourced 
and replaceable.

Doc, the intention is for the R3 HTTP scheme to also support http 
server use. However, the current HTTP scheme is just a placeholder 
until someone can update or rewrite it. Client use is jst what (barely) 
works in the placeholder.

Perhaps someone who has already written a kick-ass web server... 
:)

That would require me to get a round tuit though.

for example, i know of a server which profiled the tcp stack of their 
server and realised that some buffer sizes didn't get cached the 
same way through windows.

just changing the size of buffers and tcp payloads added a big speed 
boost.  stupid detail, but now if we have such cases, we can actually 
really go as deep as that.

Just played a little with DOS console parameters to try to make it 
look&feel like R2 console. Quite close so far (except for the font). 
Need to test it in action with R3 to see if I can use it for serious 
work. I'm very picky about the tools I use daily.

Yeah, the intention is that a GUI console will be written in REBOL, 
part of the community-created, open-source portion. Then you can 
use or adapt the console for your own apps as well if you like. How 
about having RConsole being implemented with that? :)


Right now the GUI doesn't have good-enough Unicode support to make 
the console yet, so the GUI console will have to wait for the release 
of the host code (the current priority), and the subsequent resumption 
of the GUI work.

How exactly does Cheyenne cache file loads? I have trouble getting 
a specific REBOL script to load.

I think I have found a bug where Cheyenne keeps serving an empty 
file, if I have first had it put in a server directory as a MacOSX 
shortcut and then replaced it with a real file of the same name.

is there a way to prevent caching and logging?

If you want mod-remark to take other mod-static caching, just provide 
a 'make-response handler in mod-remark, give it a higher priority 
and make sure to return a TRUE value (that will end the phase shadowing 
mod-static's handler for that phase). You can also just unload mod-static 
by commenting it inside GLOBALS section in config file (you'll have 
then to provide all the adequate handlers for serving static resources).

Looks like you've gone really deep in UniServe. Sending job to helper 
processes has a cost (building message, MOLDing, transmitting through 
TCP, LOADing, decoding message). With R3 and multithreading, this 
cost will reduce to zero, thanks to shared memory (or equivalent 
system).

yep threads will be a big plus in R3... many apps are much easier 
to write with threads (of any kind).  it breaks up the domain of 
many problems into smaller pieces.

I use code like this:

<input type="radio" name="paymethod" value="paypal" />
<input type="radio" name="paymethod" value="somethingelse" />

And this code is wrapped in a DIV.

It looks like the JS lib toQueryString creates two differnt versions... 
I'm using mooTools. Very strange. Or is there a cause for this?

Is this a cheyenne issue?

toQueryString: This was a false alarm, it's working correct.

No, I load the answer HTML page and insert some HTML on the fly before 
returning the page via a PRINT.

It looks like you've added a reformed query-string to a URL already 
having a query-string. Are you using request/query-string?

how do I use rconsole to tell cheyenne to reload a mod that has changed 
on disk... 


or does cheyenne detect mod-file changes and reload them automatically?

You can't. Mods are part of Cheyenne's kernel and the priority of 
each mod's callback is determined relatively to the other mods during 
Cheyenne boot (kind of competition for higher priority for each phase). 
Reloading a mod would required reloading all the mods, breaking most 
of the active connections (CGI, RSP, FastCGI,...). So the answer 
is : kill and reload Cheyenne.

with your permission I would like to rebrand ssh-admin to  Cheyenne-admin.

its like a stand-alone cpanel for cheyenne-based systems.


All the current file and system commands options to manage the host 
server too will be complimented with any tools needed to configure 
and control cheyenne and any mods which are installed.

I make a hash! list of all the phases' callbacks, so the impact on 
performances remains very low.

Go on for the name, my own Cheyenne admin panel is still at a very 
early stage.

I am using this site as a showcase for all of my work.  including 
web, visual arts, music, design, programming, etc.

cheyenne-admin also has a *VERY* nice GUI, using GLayout.

Note this is a linux server-side only tool (currently).
currently:
	-remote browing of files in a gui.
	-uploading/downloading any files to/from the webserver
	-running command-lines remotely
soon: 
	-chmoding remote files
	-handling webserver start/stop/restart  remotely.
	-more as time goes by.

ie. you can't run it from a client PC?

I meant the server has to be a linux box.

anyone know if there is a detailed doc on sessions within cheyenne?

what we need with !cheyenne are minial examples of services and mods 
which implement all callbacks and give just a small comment on why 
and when it is called... this would help soo much in understanding 
how to implement cheyenne extensions.


I am currently looking at the rsp code and its so huge and complex 
that its a big daunting to grasp the whole by looking at its parts.

Mod-RSP is the most complex one and it does a lot of things. Sessions 
in Cheyenne are RSP-specific. There's a synchronization system in 
RSP sessions for protection of session data from corruption in concurrent 
executions. That may be easy to understand by just reading the code. 
Anyway, feel free to ask for explanations of specific code parts.

It works using a semaphore (session/busy?: yes|no). It used to be 
very fined-grained with distinction of RSP scripts just reading session 
data from scripts modifying session data, but it was even more complex 
to maintain and the RSP source static analysis I was doing could 
never be 100% accurate (it''s easy to bypass any form of static analysis 
in R2). So now, it just blocks concurrency inside a user session 
(only 1 RSP script per user session allowed).

Yes it can, from the server perspective, it's just a sequence of 
RSP requests.

If you can think of a solid method for allowing concurrency in the 
same user session, while preserving session data from corruption, 
I'll be very interested.

a part from having handler tcp interaction, I can't think of any 
a part from identifying a way to detect which session data is being 
used by both scripts.

It's always possible to write session data to disk or to a DB and 
rely on the DB locking system to ensure that data is safe, but there's 
a performance overhead in that case that I'm afraid, may not result 
in any speed gain for the user. It also requires to have a DB engine 
installed on all servers running Cheyenne using RSP sessions, which 
might be overkill in some situations.

this being said... mod-remark will be using persistent liquid nodes 
for session and post/get parameter side of things, and its possible 
that I might store session data outside of the server itself, using 
a liquid-tcp node. 


so that access to actuall session data could be brokered outside 
the server's jurisdiction, extremely efficiently, locked one value 
at a time. and any access/change could be perpetuated to other handlers 
when they attempt to lock to one session value.

the idea being that interacting via xml request or full page refresh, 
with only part of a page parameters, creates the same output  :-)

you could actually have one page sending xml requests to the server 
in ajax, and another page refreshes with the results of those requests...


the second page could also send a page request with the some parameters 
sent, and the server will reflect all changes to the current session/page 
so far.

I thought also about doing a TCP based session server (with session 
variable granularity), but the overhead in synchronization would 
hit performances (and scalability) too much IMO. If each time you're 
setting a new value to a session variable in a script, you have to 
query a TCP server (even a local one), that would have a huge impact 
on a script execution time.

might still be much faster than a db call.  but the real game-changing 
aspect of mod-remark is that a document in ram, will only render 
parts of itself which are changed, or which cannot be cached, so, 
unless you've actually changed a parameter, it knows that it hasn't 
changed and will not need to ask for the broker to give it back the 
value.  in fact, it wont even process that part of the page, only 
using the data in its local node cache directly.

header/ footer / current login state box/ static page content... 
this can be shared accross sessions, across the entire site, or as 
granular as a single page request.

a news reader could have all the news items cached already and all 
its doing is assembling those which are matched via the request... 
really only doing a fast rejoin of many items, and then a rejoin 
of header news and footer and then spit it out.

That's quite close to how RSP are working. Each RSP script is compiled 
to optimized REBOL code and cached in memory. A RSP request will 
then result in evaluating the REBOL code which mostly just append 
static and dynamic parts in the output buffer.

the current issue is trying to share the cached data between different 
handlers... which pits me against a liquid value broker, but its 
the creation of the data on the broker which might become complex... 
so not yet shure how I will proceed..

Max, applying dataflows principles to a web framework looks really 
appealing, when will be able to see some demos?

I am aiming for sometime next week, just a few simple prototypes 
to explore how to handle the difference in  nature of web requests 
opposed to persistent  data.