AltME groups: search

I haven't finished it mainly because I'm not a LDAP user, so I'm 
not familiar with most of the LDAP concepts. The protocol itself 
is quite complex due to ASN.1 encoding.

though I'm sure its possible to rewire how Altme message passing 
works.  as long as the server has the data and client/server are 
using the same message protocol, everything can be changed.


it can even be retro-fitted on update, to prevent people from having 
to download the whole world again, since we already have the data 
on our disks.

the message protocol from the R3 dev chat.

Screen control sounds good - not too big or wooly - I'll take a look. 
SQL would be directly relevant to my project, so I'll definitely 
dig that one out. Do you mean SQL-PROTOCOL or is there something 
more recent?

Is there a protocol version of RedDB ? In order to use it under Cheyenne 
with database enhancement.

No, but it wouldn't be that hard to write a protocol wrapper replacement 
for the SQL function.

It seems to be in the HTTP protocol.

Gab. has mentioned an updated imap protocol that might be donated 
... dunno if that is still on or not.

OK, now that we have 2.7.7 released (even though there is more work 
to do, i.e. platforms and the SDK), it is time to look ahead to 2.7.8 
- which is scheduled for release in one month on February 1. The 
primary goal of this release is to migrate to REBOL's new development 
infrastructure. This means:

- Migrating the RAMBO database to a new CureCode project and retiring 
RAMBO.

- Using Carl's generation code for the manual to regenerate the R2 
manual, so we can start to get to work updating it.

- Porting the chat client to R2 using the new functions and building 
a CHAT function into R2 similar to the R3 version.


The R2 chat client might be limited to the ASCII character set, though 
support for the Latin-1 character set might be possible. Still text 
mode for now, though if anyone wants to write a GUI client (Henrik?) 
we can put it on the official RT reb site accessible from the View 
desktop. The server is accessed through a simple RPC protocol and 
is designed to be easily scriptable.


It turns out that Carl already rewrote the installer for 2.7.something, 
but it was turned off because of a couple minor bugs that we were 
able to fix in 2.7.7. With any luck, only minor fixes to the registry 
usage will be needed and we'll be good to go.


As for the rest, it's up to you. Graham seems to have a good tweak 
to the http protocol, and others may want to contribute their fixes.

Pity we can't include Romano's async protocol too .. and then we 
could start moving some of the R3 schemes back to R2

OK, text protocol over TCP or SSL. That sounds doable.

like this?
the cli connector is using the cli component nearly all major
databases delivering. the connection is made via rebols 

call/wait/info/output/error and a simple parse after, for the resultset.
I am using this prototype mainly for a q & d connect

to mysql/postgresql/monetdb/sqlite. on my list are also connectors 
for

firebird/oracle/greenplum/sybase/ingres/infobright/frontbase and 
cassandra.
pros:

1. very fast for single requests
2. no rewrite of code needed if a new version or protocol is out
3. easy 'data migration' between the db's

4. adding new db's are a matter of hours only (see the cli spec thats 
all)
5. fast prototyping and testing for new db's

6. robust, never had any trouble with cli's even with bigger resultsets

7. should be perfect also for traditional cgi (the process starting 
overhead is minimal, execpt you name is facebook)

8. very small footprint (~120 lines for connecting to 4 db's, could 
be the half)

with a nice tcp-server component like rebservice the 
cli multi connector could be very usefull as a c/s connector.
I made a test with 2.000 concurrent calls (simple select) 
on a 4 gig quadcore. the cpu was only close to 50%, a good value.

cons:


1. slow if you have very much serial inserts (unless you shape them 
into one sql query)
2. need to start a cli process for every request
3. needs a tcp server for non-local connections
4. some more, but who cares ;-)

with a solution to keep the cli open from rebservice,

these cons could disappear and the speed diff overhead to a memory 
based lib
could be marginal.

No, I mean the server-side services, official protocol specs, etc.

It might be a good idea to have the standard pop:// protocol do the 
STLS negotiation, and have pops:// do POP3S like Gmail.

Doesn't seem to parse the cookies, just passes them through. Probably 
good enough for a low-level protocol.

Brian ... has anyone looked at the network protocol enhancements 
...

And have you actually reviewed the htt protocol changes I made yet??

Can anyone think of any side effects from adding a 'dehex to the 
protocol rules ?

The standard is very explicit on how encoding should work. Reserver 
characters must never be decoded BEFORE you split the URL into its 
parts (which can only be done if you understand the scheme). Any 
other character may be decoded just like REBOL does.


So, it's not that URL! should never decode, it's that it should never 
decode the reserved characters. Then, the handler for each protocol 
needs to decode the parts separately after splitting. Since the protocols 
we use in REBOL all have the same format basically, you can use a 
single function to handle all of them, eg. like my parse-url.

This is one reason why I wrote a rest protocol.  The http protocol 
seems designed to get content the same way a browser would.  But 
as more services use http more completely, things like automatic 
redirects and thrown errors for 4xx/5xx status codes are not helpful 
(and good luck getting headers and content then).

Thanks for arranging this award for another year, Brian and Max.


I've voted for who I think will be a worthy winner -- and I hope 
everyone else takes the time to do so too.


For those who find AltME scrolling a pain, the voting protocol can 
be browsed here:
   http://www.rebol.org/aga-display-posts.r?post=r3wp558x302

It should be something like this:

mysql://user:[pass-:-localhost]/bugs


I also tried it in a different console and it only appears if the 
mysql-protocol is not loaded.

There is no reference to mysql-protocol.r anywhere in the source 
code. Could that be it?

In 0.9.8 there is a reference to mysql-protocol.r in app-init.r, 
but it's gone in 0.9.12.

Ah probably, now CureCode expects that the %mysql-protocol.r be loaded 
externally. The best way to do that is to instruct Cheyenne to load 
it directly by declaring it in the GLOBALS section of the config 
file, like this:

	worker-libs [
		%<path-to-mysql-protocol-folder>/mysql-protocol.r
	]

I have upgraded the install script in v0.9.12 archive to handle the 
mysql-protocol.r installation.

Petr, Peter doesn't talk nonsense IMO. There are a lot of things 
REBOL could use for interop. SOAP would be a protocol, not an extension. 
REBOL IPC may be a protocol too. MQ et al would be for specific systems, 
not general IPC. I'm all for a Uniserve model.


Historically, memory mapped files were one of the most efficient 
IPC mechanisms on Windows, but if a more portable approach can be 
written based on named pipes, I can live with that. Sockets are great 
as long as you don't run into firewall issues (even locally), and 
might encourage us to think in terms of protocols and messages.


As Pavel said, and one of the main design points, is the need for 
a daemon. Obviously memory mapped files aren't going to work across 
the net.

The existing IMAP protocol for R2 should have the parsers written 
already

As I said above I don't use imap, and am not familiar with the protocol. 
 All this does is login using auth=login and grabs the first mail 
in the Inbox
http://rebol.wik.is/Rebol3/Schemes/Imap4

ssl AFAIK is under the protocol, so even with an SSL module, the 
schema should not really change.

The r2 imap protocol is pretty basic ... so I don't see that we need 
do more than that ....except perhaps allow users to pass custom commands 
to the server.

re imap: i don't think that much of anything is necessary for imap 
atm. once i'm satisfied with the pop protocol, adapting it to imap 
will be quick and easy

I've got the imap protocol to the point where
1. can authenticate using cram-md5 ( reused r2 code )

2. can select different mailboxes ( unlike r2 where you have to select 
at time of opening the port )
3. length? 
4. Pick messages

However, I don't like this timeout which is there ..

Signed up, sent myself a couple of emails to that account, and then 
managed to login and download using the imap protocol

I started with daytime, moved to smtp, and then ftp, fax and now 
imap.  I am using pretty much the same template with changes appropriate 
to the protocol.

Ok.


My question above is whether there's a standard "core" that can be 
used across many protocols... a bit like net-utils or default protocol 
on R2.

I was thinking of using net-log as a way to hook into the low level 
activity of the protocol so that I can patch it as needed when interacting 
with a GUI ... eg, for progress meters

Once the protocol is installed then you just use OPEN, READ and WRITE, 
etc. directly. You only need exported helper functions for some protocols, 
but http is covered by the default actions.

Nope, not even then. Most protocol functions don't have to be accessed 
from the outside except by the port infrastructure. Functions in 
the scheme are only called by port infrastructure, and most functions 
are helper functions. The only words you export would be user-visible 
functions that users are supposed to call directly.

We can replace a protocol in memory if need be. Or load one from 
host code. Or better yet, fix the source in DevBase.

I'd like to ask how we might manage the download of a large file 
by thehttp protocol

The http protocol lacks any net-log or debugging so could write one 
and insert it into the awake handler ...

Thanks, Henrik, anyway. I'll take Gabriele's answer for a "yes, it's 
possible" because he has written the scheme, so he would know for 
sure ;-) I'll study his protocol and the underlying TCP stuff deeper 
after the weekend.

Graham, all changes to the prot-http sources seems to be authored 
by Carl, I've seen no traces of changes to the protocol introduced 
by Brian.

I haven't posted changes to the http protocol yet, sorry.

I've setup a server at www.compkarori.co.nz:8020 to help debug the 
jdbcbridge protocol.

Ok, updated the protocol to remove the waits inside the handler ... 
and it is working now.  I can now detect the close event from the 
server.

like for any protocol, the server must send the length of the data 
at first

Any simple protocol example available, something like the R2 one 
long time on rebolforces, but for R3? I'd like to implement protocol 
above 'file schema. Any help/hint welcome.

make-scheme [
	name: 'foo
	title: "FOO Protocol"
	spec: make system/standard/port-spec-head []
	actor: [
		open: func [port [port!]] [
			print 'open
make-scheme [
	name: 'foo
	title: "FOO Protocol"
	spec: make system/standard/port-spec-head []
	actor: [
		open: func [port [port!]] [
			print 'open
		]
		read: func [port [port!]] [
			print 'read
		]
	]
]

f: open foo://bar.baz
read f

Rebol [
    file: %prot-rif.r
    title: "RIF protocol"
    author: "Pavel"
    rights: 'PD
    date: 13-Aug-2010
    ]

;; Local functions

Append-RIF: func [port [port!] record [binary!] ][


 write/append port/locals/2 to-binary length? head port/locals/1		;index 
 port the end of data file will be beginning of new record

 write/append port/locals/1 record							        ;data port new record 
 into data file


 return shift length? head port/locals/2 -3					        ;number of 
 records 8 bytes per record
]


Get-RIF: func [ port [port!] i [integer!] /local numentry indexpos 
recpos recend value ][


    numentry: shift length? head port/locals/2 -3                    
       ;number of records 8 bytes per record


 if any [i = 0 i > numentry] [return none]                        
    ;numbering starts at 1 ends at file end


 indexpos: multiply subtract i 1 8					                ;compute index 
 offset
	recpos: to-integer read/seek/part port/locals/2 indexpos 8

  either ( (8 * i) = length? head port/locals/2 ) [				;last record 
  special case
		recend: length? head port/locals/1
		][

  recend: to-integer read/seek/part port/locals/2 add indexpos 8 8		;internal 
  record
		]

 return read/seek/part port/locals/1 recpos subtract recend recpos
]

;; Scheme definition
make-scheme [
	name: 'rif
	title: "RIF Protocol"
	spec: make system/standard/port-spec-head []
    awake: none

	actor: [
		open: func [port [port!] /local path ] [
            parse port/spec/ref [thru #":" 0 2 #"/" path:]
            append port/spec compose [path: (to-file path)]
            port/locals: copy []
            either (0 = length? port/locals) [

                append port/locals open/seek rejoin [port/spec/path ".dat"]

                append port/locals open/seek rejoin [port/spec/path ".idx"]
                ][

                port/locals/1 open/seek rejoin [port/spec/path ".dat"]

                port/locals/2 open/seek rejoin [port/spec/path ".idx"]
            ]
        return port
        ]

        close: func [port [port!]] [
            foreach  port port/locals [close port]
        ]

        read: func [port [port!] /seek number [integer!] ] [
            Get-RIF port number
        ]

        write: func [port [port!] record [binary!]] [
            Append-RIF port record
        ]

    ]
]

Graham: regarding the 'updating GUI from network protocol' question

I have found some older scripts I did and quickly added the progressbar 
to it so you should see how it works.
You can download it from here:
http://www.rebol.cz/cyphre/scripts/r3/net/client.r3
and
http://www.rebol.cz/cyphre/scripts/r3/net/server.r3


just run server and client on localhost and press enter in the client 
console to see how the server shows the progress of upload.

a little strange to use ftp to do that, but it's the only protocol 
allowed here :-)

I've created a git repository and stored all the protocol stuff there 
... http://github.com/gchiu/Rebol3

I don't miss a simplification of DO/TRY as much as structures for 
sequential tests that may or may not fail. REBOL doesn't have anything 
here, so you have to roll your own. Say you're trying to connect 
somewhere in a protocol and one of 50 things can go wrong, so you 
have to state 50 tests, 50 error messages, 50 exit routes. That's 
a lot of lines of almost identical code.

j: open jdbc://localhost:8000
insert j {select first 2 * from staff}
r: copy j
>> length? r
== 2

My jdbc protocol working

My jdbc protocol is async

My jdbc protocol is a R3 protocol ... this is the R3 group!

Graham: the seemingly self-recursive "close" call works, because 
of how the actor functions are bound. Compare:

>> foo: func [] [42]
>> bar: compose [foo: (func [] [foo])]
== [foo: make function! [[][foo]]]

>> bar/foo
== 42


The situation with protocol actors is similar. (With the binding/reduction 
stuff is "hidden" in make-scheme.)

well, there's only the one protocol included so far .. if you want 
your own, you have the option of including, or loading afterwards

Just reading new roadmap - http://www.rebol.com/roadmap.html... 
what I don't understand is all the fuss about R3+. From the very 
beginning, I don't like the idea of putting some usefull stuff into 
additional module. E.g. - why some usefull mezzanines or protocols 
be part of the plus package? What I fear is - anything, that is optional, 
will not be a standard. I can understand that we can't include 100 
protocols probably, but talking about + package, where the only protocol 
we have is http, and even that is not fully functional, is a bit 
preliminary :-)


Interesting note about R3 DB - what is R3 DB port? Is Rebin, RIF, 
finally coming? Or did we decide to select one DB, e.g. SQLite engine, 
and include it into R3?


As for tasking - "Experiment: how far does the current R3 multitasking 
base take us?" - no experiments, please :-) Tasking/IPC is the last 
missing stone before we can claim R3 being a beta - it needs solid 
work, and I expect 1-2 months, and the same kind of impact, as Unicode 
transition, or Extension, Callbacks had :-)

It isn't a protocol yet but rather has a module/function API. ODBC 
database access is working for me as a Windows only host kit extension, 
with UTF8 strings and bound parameters. I'm in code cleaning stage 
and I'm about to support SQLTables and SQLColumns catalogue API functions. 
Sadly, it won't be possible to use it with DATE! values for long 
(see curecode). And, yes, I'm thinking on supporting some ODBC errors 
in their own error category. There are no best practices known to 
me on how to do that, though.

Yes. Many different scenarios. I have never used REBOL's ftp protocol, 
for example.

IIRC OID is used for snmp (simple network management protocol), and 
this one is really big and important ...

An idea of NTP scheme, but servers comunicates only on 123 UDP port. 
overview of time services:

Daytime:  Ascii response,  Graham and Ladislav has written a scheme/tool 
already  port 13

Time: most simple possible server listening on port  37 answer 32bit 
unsigned number of second from 1-1-1900/0:00 (calculation of human 
readable date is not so trivial because of leaping seconds inserted 
to UTC with no rule at all, an Earth is dancing a Jive in fact)

HTTP: use inserted Date-time from any header returned from server 
port 80

SNTP: more precise protocol (contains also fraction of second in 
reply) subprotocol of NTP on UDP port 37

NTP: most precise available to compare more time servers, and calculate 
with computed transport delay and phase shift from evaluated couple 
of handshaking packets.  UDP port 37

The latter two use minimally 12  32bit binary packets for request 
and response, symmetric or asymetric cryptography possible (honestly 
I've no clue why this).

Graham's SMTP protocol doesn't seem to have a place for them, either

To clarify the SSL stuff:

Since the SSL is a layer on top of TCP the idea was that R3 will 
have all the neccesary encryption algorithms (RSA,DH, DSA, RC4, SH256 
etc.) probably in form of embedded extension as part of the host-kit. 
These algorithms needs to be fast so they will be in C (probably 
ported from the R2 codebase if possible). Then the SSL/TLS protocol 
itself won't be written in C as it was in R2 but just done in R3 
script. This way the protocol code will be:
-smaller in size than the C version

-easier to maintain because it is Rebol language, for example we 
can add 'server mode', certificate validation (simmilar to web browsers) 
etc.
-crossplatform as much as Rebol script can be


So far I did simple TLS implementation in R2 to prove that concept. 
The prototype is ~20KB of rebol script and uses only the build in 
encryption ports in R2. It covers most of the TLS functionality that 
is written natively in in form of 'tls scheme. So the next step is 
to get the encryption math to R3 (which can be useful not only for 
SSL so it is definitely worth doing that) and then try how the prototype 
will behave.

the whole bitwise thing is pretty fucked up anyway. i tried to do 
a disk editor, a pic microcontroller HEX file processor, a custom 
serial communication protocol and in all cases i had to ping-pong 
between binary! issue! integer! and had to trim to the right bit/byte 
counts. it was a nightmare all the time.

BEER was / is a Rebol implementation of BEEP (the protocol). Our 
communication layer is a C based multi-threaded BEEP implementation 
that we make available to R3 as an extension.

I would add following "negatives" (depends upon how you look into 
it):


- no /libary extension and easy wrapping of DLLs. There was a bounty 
started to bring in kind of R2 DLL capabilities using extensions, 
Max was working on something, but did not deliver. Some ppl claim, 
that working with extensions is easy enough, much more powerfull, 
and that in fact R2 /library interface was weak in comparison in 
capabilities.


- weak and underpowered CALL.No /output or /wait parameter IIRC. 
Carl said, that R2 C code to it was complex, and that the code is 
eventually awailable for volunteer to bring in to R3. The outcome 
is - CALL is limited in usage in comparison to what can be easily 
achieved in R2.


- protocols. The only protocol IIRC was available was HTTP, done 
by Gabriele. It was HTTP 1.1 compatible, but due to some bug (?) 
it was downgraded to 1.0 version. No proxy support. Other protocols 
were done by some other ppl, I do remember Graham doing some work 
here. In regards to protocols, IIRC there was some work done by Kaj, 
who brought Curl networking extension to R3.


- under Windows console is a bit more inconvenient in usage than 
in R2, we use native Windows console, yet we don't have full console 
support, so we can't replace the native R3 one by e.g. Console2 or 
some other version ...


- DBAccess - forget R2 protocols available. The rescue is ODBC extension 
for R3


- CGI - no native CGI support in R3, though it should not be difficult 
to emulate


- Sorting & Unicode - althought we have Unicode strings available, 
sort is not adapted to that, and the question is, if it can be easily 
done ...

We could as well use the group power-pack, although it was meant 
a bit differently - to create package of most usefull add-on stuff, 
so e.g. mysql protocol, rugby, uniserve, etc.

In theory, it should be possible for delayed network protocol modules 
to be autoloaded on first use.

The modules that implement the protocols could be delay-loaded and 
registered with the general protocol dispatcher. Then the dispatcher 
could import the module the first time it is needed.

ftp protocol ...

There are a few different time protocols, and the standard time servers 
don't tend to run the daytime protocol. They usually run NTP.

2 amacleod: time protocol is not very accurate, the same levely of 
accuracy you can get by reading any HTML size and distile the time 
from HTML header. OTOH NTP protocol is able to get milisecond accuracy 
but by quite difficult handshake and as far as I know not yet written 
for rebol

The REBOL SSL port might also be using some REBOL code for higher 
level protocol support, but it's not accessible, so we can't check 
that.

It could be completely entangled with the SSH protocol, but since 
they also do SFTP, one can hope it is somewhat abstracted internally