Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search
 

P2P Q&A

 [1/34] from: carl::rebol::com at: 30-Sep-2001 10:48


Question: [On the subject of Express and the REB...] Does that mean we're wasting our time on working on P2P stuff if it's going to be obsoleted soon? Answer: By no means will it be obsoleted. However, note that P2P is one of those loose terms. Most P2P is not true P2P. In fact, marketing folks sometimes call the entire collaborative market P2P... but what they really mean is person-to-person connections, rather than direct peer-to-peer networking. Express and the REB use a client-server model, but they do not exclude true P2P reblets. In fact, P2P can work better if you have a server coordinating and/or authenticating users. For instance, when you run RIM from the REB, it uses a server to locate other users, then connects directly to those users. But, where things really get interesting is when you distribute the servers themselves... yes. -Carl

 [2/34] from: etienne:alaurent:free at: 30-Sep-2001 22:40


I'm not sure to understand you. Can you explain this more ? Are you saying you will distribute soon the Express server (I forgot its name) ? --- Etienne --- Carl Sassenrath wrote:

 [3/34] from: petr:krenzelok:trz:cz at: 30-Sep-2001 23:07


Hi, some time ago Holger described P2P term in a very nice way. I can send you the message once I get to my work's computer tomorrow morning. What does Carl mean by distributing servers themselves is beyond my imagination now. I just don't think it is possible to pass sockets to another processes. I think few ppl here already asked about it, but it was never answered IIRC ... I can be of course completly off-topic here :-) -pekr-

 [4/34] from: holger:rebol at: 30-Sep-2001 16:08


On Sun, Sep 30, 2001 at 11:07:08PM +0200, Petr Krenzelok wrote:
> Hi, > some time ago Holger described P2P term in a very nice way. I can send you
<<quoted lines omitted: 3>>
> another processes. I think few ppl here already asked about it, but it was > never answered IIRC ... I can be of course completly off-topic here :-)
Distribute in the topological sense, not the marketing/sales sense, i.e. Carl is talking about distributing servers across a network of machines. There is no need to have only one single central server. One can think of all kinds of topologies, e.g. slave servers running off one master, cascaded servers, multi-star setups, and even servers which are dynamically "created" from clients based on network load (assuming clients which are reachable the way servers are, i.e. no firewalls etc.). These servers would act as a single "logical" server (one name, a common state), distributed across multiple machines. Express does not do these things yet, but there is no reason why it should not be able to do it in the future, transparently to applications. -- Holger Kruse [holger--rebol--com]

 [5/34] from: petr:krenzelok:trz:cz at: 1-Oct-2001 6:15


[holger--rebol--com] wrote:
> On Sun, Sep 30, 2001 at 11:07:08PM +0200, Petr Krenzelok wrote: > > Hi,
<<quoted lines omitted: 18>>
> Express does not do these things yet, but there is no reason why it > should not be able to do it in the future, transparently to applications.
I can understand it. That would be great. I think that Maarten's Rugby is heading that way. Now more direct and technical question - is it technically possible to accept connection (conn: first listen-port), and once done, pass the connection to another node (rebol process on the same machine, remote rebol process) to handle such request? I mean - the topology where we have one server listening, and distributing requests to another processes/machines, without further involvement ... Theoretically we are talking packets here, but once connection is approved between two machines, both sides register remote ip, port number assigned by operating system, etc. So? :-) Thanks, -pekr-

 [6/34] from: holger:rebol at: 30-Sep-2001 22:17


On Mon, Oct 01, 2001 at 06:15:42AM +0200, Petr Krenzelok wrote:
> I can understand it. That would be great. I think that Maarten's Rugby is > heading that way. Now more direct and technical question - is it technically
<<quoted lines omitted: 5>>
> connection is approved between two machines, both sides register remote ip, port > number assigned by operating system, etc. So? :-)
Only in one of two ways: either by having the peer create a separate connection (in that case the first server acts as a lookup system), or by having the first server forward all data to another machine, through a separate connection (in that case the first server acts as a proxy). -- Holger Kruse [holger--rebol--com]

 [7/34] from: koopmans:itr:ing:nl at: 1-Oct-2001 10:42


> I can understand it. That would be great. I think that Maarten's Rugby is > heading that way. Now more direct and technical question - is it > technically possible to accept connection (conn: first listen-port), and > once done, pass the connection to another node (rebol process on the same > machine, remote rebol process) to handle such request?
Rugby is already there. Rugby allows you to expose Rebol funcs. Rebol funcs can accept blocks. Blocks can be executed. Provide some extra security to preventy malicious blocks from being executed (for example by creating a dialect) and you are done. Combine this with View where you can integrate servers and clients due to the event system and with the http transport layer (which is also non-blocking).... All of the things Holger mentions are currently done at my daytime job using Rugby and distributing tasks over 10 machines on all kind of platforms using Command, including FreeBSD, Linux, Solaris and Windows. If you are interested in writing your own high-performance server you could look into hipe-serv.r in the Rugby distro. It is a generic non-blocking I/O engine. You supply it one function (the handler) and the rest is taken care off. See web-handler (the http proxy) and do-handler (the Rugby handler) as examples in server.r in the distro. Another good example is the rugby console that I put on the REB sometime ago. It allows you to run rugby servers as you type on any Core 2.5 engined Rebol, add and remove functions dynamically, etc. HTH, Maarten

 [8/34] from: greggirwin:starband at: 1-Oct-2001 9:27


There is an interesting book out from O'Reilly called 'Peer to Peer - Harnessing the Power of Disruptive Technologies'. It's a collection of essays by various P2P folks, some technical in nature and some more 'conecpt' oriented. Some of the contributors are the guys behind [SETI--Home], Gnutella, and Jabber for example. Lots of good, thought provoking, discussion in there IMO. --Gregg

 [9/34] from: etienne:alaurent:free at: 2-Oct-2001 0:27


In my opinion, but I can make a mistake, concepts behind Rugby are great and powerfull. But ... But, Rebol is not enough powerful now to take advantage of these concepts : no multi-threading, no asyncronism. So, Rugby is good to test and prototype, but not enough powerful for real life. In some words, WE NEED MULTI-THREADING AND ASYNCRONISM TO BUILD KILLER APPS IN REBOL !! Wait and see what RT will make in the very near future. --- Etienne --- Maarten Koopmans wrote:

 [10/34] from: petr:krenzelok:trz:cz at: 2-Oct-2001 8:05


Etienne ALAURENT wrote:
> In my opinion, but I can make a mistake, concepts behind Rugby are great > and powerfull.
<<quoted lines omitted: 5>>
> In some words, WE NEED MULTI-THREADING AND ASYNCRONISM TO BUILD KILLER > APPS IN REBOL !!
But - according to Holger (IIRC), RT will not add real multi-threading support, because there is some problem in regards to multiplatform issue. Only some kind of soft-multi-threading is planned iirc. We don't need multi-threading that much if we are async wherever we can. And that's another problem: - Core lacks timers. Real timers, something like make timer! time-period. First I saw timers in faces, I was confused why kernel of system lacks timers. Several ppl already requested event system or timers in Core, but we were told to use View. - much can be done with proper async behavior. But async protocols for e.g. are planned for Rebol 3.0, which is not likely to appear anytime soon ... So, what will RT bring us to help the situation? You know something, right Etienne? :-) -pekr-

 [11/34] from: holger:rebol at: 2-Oct-2001 4:07


On Tue, Oct 02, 2001 at 08:05:31AM +0200, Petr Krenzelok wrote:
> But - according to Holger (IIRC), RT will not add real multi-threading support, > because there is some problem in regards to multiplatform issue. Only some kind of > soft-multi-threading is planned iirc.
No, I never said that. I am not sure what you mean by real multi-threading vs. soft multi-threading. We do plan to add support for multi-threading, just not immediately.
> We don't need multi-threading that much if we are async wherever we can. And > that's another problem: > > - Core lacks timers. Real timers, something like make timer! time-period. First I > saw timers in faces, I was confused why kernel of system lacks timers. Several ppl > already requested event system or timers in Core, but we were told to use View.
Another thing planned for Core 3.0 is low-level port timeouts with 1 ms granularity, at the handler level. -- Holger Kruse [holger--rebol--com]

 [12/34] from: petr:krenzelok:trz:cz at: 2-Oct-2001 14:29


[holger--rebol--com] wrote:
> On Tue, Oct 02, 2001 at 08:05:31AM +0200, Petr Krenzelok wrote: > >
<<quoted lines omitted: 4>>
> multi-threading vs. soft multi-threading. We do plan to add support > for multi-threading, just not immediately.
Ah, I am sorry then, take my apology. Maybe I wrongly understood your following quote: On Wed, May 23, 2001 at 08:50:42PM +0200, Maarten Koopmans wrote: -> Doing things in the background suggests that you switch to a multithreaded -> stack-based machine? No, still single-threaded, but switching to a state machine-driven protocol engine, triggered from within 'wait and some other natives.
> > We don't need multi-threading that much if we are async wherever we can. And > > that's another problem:
<<quoted lines omitted: 4>>
> Another thing planned for Core 3.0 is low-level port timeouts with 1 ms > granularity, at the handler level.
That's cool. As we speak probably some more distant future, are some multi-media related things thought of already? Some of us would be interested probably in knowing, how/where will View evolve from its current state .... -pekr-

 [13/34] from: koopmans:itr:ing:nl at: 2-Oct-2001 16:04


Ok, this one is gonna be lengthy.... Pekr, the discussion I had with Holger was about how to implement async I/O. Hence your confusion. Now for multithreading / async I/O: The first thing to realize is that threads are really processes with shared memory. (And shared memory is the root of all evil.....) This also implies that mutli-threading for performance is only really working on multiple processor systems. Most systems converge to one processor because of price, especially for end-users. Async I/O will be very nice, but what's really sufficient for most people is a feature Rebol already has: non-blocking I/O This means that you can open a tcp connection (for example) and check every now and then if there is data, copy it, do something else etc. Concurrent behaviour can be achieved by cleverly combining non-blocking I/O with system/ports/wait-list and the view event system, especially if you have your own event loop written. A good point to remember here is that in the end your O/S (at least formost of us) also runs one processor. Now I am not saying that you should write your own OS, it's merely a blunt reminder that most things can be done in most environments if you really want to (threading at the interpreter level, implemented in REBOL for example). Take a look at the rugby console (rugby-console.r) on the REB, for example. It runs a server while you can type on the console in Core 2.5 Another example to prove my point: Ruby, a Japanese OO-like language with its own light-weight threading system, even on DOS! You could write a light-weight threading system if you'd really wanted too in Rebol as well. Now as for Rugby: you can call functions in other REBOL processes in Rugby using the /deferred or /http-deferred refinements. They return immediately with a ticket number for your request. You can use this number to check regularly if the result is there, and if so fetch it. This is done by using a client-side non-blocking TCP port. Every time you call result-available? <ticket> it just copies whatever it can from the TCP port and checks to see if the result is completely there. If so it returns true, otherwise false. If you get true, you simply call get-result <ticket> and the result is there. NOTE: it looks like this is not working on NT 4.0 sp3. All other platforms are OK. A sample, presuming the sample echo server is running: ;load rugby do %rugby.r ;This loads the rugby sample echo function on the client do get-rugby-service/http http://localhost:8002 t: echo/deferred "Who needs threading" while [ not result-available? t ] [ print "Doing nothing useful" print "But something different than fetching the echo" ] ;Now the result is there print get-result t So instead of worrying about the features not there you should look to what *is* there and how you can cleverly use it to solve your problems. The fact that one paradigma is missing doesn't mean that you can't solve it another way. In the end it all runs on the same computer..... --Maarten

 [14/34] from: holger:rebol at: 2-Oct-2001 7:03


On Tue, Oct 02, 2001 at 02:29:50PM +0200, Petr Krenzelok wrote:
> > No, I never said that. I am not sure what you mean by real > > multi-threading vs. soft multi-threading. We do plan to add support
<<quoted lines omitted: 5>>
> No, still single-threaded, but switching to a state machine-driven > protocol engine, triggered from within 'wait and some other natives."
These are two different issues. One is support for network i/o virtually in the background , using asynchronous ports. That is something planned for the not-too-distant future (Core 3.0). This will not require multi-threading. Multi-threading at the language level is a different issue. It IS planned, but for later.
> That's cool. As we speak probably some more distant future, are some multi-media related > things thought of already? Some of us would be interested probably in knowing, how/where > will View evolve from its current state ....
We will have to see. Technically, there is not much of a problem. The internal View engine handles dynamic data better than any other graphical compositing system I have ever seen. The timer limitations in the current version really only exist because that part of View was written before we rewrote the event system for Core 2.3, but we should be able to allow 1 ms granularity in faces fairly soon. Of course whether the CPU can handle it everywhere is a different issue :-). Multi-media, of course, requires streaming, but this is something planned for Core 3.0 as well, at least for network protocols and some utility ports (encryption, compression and checksums). The really big hurdle for multi-media is licensing, especially in cross-platform environments. People these days expect support for .avi, .mpg, .mov, .rp etc., and many codecs. In many cases this is technically possible, but not legally, and only on few platforms. I would expect support for this in REBOL to be driven primarily by commercial demand and very particular licensing requests, so it is difficult to make any predictions at this time. -- Holger Kruse [holger--rebol--com]

 [15/34] from: etienne:alaurent:free at: 2-Oct-2001 19:25


Petr Krenzelok wrote:
>So, what will RT bring us to help the situation? You know something, right >Etienne? :-) > >-pekr- >
I'm sorry to not be in the RT confidence. So I can't say nothing :-( --- Etienne

 [16/34] from: gchiu:compkarori at: 3-Oct-2001 9:09


On Tue, 2 Oct 2001 16:04:53 +0200 Maarten Koopmans <[koopmans--itr--ing--nl]> wrote:
> NOTE: it looks like this is not working on NT 4.0 sp3. > All other platforms > are OK.
To be more explicit, it seems not to be working on NT4 sp6 when run in server mode. Client mode seems to be okay. -- Graham Chiu

 [17/34] from: dockimbel:free at: 3-Oct-2001 1:25


Hi all, Maarten Koopmans wrote:
> Ok, this one is gonna be lengthy....
I'm afraid mine too...:) [...]
> The first thing to realize is that threads are really processes with shared > memory. (And shared memory is the root of all evil.....) This also implies > that mutli-threading for performance is only really working on multiple > processor systems.[...]
I do not agree. Multi-threading works and performs very well on a single processor. [...]
> Now I am not > saying that you should write your own OS, it's merely a blunt reminder that > most things can be done in most environments if you really want to (threading > at the interpreter level, implemented in REBOL for example).
In Rebol ? I already thought about doing that, but firstly, i see no way to do it in a premptive mode but only in a cooperative mode (like Win3.1) and secondly, i'm afraid that it will spend more time switching between your scripts than evaluating them...:) IMHO, the only viable way is to implement it at least at the interpreter level (in C).
> Take a look at the rugby console (rugby-console.r) on the REB, for example. > It runs a server while you can type on the console in Core 2.5
This is just non-blocking I/O ! You're multiplexing between several I/O which spend most of their time waiting for something to happen on ports ! When something finally happens on a port, you have to pray that the event processing won't take much time else you'll block all other ports...It can work fine if all requests processing take very little time, but if any request takes several seconds to complete, all others will...wait ! With a single process server, you can answer only one request at the same time ! You can build a high-perf server based on a single thread only if you're spending most of the time multiplexing I/O events and very few time in request processing. That works very well for a web server (building a HTTP response doesn't take much time) like Zeus or MEDUSA, but i'm don't think that an ORB or Application Server can match these criteria. Apart from that, i'm sure that Rugby works well for simple requests (passing a message from a port to another, giving date & time, even performing a small request on a database,...) provided that the average time between 2 requests is at least the same that a request processing average time...else watch out for the lag ! :)
> Another example to prove my point: Ruby, a Japanese OO-like language with its > own light-weight threading system, even on DOS! You could write a > light-weight threading system if you'd really wanted too in Rebol as well.
Same answer as above. About Ruby, i assume that's a soft-threading model implemented at the C level.
> Now as for Rugby: you can call functions in other REBOL processes in Rugby > using the /deferred or /http-deferred refinements. They return immediately > with a ticket number for your request. You can use this number to check > regularly if the result is there, and if so fetch it.
Sure ! Multi-processing is way better that single-processing for this kind of server ! But multi-threading is even better : - Shared memory (Multitasking heaven, but notoriously difficult to deal with!), so that a thread could access words from others threads and this way exchange data way faster and easier than using tcp/ip connections (especially if you have a lot of threads). Memory management is also more efficient. (for example: data caching) - Lightweight context-switching : better performance than with multi-processing. We need multithread to make really serious apps with Rebol, especially for serious server apps ! IIRC, /Serve use multithreading internally. [...]
> t: echo/deferred "Who needs threading"
Me and other Rebol developpers who try to find workarounds for their projects and who could finally choose drastic solutions like jumping to another langague, in a not-so-distant future. ;-) [...]
> So instead of worrying about the features not there you should look to what > *is* there and how you can cleverly use it to solve your problems. The fact > that one paradigma is missing doesn't mean that you can't solve it another > way. In the end it all runs on the same computer.....
Yep, both Win3.1 and BeOS 5 run on my computer, and guess which is running better/faster/cleaner ? ;-) Best regards, DocKimbel.

 [18/34] from: dockimbel:free at: 3-Oct-2001 1:27


[holger--rebol--com] wrote:
> On Tue, Oct 02, 2001 at 08:05:31AM +0200, Petr Krenzelok wrote: > >
<<quoted lines omitted: 4>>
> multi-threading vs. soft multi-threading. We do plan to add support > for multi-threading, just not immediately.
IMHO, i think that petr was talking about OS threads (at user or kernel level) vs Application threads which rely on its own scheduler and threading model (independant from the OS). IIRC, the last one is also called soft-threading (threads emulated at the application level). I assume that RT plans to implement a soft-threading system which could be supported by any platform (even mono-tasking OS, à la MacOS). (I will be so happy when this will come true !) ;-) Supporting OS threads where available, would be also a great feature ! (TIA) Regards, DocKimbel.

 [19/34] from: cyphre:volny:cz at: 3-Oct-2001 10:13


----- Original Message ----- From: "Nenad Rakocevic" <[dockimbel--free--fr]> To: <[rebol-list--rebol--com]> Sent: Wednesday, October 03, 2001 1:27 AM Subject: [REBOL] Re: P2P Q&A
> I assume that RT plans to implement a soft-threading system which could be
supported by any
> platform (even mono-tasking OS, à la MacOS). (I will be so happy when this
will come true !) ;-)
> Supporting OS threads where available, would be also a great feature !
(TIA)
Hi Nenad, I absolutely agree with you ;-) I hit this barier of single Rebol thread when developing SWIS. We need multi-threading at native level in REBOL! Then we could build the real great REBOS :) (I don't want to have running xy Rebol processes on my machine for each reblet eating lots of memory etc. I need only big one multithreaded Rebol.) Regards Cyphre

 [20/34] from: m:koopmans2:chello:nl at: 3-Oct-2001 9:03


And it will be lengthy again....
> I'm afraid mine too...:) > > [...] > > The first thing to realize is that threads are really processes with
shared
> > memory. (And shared memory is the root of all evil.....) This also
implies
> > that mutli-threading for performance is only really working on multiple > > processor systems.[...] > > I do not agree. Multi-threading works and performs very well on a single
processor.
It works well but you are switching between threads which is overhead you can save if you do it yourself within one process. Unless you have native threads, SMP, and multiple processors, because you are not switching then.
> [...] > > Now I am not > > saying that you should write your own OS, it's merely a blunt reminder
that
> > most things can be done in most environments if you really want to
(threading
> > at the interpreter level, implemented in REBOL for example). > > In Rebol ? I already thought about doing that, but firstly, i see no way
to do
> it in a premptive mode but only in a cooperative mode (like Win3.1) and
secondly,
> i'm afraid that it will spend more time switching between your scripts
than evaluating
> them...:) IMHO, the only viable way is to implement it at least at the
interpreter level (in C).
You have non-blocking I/O so you would change all IO to have that as a default mode, so it is interruptable. Then you'd give a weight to all the natives and count reductions on the rest in your evaluator and do the switching after n reductions. Still a lot of work though.
> > Take a look at the rugby console (rugby-console.r) on the REB, for
example.
> > It runs a server while you can type on the console in Core 2.5 > > This is just non-blocking I/O ! You're multiplexing between several I/O
which spend
> most of their time waiting for something to happen on ports ! When
something finally happens on
> a port, you have to pray that the event processing won't take much time
else you'll block
> all other ports...It can work fine if all requests processing take very
little time, but if
> any request takes several seconds to complete, all others will...wait ! >
That's true. My point was that you can get to things scheduled as far as I/O is concerned. If you program efficiently (divide your tasks and have them pass control back every now and then) this works fine.
> With a single process server, you can answer only one request at the same
time ! You can
> build a high-perf server based on a single thread only if you're spending
most of the
> time multiplexing I/O events and very few time in request processing. That
works very
> well for a web server (building a HTTP response doesn't take much time)
like Zeus or
> MEDUSA, but i'm don't think that an ORB or Application Server can match
these criteria.
NO!! I did middleware ;-) The point is related to the single processor problem: you are in the end switching between threads and are at most one is running at the same time. If you program your functions as little state machines that pass control back after every state and continue on the next invocation you get the same result. It is a lot of work, but so is debugging multithreaded applications.
> Apart from that, i'm sure that Rugby works well for simple requests
(passing a message from
> a port to another, giving date & time, even performing a small request on
a database,...)
> provided that the average time between 2 requests is at least the same
that a request
> processing average time...else watch out for the lag ! :) >
NO! again. I use it at work over 10 machines doing database queries, crypto'ing stuff etc. Works very fast. Remote encrypting strings (opening a port, en / de crypting , closing ) goes easily to 100-200 /sec for most strings of 128 bytes on a fast or giga ethernet. Encrypting takes all of your CPU and takes much longer than the requests. But i agree that careful programming is required, as in multithreading.
> > Another example to prove my point: Ruby, a Japanese OO-like language
with its
> > own light-weight threading system, even on DOS! You could write a > > light-weight threading system if you'd really wanted too in Rebol as
well.
> Same answer as above. About Ruby, i assume that's a soft-threading model
implemented
> at the C level. >
The point is that it can be done. Native threading is another process with shared memory. And on one processor you are just switching back and forth....
> > Now as for Rugby: you can call functions in other REBOL processes in
Rugby
> > using the /deferred or /http-deferred refinements. They return
immediately
> > with a ticket number for your request. You can use this number to check > > regularly if the result is there, and if so fetch it. > > Sure ! Multi-processing is way better that single-processing for this kind
of server !
> But multi-threading is even better : > > - Shared memory (Multitasking heaven, but notoriously difficult to deal
with!), so that
> a thread could access words from others threads and this way exchange data
way faster
> and easier than using tcp/ip connections (especially if you have a lot of
threads).
> Memory management is also more efficient. (for example: data caching) > > - Lightweight context-switching : better performance than with
multi-processing.
I agree. But not that it is better. Shared memory almost always gives race conditions you cannot predict. IMHO a nono.
> We need multithread to make really serious apps with Rebol, especially for
serious server
> apps ! IIRC, /Serve use multithreading internally. > > [...]
Async I/O I thought ;-) Jeff, Holger?
> > t: echo/deferred "Who needs threading" > > Me and other Rebol developpers who try to find workarounds for their
projects and who could
> finally choose drastic solutions like jumping to another langague, in a
not-so-distant future. ;-)
As I try to prove, that is not necessary, though you may feel uncomfortable with my proposed solutions of find them distasteful. Actually, I was just trying to help in showing " that there is more than one way to do it" (thanks Larry)
> [...] > > So instead of worrying about the features not there you should look to
what
> > *is* there and how you can cleverly use it to solve your problems. The
fact
> > that one paradigma is missing doesn't mean that you can't solve it
another
> > way. In the end it all runs on the same computer..... > > Yep, both Win3.1 and BeOS 5 run on my computer, and guess which is running
better/faster/cleaner ? ;-) Don't mix up threading schedulers and schemes here ;-)

 [21/34] from: m::koopmans2::chello::nl at: 3-Oct-2001 11:14

Threding? Was: P2P Q&A


Hey all, I don't see any mail coming through.... Anyway about my previous 'Who needs threading' posts: I don't mean thah threads aren't useful. I mean that you can get away lots of times without using them, within one process as well. I think that performance is not the issue here, but programmer convenience (and that is a point of Rebol, hence.... you'd expect threading). --Maarten

 [22/34] from: cyphre:volny:cz at: 3-Oct-2001 15:17

Re: P2P Q&A


----- Original Message ----- From: <[holger--rebol--com]> To: <[rebol-list--rebol--com]> Sent: Tuesday, October 02, 2001 4:03 PM Subject: [REBOL] Re: P2P Q&A
> We will have to see. Technically, there is not much of a problem. The > internal View engine handles dynamic data better than any other
<<quoted lines omitted: 3>>
> be able to allow 1 ms granularity in faces fairly soon. Of course > whether the CPU can handle it everywhere is a different issue :-).
Hi Holger and all, Good news ;-) It is very frustrating when your View code is slowing down by timer instead of CPU... View engine is really great but we are still missing some features which would enhance power of View in a great way.(Maybe I'm wrong but it wouldn't be so difficult for RT crew to implement some of them in the next release) Some time ago I posted on /Express developers conference little enhancement list(reflecting View programming exeprience from some developers and myself) but without any response from RT so here it is again: -possibility of creating/changing events -recursive filtering of events(recursive removing time-event-based functions...) -better mouse handling (show/hide/change mouse pointer, fixed OVER event) -event sensing when face moved -native face collision support -alpha channel and convolve funcitons available at developer's level -more complex working of effect pipeline (alowing any combination of fx commands) -improved(crash when drawing outside of bounds, "circle" bug...) and enhanced(size of lines, filled-patterns, elipse, anti-aliasing..., circle gradients) DRAW dialect -native rotation of face's content in any angle -at least simple font engine (for equal visualisation on multiple platforms) Would be at least some of this points solved in near future? I would appreciate your(Holger?Anyone?) comments... Regards Cyphre

 [23/34] from: holger:rebol at: 3-Oct-2001 8:29


On Wed, Oct 03, 2001 at 09:03:51AM +0200, Maarten Koopmans wrote:
> It works well but you are switching between threads which is overhead you > can save if you do it yourself within one process. Unless you have native > threads, SMP, and multiple processors, because you are not switching then.
Actually, you are, if the threads are tightly coupled, and you may end up with less performance than if you just forgot about SMP support in your threading layer altogether. Sun learned that the hard way in their attempt to support Solaris SMP in Java. There are basically three types of threading support: 1. Single CPU, non-preemptive, switching completely under application control (in the case of an interpreter "application" is the interpreter, not the script). 2. Single CPU, preemptive (at least partially, e.g. by having no control over switching during OS calls, or by having Unix signals trigger switching). 3. Multiple CPUs. Whether switching is preemptive on each CPU or not does not matter (for the purpose of this analysis). Intuitively 3 appears to perform better than 2, and 2 better than 1. The rule is: if your threads are loosely coupled (do not have many resource dependencies) then the intuition is correct, and 3 is preferable. However if threads are tightly coupled then it is the other way around: 1 performs best and 3 worst. Here is why: With tightly coupled threads only one thread can physically run at any time anyway, because it needs to lock all (low-level) resources. For interpreters you would have to, e.g., lock symbol tables, name-value lookup tables (for REBOL: contexts) and a lot of other things. Almost every statement in a script makes changes in some way, so you need to lock exclusively. Locking at a smaller granularity than "everything" is usually impossible, because there is no natural hierarchy to data structures, and multiple locks would thus inevitably either lead to deadlocks or to such an enormous number of master lock lock layers and individual locks in order to avoid deadlocks that the overall performance gets even worse than by just using one single global lock. Knowing that, you can pretty much forget about SMP benefits, because only one thread and thus one CPU can run at any time anyway. That means cases 2 and 3 become very similar in performance. 3 is actually worse, because when switching from one thread to another in case 2 you only have ONE THREAD switch. In case 3 you have TWO TASK switches (because each CPU switches between one thread in your overall task and some OTHER task). MUCH more overhead. By going from 2 to 3 you are trading light-weight switching for heavy-weight switching, without any net benefit from parallelism: bad idea. And 1 performs better than 2 and 3 because it allows implicit locking. If the runtime system controls thread switching then it does not need to lock resources that are shared among threads at all. The active thread owns them by convention while it executes. Of course this only applies to low-level resources, within the interpreter, which are accessed "below" the switching layer. It says nothing about high-level resources which are accessed "above" the switching layer, e.g. non-atomic data structures within your script. You will still have to provide semaphore locks for those, but that is true for all three cases. It is interesting to see that the natural reaction of a lot of people is to assume that preemptive switching is always superior to non-preemptive switching. This is actually not the case. The intuition probably comes from the experience with Windows, where the change to preemptive switching provided so many advantages. However that is only because a lot of Windows applications were derived from MS-DOS or modeled after existing MS-DOS applications, and were therefore not written in a multitasking-friendly manner to start with. With non-preemptive switching a run-away application could lock up the whole system. With preemptive switching it would only result in a task not responding , keeping the system alive. If all tasks "played by the rules" then you would not see any difference between preemptive and non-preemptive switching at all. Of course for an operating system you would still want preemptive switching at the task level, but that's a different matter... Things are a lot different if each thread runs within an interpreter (regardless of whether it is a "real" interpreter like REBOL or a bytecode interpreter like Java). In that situation the interpreter itself can "play nicely" with the threading engine, within the interpreter loop, making run-away threads impossible, i.e. even with something like "forever []" in a non-preemptive threading environment you would still not lock up other threads, because the interpreter itself would preempt threads, even if the underlying threading engine is non-preemptive. This means in this kind of environment preemptive threading has no advantages over non-preemptive threading, but it does have one disadvantage: higher overhead and thus worse performance because of locking requirements. The same is true for SMP threading (preemptive or non-preemptive). It is no surprise that, from my experience, most people running Java in Solaris SMP environments choose not to enable Java's native SMP support, but stick with Java's single-CPU "green-threads" layer instead, which is basically a non-preemptive single-CPU threading engine with wrappers around system functions to allow "controlled preemption". Same thing for REBOL: if/when we do support multi-threading it is very likely going to be a solution that falls into category one, supporting single CPUs only (per REBOL process), with implicit internal locking and some primitives at the REBOL language layer to manipulate threads and priorities, and provide mutual exclusion. At the script layer the engine would appear preemptive, but below the hood it would not be, thus no SMP.
> > We need multithread to make really serious apps with Rebol, especially for > serious server > > apps ! IIRC, /Serve use multithreading internally. > > > > [...] > Async I/O I thought ;-) Jeff, Holger?
Yes, async i/o. No real multithreading. Of course single-threading and async i/o combined with a state machine per pending task leads to pretty much exactly the same performance (sometimes even better) as multi-threading with blocking i/o per thread. One advantage of the single-threading solution is that load balancing, fairness, resource management, sequencing etc. usually become much easier, because the information required to manage that is explicitly available in the data structures, not hidden within the threading environment. The main drawback is that writing programs that way takes some getting used to. Sometimes people refer to the first technique as "multi-threading" as well, simply because the end result is the same, and to benefit from the "buzz" that the term multi-threading creates among the less technical crowd :-). For instance a lot of so-called "multi-threaded filesystems" or "multi-threaded TCP/IP stacks" don't really use multi-threading at all, but actually use single-threading with async i/o and state machines. The user never knows the difference, neither do benchmarks. You only recognize it once you see the source code. -- Holger Kruse [holger--rebol--com]

 [24/34] from: m:koopmans2:chello:nl at: 3-Oct-2001 18:05


Thanks Holger. The point you make when talking about serve is exactly what I am trying to say: you can get the benefits without the worries in a slightly different programming style. If RTdoes threading within the interpreter (sometime) there really shouldn't be a difference when programming your own eval loop with clever switching between differnet tasks. It all runs on the same computer.... --Maarten

 [25/34] from: holger:rebol at: 3-Oct-2001 10:37


On Wed, Oct 03, 2001 at 03:17:30PM +0200, Richard Smolak wrote:
> -possibility of creating/changing events
On our list. Being able to "remote control" View would be quite cool...
> -recursive filtering of events(recursive removing time-event-based > functions...)
I am not sure I understand what you mean by that.
> -better mouse handling (show/hide/change mouse pointer, fixed OVER event)
Perhaps, but keep in mind that this is highly platform-dependant. The best you can probably do across platforms is either to use named, default mouse pointers (such as 'normal, 'busy, 'text, 'pointer etc.) or custom graphics up to 16x16 monochrome transparent.
> -event sensing when face moved > -native face collision support
Those two sound like you want to use faces as sprites in games. Not really what they were intended for...
> -alpha channel and convolve funcitons available at developer's level
On our list.
> -more complex working of effect pipeline (alowing any combination of fx > commands)
Actually all combinations are supposed to work, but there are a few bugs. On our list.
> -improved(crash when drawing outside of bounds, "circle" bug...)
Do you have reproducable cases ? That "crash when drawing outside of bounds" is a running joke in our office. Every time Sterling wants to demonstrate it to me he cannot reproduce it. I have yet to actually see it :-). Circle bug ? What bug ?
> enhanced(size of lines, filled-patterns, elipse, anti-aliasing..., circle > gradients) DRAW dialect
Hmmmm.... Turning View into X11... Filled patterns have been on the list all along and are still planned. Circle gradients: perhaps: Custom widths, anti-aliasing and ellipses: not sure. The code for those tends to get complex and big very quickly.
> -native rotation of face's content in any angle
Difficult to do if you expect results in reasonable quality :), i.e. anti-aliased etc.
> -at least simple font engine (for equal visualisation on multiple platforms)
We looked at some existing engines (FreeType etc.), but they are all either huge or expensive or both, plus they typically do not include actual fonts. Something we could probably do is include a few (2 or 3) fixed-size, fixed-width, non-scalable bitmap fonts in the View binary. I am not sure if that would help match though. -- Holger Kruse [holger--rebol--com]

 [26/34] from: greggirwin:starband at: 3-Oct-2001 11:51


Hi Cyphre,
>> -better mouse handling (show/hide/change mouse pointer, fixed OVER event)
YES! I'm not sure what you meand by "fixed OVER event" but I did find, recently, that if I resize a face with the mouse, but limit the maximum size, the event/offset (if the mouse is off the face but button still down) matches the maximum face span value. I.e. dragging larger than the max allowed size works fine but as soon as you start moving back in toward the face it starts shrinking again unless you do a little dance and even then it's wierd.
>> -alpha channel and convolve funcitons available at developer's level
I tried to implement Kirsch edge detection recently, without success. :) So, yes, something like a hook into the effect pipeline would be nice. Maybe a parallel to insert-event-func (insert-effect-func)?
>> -improved(crash when drawing outside of bounds, "circle" bug...) and >> enhanced(size of lines, filled-patterns, elipse, anti-aliasing...,
circle
>> gradients) DRAW dialect
I'm all for basics (line weights would be high on my list) but, again, being able to insert our own draw effects would be cool. Not sure how you would extend the dialects to deal with this, unless you did something like use compose. I would like an easy way to manage the various faces created by View/New. I haven't messed around yet to see what the answer might be but I like building little apps that are "aware" of each others locations so they can snap-to and link up in an intelligent manner. Another thing that's important to me, which I haven't seen mentioned much, is printing. If I can have fine-grained control of the printer, with a bonus of being able to render that output to the screen as well, that's a big plus. At this point it looks like super-PDF-maker might be handy. :) OTOH, there's no reason we couldn't have a REBOL format, or even a dialect, that worked like (or mapped to) PostScript. NeXT used PostScript for both printing and display. Maybe there are multiple approaches. I really want simple output most of the time, but there's no denying the power of PostScript, PDF, and TeX. Then again, maybe the goal is a paperless environment, in which case we just have to be able to convince people that they don't *need* hardcopy. --Gregg

 [27/34] from: d4marcus:dtek:chalmers:se at: 3-Oct-2001 20:59


On Tue, 2 Oct 2001, Maarten Koopmans wrote:
> Another example to prove my point: Ruby, a Japanese OO-like language with its > own light-weight threading system, even on DOS! You could write a
<<quoted lines omitted: 3>>
> with a ticket number for your request. You can use this number to check > regularly if the result is there, and if so fetch it.
It would be cool though if Rebol had something like this builtin: t: thread [file] [ list-dir file: to-file ask "Give a filename to write the result to: " ] result: do massively-timeconsuming-calculations file: await t/file write file result Instead of letting the user wait for the result either before or after typing the filename, it runs as the user types. I suppose this could be implemented with ports as you suggest, but perhaps it would be slightly more complicated than the above? Marcus ------------------------------------ If you find that life spits on you calm down and pretend it's raining

 [28/34] from: petr:krenzelok:trz:cz at: 3-Oct-2001 21:39


----- Original Message ----- From: <[holger--rebol--com]> To: <[rebol-list--rebol--com]> Sent: Wednesday, October 03, 2001 7:37 PM Subject: [REBOL] Re: P2P Q&A
> On Wed, Oct 03, 2001 at 03:17:30PM +0200, Richard Smolak wrote: > > > -possibility of creating/changing events > > On our list. Being able to "remote control" View would be quite cool...
you can bet on it :-)
> > -recursive filtering of events(recursive removing time-event-based > > functions...) > > I am not sure I understand what you mean by that.
I am not sure I remember it correctly, but does Cyphre mean following? : when you hid face, containing sub-faces, all timer in subsequent faces are stopped/removed from queue? Cyphre?
> > -better mouse handling (show/hide/change mouse pointer, fixed OVER
event)
> Perhaps, but keep in mind that this is highly platform-dependant. The > best you can probably do across platforms is either to use named, > default mouse pointers (such as 'normal, 'busy, 'text, 'pointer etc.) > or custom graphics up to 16x16 monochrome transparent. >From the times of first alpha of View I suggested ability to hide mouse
pointer. That will be needed for kiosk mode (full screen). I suggested to replace mouse pointer with face, containing sub-face, placed somewhere in system structure. Imagine transparent face containing some 100x100 hand pointer, where subface of some 20X20 would be sensitive, e.g. finger pointing at target ...It can be of course done even today, but how to hide that mouse cursor? :-) bt: what about event transparent faces? Or even alpha transparent faces? E.g. some rule which would allow events to be sent to underlying face, e.g. famous Tao Intent capability, where you have ring, and boing ball in background. You can catch/move boing ball in middle of ring, as it is transparent ....
> > -event sensing when face moved > > -native face collision support > > Those two sound like you want to use faces as sprites in games. Not > really what they were intended for...
I thought so ;-) But we want shadow of the beast to be ported to View :-)
> > -alpha channel and convolve funcitons available at developer's level > > On our list.
cool ...
> > -improved(crash when drawing outside of bounds, "circle" bug...) > > Do you have reproducable cases ? That "crash when drawing outside of > bounds" is a running joke in our office. Every time Sterling wants to > demonstrate it to me he cannot reproduce it. I have yet to actually > see it :-).
heh, I just hope your answer doesn't upset Cyphre, as I can bet he talked about in in Express conference, as well as sent it to feedback, from which I can bet, he haven't received any feedback anyway ;-)
> > enhanced(size of lines, filled-patterns, elipse, anti-aliasing...,
circle
> > gradients) DRAW dialect > > Hmmmm.... Turning View into X11... Filled patterns have been on the list > all along and are still planned. Circle gradients: perhaps: Custom > widths, anti-aliasing and ellipses: not sure. The code for those tends > to get complex and big very quickly.
I just wonder - what techniques does flash use allowing it fast and nice effects, movements, etc.?
> > -native rotation of face's content in any angle > > Difficult to do if you expect results in reasonable quality :), i.e. > anti-aliased etc.
so give us some not so huge general ability to create own effects :-) You know, we want Rebol to become distributed Scala in multimedia quality. If you would allow us to live inside of browser, that would be end of JavaScript :-) Of course you would have to support link to browser document object. Killer app imo ...
> > -at least simple font engine (for equal visualisation on multiple
platforms)
> We looked at some existing engines (FreeType etc.), but they are all > either huge or expensive or both, plus they typically do not include > actual fonts. > > Something we could probably do is include a few (2 or 3) fixed-size, > fixed-width, non-scalable bitmap fonts in the View binary. I am not > sure if that would help match though.
Not sure it is worth the effort then ... better concenrate on Rebol 3.0 and fine granularity of IO :-) OK now, who's gonna do compiler finally? :-) Cheers, -pekr-

 [29/34] from: holger:rebol at: 3-Oct-2001 13:30


On Wed, Oct 03, 2001 at 09:39:04PM +0200, Petr Krenzelok wrote:
> I am not sure I remember it correctly, but does Cyphre mean following? : > when you hid face, containing sub-faces, all timer in subsequent faces are > stopped/removed from queue? Cyphre?
AFAIR this is fixed. I don't remember if the fix is in the current View release though.
> bt: what about event transparent faces?
'over events are actually passed through the whole face tree now, allowing you to capture them in a parent face. This behavior has to be explicitly enabled to allow backwards compatibility. AFAIR this feature is not in the current View release though.
> Or even alpha transparent faces?
Will come as a side effect once alpha-channels are added. -- Holger Kruse [holger--rebol--com]

 [30/34] from: carl:cybercraft at: 4-Oct-2001 12:19


On 04-Oct-01, Petr Krenzelok wrote:
> ----- Original Message ----- > From: <[holger--rebol--com]> > To: <[rebol-list--rebol--com]> > Sent: Wednesday, October 03, 2001 7:37 PM > Subject: [REBOL] Re: P2P Q&A >> On Wed, Oct 03, 2001 at 03:17:30PM +0200, Richard Smolak wrote:
[snip]
>>> -improved(crash when drawing outside of bounds, "circle" bug...) >> Do you have reproducable cases ? That "crash when drawing outside
<<quoted lines omitted: 6>>
> feedback, from which I can bet, he haven't received any feedback > anyway ;-)
The "circle" bug I know of is that the circle shows as a square if the pen and fill colours are the same. (That's off the top of my head, but I think it's right.)
>>> -at least simple font engine (for equal visualisation on >>> multiple platforms)
<<quoted lines omitted: 5>>
>> fixed-width, non-scalable bitmap fonts in the View binary. I am >> not sure if that would help match though.
Text is so important for cross-platform compatibility that you need to sort this out one way or another. If it's not possible to include your own font engine in View, then how about getting three scalable fonts created for REBOL, (fixed-width, serif and sans-serif), and porting them to each of the platforms you support, ensuring that the font sizes actually match aross platforms? This would require users to install the fonts as well as installing View, but it'd be worth it if it meant we no longer had to guess if text would or wouldn't fit perfectly within buttons. A font engine in View itself would be the coolest though. We could make our own fonts then. -- Carl Read

 [31/34] from: cyphre:volny:cz at: 4-Oct-2001 10:32


----- Original Message ----- From: <[holger--rebol--com]> To: <[rebol-list--rebol--com]> Sent: Wednesday, October 03, 2001 10:30 PM Subject: [REBOL] Re: P2P Q&A
> On Wed, Oct 03, 2001 at 09:39:04PM +0200, Petr Krenzelok wrote: > > I am not sure I remember it correctly, but does Cyphre mean following? : > > when you hid face, containing sub-faces, all timer in subsequent faces
are
> > stopped/removed from queue? Cyphre? > > AFAIR this is fixed. I don't remember if the fix is in the current View > release though. >
Yes, the problem with recursive removing time-events when face is 'hide is fixed(If I remember from last version of /Link or so..). But I think we need sometimes recursively remove/freeze time events without hiding the face. I know, we can recursively search the whole face-tree and set rate to 0 or so(this would be sloow when searching more complex faces) but why not use the part of code from 'hide command and make similar one for example 'freeze. So when you: freeze my-face , my-face and all its subfaces won't receive time events but still remains shown...this will work like tap in pipe cascade of event-tree... Maybe this could be solved differently when creating/changing of events will be implemented. Then we could for example filter all time events containing destination my-face using own functions inserted on the top of the event tree...for example: func [face event][if all [event/type = 'time event/dest = my-face] [return none] return event] Cyphre

 [32/34] from: petr:krenzelok:trz:cz at: 4-Oct-2001 11:01


Richard Smolak wrote:
> ----- Original Message ----- > From: <[holger--rebol--com]>
<<quoted lines omitted: 19>>
> won't receive time events but still remains shown...this will work like tap > in pipe cascade of event-tree...
I wonder what happened to my proposed find/deep or locate suggestion. If it would implemented in C level, it would fly ... -pekr-

 [33/34] from: cyphre:volny:cz at: 4-Oct-2001 11:09


Hello Holger, Thanks for your reply ;) ----- Original Message ----- From: <[holger--rebol--com]> To: <[rebol-list--rebol--com]> Sent: Wednesday, October 03, 2001 7:37 PM Subject: [REBOL] Re: P2P Q&A
> On Wed, Oct 03, 2001 at 03:17:30PM +0200, Richard Smolak wrote: > > > -possibility of creating/changing events > > On our list. Being able to "remote control" View would be quite cool... >
Great, I'm so impatient :-)
> > -better mouse handling (show/hide/change mouse pointer, fixed OVER
event)
> Perhaps, but keep in mind that this is highly platform-dependant. The > best you can probably do across platforms is either to use named,
<<quoted lines omitted: 4>>
> Those two sound like you want to use faces as sprites in games. Not > really what they were intended for...
Yes but View engine is fast enough for creating 2D and even some simple 3D games so why not implement collisions? I cannot see into the guts of View engine but this wouldn't be so hard to implement imo. It the face has in options word 'sprite then each visible pixel(or pixel of user chosen color(s))of this face will be checked during rendering... But maybe I'm wrong and the problematics is more complicated. Regarding event sensing: Few moths ago I had a chat with Carl about that feature at link conference. Carl put this into bug database at /Link developers server so I thought this will be solved in any future release...
> > -alpha channel and convolve funcitons available at developer's level > > On our list. >
great :)
> > -more complex working of effect pipeline (alowing any combination of fx > > commands) > > Actually all combinations are supposed to work, but there are a few bugs. > On our list. >
Great!! Sometimes, I'm puzzled when building effect block ;-)
> > -improved(crash when drawing outside of bounds, "circle" bug...) > > Do you have reproducable cases ? That "crash when drawing outside of > bounds" is a running joke in our office. Every time Sterling wants to > demonstrate it to me he cannot reproduce it. I have yet to actually > see it :-). >
try: do http://www.sweb.cz/rebolek/matrix.r nice 3D demo isn't it? ;-) try to use cursor arrows to rotate....works great... then try '+' an '-' keys on the keypad...when you zoom "out of the bounds" /View will crash... I think it is due some problem during creating of image datatype itself. Looks like when you create new image and draw out of bounds...looks like we can "draw" into unallocated memory or so? I have found out that rebol crashes when usually "fill" into negative coordinates but this is not a rule...
> Circle bug ? What bug ?
You don't know legendary "circle" bug? ;-) try this: view layout [box with [effect: [draw [pen red fill-pen red circle 50x50 20]]]]
> > enhanced(size of lines, filled-patterns, elipse, anti-aliasing...,
circle
> > gradients) DRAW dialect > > Hmmmm.... Turning View into X11... Filled patterns have been on the list > all along and are still planned. Circle gradients: perhaps: Custom > widths, anti-aliasing and ellipses: not sure. The code for those tends > to get complex and big very quickly. >
Why not elipses when you have done circle? I thought It's question of one parameter...
> > -native rotation of face's content in any angle > > Difficult to do if you expect results in reasonable quality :), i.e. > anti-aliased etc. >
I will be very happy even with rotation without anti-aliasing :) Anyway, thanks for your comment. It put more light into the situation about the way of the next View... Last question, when we can expect next release of /View? ;-) regards, Cyphre

 [34/34] from: arolls:idatam:au at: 5-Oct-2001 15:41


Hooray! Alpha channel! Looking forward to it. -Anton.

Notes
  • Quoted lines have been omitted from some messages.
    View the message alone to see the lines that have been omitted