Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search
 

[grid]

 [1/14] from: atruter:labyrinth:au at: 8-Jul-2004 12:14


> How much memory does it use (morbid curiosity :)?
Loading the 4,000,000 value block (4 columns * 1,000,000 rows) consumes 187Mb, while converting these values to 1,000,000 strings and displaying in a text-list consumes another 204Mb. I've been predominantly concerned with performance to date, so haven't spent as much time optimising memory usage as I could.
> I did a proof-of-concept thing a while back that used dynamically > generated draw commands to display virtual grid/tree data. The grid > version is below; watch for wrap.
If I get a chance I'll see if I can get my data using this to compare the difference. Regards, Ashley

 [2/14] from: greggirwin:mindspring at: 8-Jul-2004 22:37


Hi Ashley,
>> How much memory does it use (morbid curiosity :)?
AT> Loading the 4,000,000 value block (4 columns * 1,000,000 rows) consumes AT> 187Mb, while converting these values to 1,000,000 strings and displaying AT> in a text-list consumes another 204Mb. I've been predominantly concerned AT> with performance to date, so haven't spent as much time optimising memory AT> usage as I could. Wow! Cool that it doesn't fall over and die, but is there a reason you need to hold all 1'000'000 rows? That is, why not window smaller sets? -- Gregg

 [3/14] from: atruter:labyrinth:au at: 9-Jul-2004 18:30


Hi Gregg,
> Wow! Cool that it doesn't fall over and die, but is there a reason you > need to hold all 1'000'000 rows? That is, why not window smaller sets?
Agreed, the 'text-list approach is QAD but works well for smaller sets. I adapted your grid style and it uses very little memory over that required to load the table, except that right-justification of a column is not so easy and I'd like every column to be as wide as the widest value in it (whether currently displayed or not). ;) Regards, Ashley

 [4/14] from: antonr:lexicon at: 9-Jul-2004 21:52


Ah, for that you need to pre-scan the data, which can be time-consuming for the first view. You have to layout each cell as it would appear in the grid (depends how you are doing that) and see how wide it is. Anton.

 [5/14] from: greggirwin:mindspring at: 9-Jul-2004 9:11


Hi Ashley, AT> Agreed, the 'text-list approach is QAD but works well for smaller sets. I AT> adapted your grid style and it uses very little memory over that required AT> to load the table, except that right-justification of a column is not so AT> easy and I'd like every column to be as wide as the widest value in it AT> (whether currently displayed or not). ;) All the more reason to do windowing. There's a reason Google only displays a few results at a time; people can only make sense of so much information at once. Nobody will be able to browse through 1,000,000 records, so see if you can whittle down the data to a manageable amount. Even if you retrieve all the data, just calculate the display based on a window into it and recalculate as necessary. Efficiency like this is often a lot more work, so you have to prioritize it against other needs as well of course. Good luck! -- Gregg

 [6/14] from: ed:brittlestar at: 9-Jul-2004 21:22


Gregg wrote:
> All the more reason to do windowing. There's a reason Google only > displays a few results at a time; people can only make sense of so > much information at once. Nobody will be able to browse through > 1,000,000 records, so see if you can whittle down the data to a > manageable amount. Even if you retrieve all the data, just calculate > the display based on a window into it and recalculate as necessary.
Scrolling a pane of content within /View often leads to noticeable lags in screen refresh/responsiveness. For an example, try the REBOL world in AltME on a run-of-the-mill Pentium II or III. To avoid the lag, I almost always use paging controls to skip through sets of content. The technique works well and users seem comfortable with that convention. I'm eager to try out your grid to see if presents an option for manipulating significant sets of data-- a "must-have" for a large percentage of apps & utilities. Thanks for sharing that. // Ed

 [7/14] from: atruter:labyrinth:au at: 10-Jul-2004 13:00


Out of interest I changed the output target to an HTML file (using cols and colgroups to "help" the browser as much as possible). REBOL generated the 67Mb HTML file from a 35Mb source file in under 5 seconds but the browser (IE) died after consuming 1.2Gb of memory and a couple of minutes of disk thrashing (on a P4 with 1Gb RAM). Regards, Ashley

 [8/14] from: g:santilli:tiscalinet:it at: 10-Jul-2004 9:52


Hi Ed, On Saturday, July 10, 2004, 3:22:57 AM, you wrote: EOC> Scrolling a pane of content within /View often leads to noticeable lags in EOC> screen refresh/responsiveness. For an example, try the REBOL world in AltME EOC> on a run-of-the-mill Pentium II or III. This is because you are getting more mouse events than you can handle. Event filtering will put that lag to zero. Regards, Gabriele. -- Gabriele Santilli <[g--santilli--tiscalinet--it]> -- REBOL Programmer Amiga Group Italia sez. L'Aquila --- SOON: http://www.rebol.it/

 [9/14] from: james:mustard at: 10-Jul-2004 23:20


AT > Out of interest I changed the output target to an HTML file (using cols AT > and colgroups to "help" the browser as much as possible). REBOL generated AT > the 67Mb HTML file from a 35Mb source file in under 5 seconds but the AT > browser (IE) died after consuming 1.2Gb of memory and a couple of minutes AT > of disk thrashing (on a P4 with 1Gb RAM). Did you send each row as a separate table? This can make a HUGE difference as to whether a browser will die with such large data sets. I would be surprised if even mozilla / firefox / opera could open such a large dataset in a single table... James. ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.

 [10/14] from: ed:brittlestar at: 10-Jul-2004 9:22


> James Marsden wrote: > Did you send each row as a separate table? This can make a HUGE
difference as
> to whether a browser will die with such large data sets. > > I would be surprised if even mozilla / firefox / opera could open such a
large
> dataset in a single table...
Another option might be to add the CSS1 attribute: STYLE="table-layout: fixed" to the <TABLE> tag, which may save significant processing time for some browsers (IE and NS 5+) . More information at: <http://hotwired.lycos.com/webmonkey/98/26/index1a_page2.html?tw=design> although, there are probably better references on this available. Regards, Ed

 [11/14] from: ed:brittlestar at: 10-Jul-2004 14:50


> EOC> Scrolling a pane of content within /View often leads to noticeable
lags in
> EOC> screen refresh/responsiveness. For an example, try the REBOL world in
AltME
> EOC> on a run-of-the-mill Pentium II or III. > > GS> This is because you are getting more mouse events than you can > GS> handle. Event filtering will put that lag to zero.
I assumed it was because scrolling requires /View to refresh the screen at a faster rate than the compositing engine can handle. Is event filtering available now, or is that something planned for a future release of VID? Thanks for your insight. Ed

 [12/14] from: carl:cybercraft at: 11-Jul-2004 12:07


>> EOC> Scrolling a pane of content within /View often leads to noticeable >> lags in
<<quoted lines omitted: 7>>
>Is event filtering available now, or is that something planned for a future >release of VID?
If I find events are queuing up I just set a flag in the routine that's taking the time and ignore any events that call it until the flag's unset again. ie, something like this... processing: false routine: does [ processing: true ... ... processing: false ] view layout [ ... feel [engage: func [face action event] [ if all [action = '... not processing][routine] ] ] Obviously you shouldn't skip events like key-presses where users are typing in text, but for ones you can skip it usually works quite well. -- Carl Read

 [13/14] from: g:santilli:tiscalinet:it at: 11-Jul-2004 11:36


Hi Ed, On Saturday, July 10, 2004, 8:50:26 PM, you wrote: EOC> Is event filtering available now, or is that something planned for a future EOC> release of VID? Available now by just using my filtering wake-event function or Romano's eat function. If you have an application that needs to scroll and lags, try just adding this at the beginning and let me know the results: #do [document { ===Event Filtering Patch to wake-event (wake-event.r) This patch changes the awake function of the event port. The new function filters queued events so that "useless" events are not propagated to the event system. (For example, queued mouse events are usually useless, unless you are writing a freehand paint program.) It is possible to customize the function by changing the kind of events that are filtered when queued. The NO-QUEUE object is used for this; an event of a type that matches a word in the object, is never queued. }] ; Event filtering --- speeds up view! context [ no-queue: context [move: offset: none] wake-event: func [event /local no-btn] bind [ either not pop-face [ do event empty? screen-face/pane ] [ either any [pop-face = event/face within? event/offset win-offset? pop-face pop-face/size] [ no-btn: false if block? get in pop-face 'pane [ no-btn: foreach item pop-face/pane [if get in item 'action [break/return false] true] ] if any [all [event/type = 'up no-btn] event/type = 'close] [hide-popup] do event ] [ if pop-face/action [ if not find [move time] event/type [hide-popup] do event ] ] none? find pop-list pop-face ] ] in system/view 'self system/ports/wait-list/1/awake: func [port /local event events lasttype] [ events: clear [] while [event: pick port 1] [ either all [in no-queue event/type lasttype = event/type] [ change back tail events event ] [ lasttype: event/type insert tail events event ] ] foreach event events [ if wake-event event [return true] ] false ] ] Regards, Gabriele. -- Gabriele Santilli <[g--santilli--tiscalinet--it]> -- REBOL Programmer Amiga Group Italia sez. L'Aquila --- SOON: http://www.rebol.it/

 [14/14] from: ed::brittlestar::com at: 12-Jul-2004 8:19


Gabriele wrote:
> If you have an application that needs to scroll and lags, try just > adding this at the beginning and let me know the results: <snip>
Thank you, Gabriele and Carl (and Romano). I'll experiment with your techniques. Code like this amazes me.You and other gurus on this list have mastered the intricacies of View to such a level, it's pretty stunning. Thanks again. Ed

Notes
  • Quoted lines have been omitted from some messages.
    View the message alone to see the lines that have been omitted