[grid]
[1/14] from: atruter:labyrinth:au at: 8-Jul-2004 12:14
> How much memory does it use (morbid curiosity :)?
Loading the 4,000,000 value block (4 columns * 1,000,000 rows) consumes
187Mb, while converting these values to 1,000,000 strings and displaying
in a text-list consumes another 204Mb. I've been predominantly concerned
with performance to date, so haven't spent as much time optimising memory
usage as I could.
> I did a proof-of-concept thing a while back that used dynamically
> generated draw commands to display virtual grid/tree data. The grid
> version is below; watch for wrap.
If I get a chance I'll see if I can get my data using this to compare the
difference.
Regards,
Ashley
[2/14] from: greggirwin:mindspring at: 8-Jul-2004 22:37
Hi Ashley,
>> How much memory does it use (morbid curiosity :)?
AT> Loading the 4,000,000 value block (4 columns * 1,000,000 rows) consumes
AT> 187Mb, while converting these values to 1,000,000 strings and displaying
AT> in a text-list consumes another 204Mb. I've been predominantly concerned
AT> with performance to date, so haven't spent as much time optimising memory
AT> usage as I could.
Wow! Cool that it doesn't fall over and die, but is there a reason you
need to hold all 1'000'000 rows? That is, why not window smaller sets?
-- Gregg
[3/14] from: atruter:labyrinth:au at: 9-Jul-2004 18:30
Hi Gregg,
> Wow! Cool that it doesn't fall over and die, but is there a reason you
> need to hold all 1'000'000 rows? That is, why not window smaller sets?
Agreed, the 'text-list approach is QAD but works well for smaller sets. I
adapted your grid style and it uses very little memory over that required
to load the table, except that right-justification of a column is not so
easy and I'd like every column to be as wide as the widest value in it
(whether currently displayed or not). ;)
Regards,
Ashley
[4/14] from: antonr:lexicon at: 9-Jul-2004 21:52
Ah, for that you need to pre-scan the data,
which can be time-consuming for the first view.
You have to layout each cell as it would appear
in the grid (depends how you are doing that) and
see how wide it is.
Anton.
[5/14] from: greggirwin:mindspring at: 9-Jul-2004 9:11
Hi Ashley,
AT> Agreed, the 'text-list approach is QAD but works well for smaller sets. I
AT> adapted your grid style and it uses very little memory over that required
AT> to load the table, except that right-justification of a column is not so
AT> easy and I'd like every column to be as wide as the widest value in it
AT> (whether currently displayed or not). ;)
All the more reason to do windowing. There's a reason Google only
displays a few results at a time; people can only make sense of so
much information at once. Nobody will be able to browse through
1,000,000 records, so see if you can whittle down the data to a
manageable amount. Even if you retrieve all the data, just calculate
the display based on a window into it and recalculate as necessary.
Efficiency like this is often a lot more work, so you have to
prioritize it against other needs as well of course.
Good luck!
-- Gregg
[6/14] from: ed:brittlestar at: 9-Jul-2004 21:22
Gregg wrote:
> All the more reason to do windowing. There's a reason Google only
> displays a few results at a time; people can only make sense of so
> much information at once. Nobody will be able to browse through
> 1,000,000 records, so see if you can whittle down the data to a
> manageable amount. Even if you retrieve all the data, just calculate
> the display based on a window into it and recalculate as necessary.
Scrolling a pane of content within /View often leads to noticeable lags in
screen refresh/responsiveness. For an example, try the REBOL world in AltME
on a run-of-the-mill Pentium II or III.
To avoid the lag, I almost always use paging controls to skip through sets
of content. The technique works well and users seem comfortable with that
convention. I'm eager to try out your grid to see if presents an option for
manipulating significant sets of data-- a "must-have" for a large percentage
of apps & utilities.
Thanks for sharing that.
// Ed
[7/14] from: atruter:labyrinth:au at: 10-Jul-2004 13:00
Out of interest I changed the output target to an HTML file (using cols
and colgroups to "help" the browser as much as possible). REBOL generated
the 67Mb HTML file from a 35Mb source file in under 5 seconds but the
browser (IE) died after consuming 1.2Gb of memory and a couple of minutes
of disk thrashing (on a P4 with 1Gb RAM).
Regards,
Ashley
[8/14] from: g:santilli:tiscalinet:it at: 10-Jul-2004 9:52
Hi Ed,
On Saturday, July 10, 2004, 3:22:57 AM, you wrote:
EOC> Scrolling a pane of content within /View often leads to noticeable lags in
EOC> screen refresh/responsiveness. For an example, try the REBOL world in AltME
EOC> on a run-of-the-mill Pentium II or III.
This is because you are getting more mouse events than you can
handle. Event filtering will put that lag to zero.
Regards,
Gabriele.
--
Gabriele Santilli <[g--santilli--tiscalinet--it]> -- REBOL Programmer
Amiga Group Italia sez. L'Aquila --- SOON: http://www.rebol.it/
[9/14] from: james:mustard at: 10-Jul-2004 23:20
AT > Out of interest I changed the output target to an HTML file (using cols
AT > and colgroups to "help" the browser as much as possible). REBOL generated
AT > the 67Mb HTML file from a 35Mb source file in under 5 seconds but the
AT > browser (IE) died after consuming 1.2Gb of memory and a couple of minutes
AT > of disk thrashing (on a P4 with 1Gb RAM).
Did you send each row as a separate table? This can make a HUGE difference as
to whether a browser will die with such large data sets.
I would be surprised if even mozilla / firefox / opera could open such a large
dataset in a single table...
James.
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
[10/14] from: ed:brittlestar at: 10-Jul-2004 9:22
> James Marsden wrote:
> Did you send each row as a separate table? This can make a HUGE
difference as
> to whether a browser will die with such large data sets.
>
> I would be surprised if even mozilla / firefox / opera could open such a
large
> dataset in a single table...
Another option might be to add the CSS1 attribute:
STYLE="table-layout: fixed"
to the <TABLE> tag, which may save significant processing time for some
browsers (IE and NS 5+) .
More information at:
<http://hotwired.lycos.com/webmonkey/98/26/index1a_page2.html?tw=design>
although, there are probably better references on this available.
Regards,
Ed
[11/14] from: ed:brittlestar at: 10-Jul-2004 14:50
> EOC> Scrolling a pane of content within /View often leads to noticeable
lags in
> EOC> screen refresh/responsiveness. For an example, try the REBOL world in
AltME
> EOC> on a run-of-the-mill Pentium II or III.
>
> GS> This is because you are getting more mouse events than you can
> GS> handle. Event filtering will put that lag to zero.
I assumed it was because scrolling requires /View to refresh the screen at a
faster rate than the compositing engine can handle.
Is event filtering available now, or is that something planned for a future
release of VID?
Thanks for your insight.
Ed
[12/14] from: carl:cybercraft at: 11-Jul-2004 12:07
>> EOC> Scrolling a pane of content within /View often leads to noticeable
>> lags in
<<quoted lines omitted: 7>>
>Is event filtering available now, or is that something planned for a future
>release of VID?
If I find events are queuing up I just set a flag in the routine that's taking the time
and ignore any events that call it until the flag's unset again. ie, something like
this...
processing: false
routine: does [
processing: true
...
...
processing: false
]
view layout [
... feel [engage: func [face action event] [
if all [action = '... not processing][routine]
]
]
Obviously you shouldn't skip events like key-presses where users are typing in text,
but for ones you can skip it usually works quite well.
-- Carl Read
[13/14] from: g:santilli:tiscalinet:it at: 11-Jul-2004 11:36
Hi Ed,
On Saturday, July 10, 2004, 8:50:26 PM, you wrote:
EOC> Is event filtering available now, or is that something planned for a future
EOC> release of VID?
Available now by just using my filtering wake-event function or
Romano's eat function.
If you have an application that needs to scroll and lags, try just
adding this at the beginning and let me know the results:
#do [document {
===Event Filtering Patch to wake-event (wake-event.r)
This patch changes the awake function of the event port. The new
function filters queued events so that "useless" events are not
propagated to the event system. (For example, queued mouse events
are usually useless, unless you are writing a freehand paint
program.)
It is possible to customize the function by changing the kind of
events that are filtered when queued. The NO-QUEUE object is used
for this; an event of a type that matches a word in the object,
is never queued.
}]
; Event filtering --- speeds up view!
context [
no-queue: context [move: offset: none]
wake-event: func [event /local no-btn] bind [
either not pop-face [
do event
empty? screen-face/pane
] [
either any [pop-face = event/face within? event/offset win-offset? pop-face pop-face/size]
[
no-btn: false
if block? get in pop-face 'pane [
no-btn: foreach item pop-face/pane [if get in item 'action [break/return false] true]
]
if any [all [event/type = 'up no-btn] event/type = 'close] [hide-popup]
do event
] [
if pop-face/action [
if not find [move time] event/type [hide-popup]
do event
]
]
none? find pop-list pop-face
]
] in system/view 'self
system/ports/wait-list/1/awake: func [port /local event events lasttype] [
events: clear []
while [event: pick port 1] [
either all [in no-queue event/type lasttype = event/type] [
change back tail events event
] [
lasttype: event/type
insert tail events event
]
]
foreach event events [
if wake-event event [return true]
]
false
]
]
Regards,
Gabriele.
--
Gabriele Santilli <[g--santilli--tiscalinet--it]> -- REBOL Programmer
Amiga Group Italia sez. L'Aquila --- SOON: http://www.rebol.it/
[14/14] from: ed::brittlestar::com at: 12-Jul-2004 8:19
Gabriele wrote:
> If you have an application that needs to scroll and lags, try just
> adding this at the beginning and let me know the results: <snip>
Thank you, Gabriele and Carl (and Romano). I'll experiment with your
techniques.
Code like this amazes me.You and other gurus on this list have mastered the
intricacies of View to such a level, it's pretty stunning.
Thanks again.
Ed
Notes
- Quoted lines have been omitted from some messages.
View the message alone to see the lines that have been omitted