AltME groups: search

I was also getting it (linux), but now it the file appears to be 
gone and it's stopped

Let us know if this continues Max. File sharing has been disabled 
on this world, but there seems to be a glitch. I haven't seen it, 
and it's cleared up for others, so you may be our special test case 
to help track it down.

MaxV, fumbling in the dark, but does it also occur, if you click 
the drop down in the upper left corner (where the world name is labeled) 
and select the world you are logged into?

yes and so have some rebol sites

Thanks Max. My connection is sometimes to blame, and we're spoiled 
by how fast things usually are with AltMe.

Bug: I just posted in Core group. There was a delay before the message 
was echoed back to me, so before that happened I switched to another 
group to continue reading. Then I switched back to Core, and found 
my message echoed properly. However, my message was still in the 
New message text entry area.

Thanks. 

That's useful evidence that the problem is not network or server 
related [rebol3 and rebol-gate run on the same server]


Next step ---  is it better or worse in groups with many messages? 
(ie is the problem the world or the group):

--could you try a test post or two in a couple of groups that have 
less than 100 posts?
  list available here: http://www.rebol.org/aga-index.r

I was sending messages to chris here .. and they were taking 30 seconds 
too

-- Do you have a _lot_ of private messages between you and Chris?

-- Do you see similar delays when private messaging with other members 
of this world?

[It could be that the private message distribution process is affected, 
while the broadcast of public messages is  not]

Hmm, so if Graham rejoined as GChiu (say) he'd have an empty private 
message set, and his messaging would be fast again....

....Sounds like a good justification for a second account for a prolific 
member of a busy world.

But maybe it does :) Who knows but Carl and those who wrote it :)

yes... and its why I PM so little now.  its just painfull.

I can see one larger group....REBOL3-old was 4.3meg.

And that was retired and replaced by the REBOL3 precisely because 
of the speed problem.

Doesn't the speed also have to do with the message limit per chat 
settings? I've decreased mine and  it's much faster. Though I do 
wonder how accurate that setting is or what it really measures. I 
set mine to a higher number after a search and warning that it needs 
to be set higher. I've thought that there's no way there could be 
that many messages (several 1000's) in that one topic.

it just limits what is visible in your scrollable view above.   it 
will affect speed so much as it uses less faces to display the list 
and thus less RAM is managed by the GC.

doing an OS (file content) search in the altme directory usually 
is easier and faster too.

I could look at the back off my laptop and count the flashes??

5 seconds for me.  No other bugs with AltME for the most part.

I have learned though that if you have a message ready to be sent, 
and change groups a few times, when you hit send, it is not 100% 
sure it will be sent to the corrrect group.

Looking at my Altme file activity (Filemon) when I post a message 
:

- Nothing happens until the server has done its job and send back 
the new message for the corresponding group/user.

- Then I see that the corresponding chat file is wrote entirely by 
block of 4096B !

- Then the state file (chatMYALTMEID) is updated (wrote by chunk 
of 4096B).

James : It would be interesting for us if you can look the file activity 
of a world server when a new post is send.

I use Filemon from Sysinternals (no included in Process monitor) 
for that.
Can you test this for us and tell us what happens ?

Didier, Process Monitor includes both file and registry monitoring 
- it effectively obsoletes regmon and filemon. You might be thinking 
of Process Explorer.

Also, technically, a given send box should be directly connected 
to the given group, so you can send, and even if it is in transit 
or waiting you can go on and work on other messages.

We do tag messages this way in Qtask, but we have seen this same 
bug, but don't have proof (as I do in AltME) that it is the program 
and not the operator.  But I'm always on the look out for it since 
it can be a very dangerous bug for some people.


This whole class of issue falls to a perceived form of latency, and 
a series of features are needed to suppress it.  For example, instead 
of graying the out going message area, just make it look like it 
was sent, but also instantly add a “message hour glass” in the “New 
message” bar when on that group, and, also create a new tinted empty 
box in the thread saying something like “Waiting for new message 
to post”.


This frees you up to go elsewhere.  If you come back to the same 
group it is clear what state it is in.  If you want to retrieve the 
sent message just click on the “Message hour glass” and it will bring 
it back into the input box.


Lastly, of course, if you try to leave AltME before it is done sending, 
need to offer people a dialog warning them , and giving them access 
to the messages waiting.

There is more, but yeah, it takes a lot to make this work.

This is very interesting ... I was trying to post to a group and 
got "internet busy ..." so instead of restarting Altme I opened up 
another Altme client and can post merrily away

Wonderful ... Altme is showing me the wrong time too ... and not 
my new daylight savings time

I'm not sure Graham. I've never seen that behavior until just now. 
I have two worlds I keep open at all times, and their Internet Busy 
issues almost always coincide with general net issues. Right now 
though, after waking from sleep, one of them is stuck on Internet 
Busy.

I get "stuck on internet busy"  often... 


when I have an issue with it... I just close altme and start a new 
one... it takes 5 seconds so its not a big deal.

I'm getting "Internet busy" a lot too.

I've just posted twice privately to Graham, and each time it took 
too long for the message to be echoed back to me, so I switched groups 
to continue reading. After a few minutes, I went back and the message 
had appeared. However, in both instances, my message was not cleared 
from the New message entry area, so I had to clear it manually.

(and again...)

this altme is getting slower and slower ...

Are there are any benchmarks about max. file-size on AltME and number 
of files it can handle?

And the opposite direction, hilighted text outside of AltME is automatically 
copied to the X clipboard, which can then be pasted here

how about Rebol-and-no-trolls-world

I've just asked Carl to help in maintaining Altme in another comm 
channel... hopefully he'll see that and reply to me in a not to distant 
future.

So, db on both servers and clients?

Interesting, why use a DB and add bloat, why not just write just 
enough code to keep it fast?
For example, loading in messages in groups of 100....

That's essentially the data structure we use in the archive at REBOL.org, 
Reichart.

Works fast and has scaled so far -- with nearly 150,000 posts from 
this world:
   http://www.rebol.org/aga-groups-index.r?world=r3wp

Graham, yep, that's how I would solve it.


Reichart, well, 250KB for SQLite is not that much bloat IMO and supports 
full-text searching as well.

I love AltMe, but it has things that annoy me as well. Since we don't 
know exactly why it doesn't scale (yes, we can guess), I don't think 
just making the back end a DB is a guaranteed fix, or perhaps even 
that simple. And I don't think it will happen. I fear the R3 chat/rebdev/??? 
model will scale even worse than AltMe as I think the msg db is a 
single file and still a naive implementation designed for small-scale 
use.


Like Graham, I've thought that something like Git--a system designed 
for distributed, differenced, repositories of information--might 
be used. I haven't seen it done however.

What ar our options?

1) Live with AltMe, and roll worlds when they get too slow.
2) Find an alternative.
3) Implement our own.

Is #1 painful enough to drive us to #2 or #3?

I'd rather have 10 years or whatever in one database and not mulitple 
worlds as one wears out

actually the r3 chat system *client* is still pretty simple... but 
the way messages are managed via the server seems to be much more 
flexible.  every msg and file is its own single item.  they are downloaded 
as items, not as groups of items.


so actually, all that needs to be done to make r3 chat scale is to 
improve how it stores its messages on disk and on RAM.   


the fact that the server supports moderation, threads, user levels, 
files, and things like item re-classification is very nice.


really, all it needs is a gui client.... that's If the server is 
able to respond quickly to requests.

though I'm sure its possible to rewire how Altme message passing 
works.  as long as the server has the data and client/server are 
using the same message protocol, everything can be changed.


it can even be retro-fitted on update, to prevent people from having 
to download the whole world again, since we already have the data 
on our disks.

Better to start a new world than to stay in this one and see personal 
forks like Graham started:-)

And we are probably all lazy dogs :-) I like how we speculate, yet 
noone has ever asked Carl. Actually - I did, and it resulted in a 
fix of expired worlds. And it is always the same and the same. Carl 
once told me - tell me what the problem is, and I will look into 
it. So - anyone to ask Carl for open-sourcing AltME? :-)

I guess cindy was tired of looking at them and it was either ebay 
or the garbage...  ;-)

I think that someone from RM Assest who is now close to Carl would 
have to ask Carl, about the possiblity of open-sourcing AltME, but 
- AltME is used even by some commercial entities, so I don't know 
if it is likely to happen. As for me, I prefer Carl working on R3. 
And I can't see the situation being dramatic, nor do I agree with 
the launch of the new world. In such a case, I might consider not 
being present. I am used to be here, and apart from personal messages, 
there is not any significant slowdown or outages lately. We were 
in much worse situation in the past imo ...

How about ... we create a backup of this world somewhere, and prune 
all the rarely used groups from here?

Every few days someone stumbles by but only to quickly leave again 
in general. Every few weeks one of those passersby actually asks 
something. And then, about one out of three of those questions is 
actually REBOL-related.

Maybe I should setup altme on another PC and see whether the slowdown 
is a local effect ... ie see how long it takes for a message to appear 
on the other instance

I think it's a combination of AltMe and the server it's running on.

I want to put a second world on my computer / world server and in 
the past I spent a fair bit of time resetting worlds due to them 
becoming unreachable.

It has been fixed. I run two worlds, and no unexpected outages since 
the time of the fix ....

Brock, I've been running 2 unreserved and 1 reserved world for years 
without any issues.

updated the server and now accessible

if Carl is just doing a TAR while the altme server is running, it's 
not strange that one or two files don't get backed up correctly. 
a quick workarourd is to keep more than just the last backup, as 
the chance of a file missing from two backups is pretty small.


of course, ideally one would be using ZFS and backing up from a snapshot...

It doesn't help that the client can't detect the issue and save the 
old data somehow. Once you sync a corrupted group, your local store 
is gone (unless you're backing them up locally ;-).

is it just me or does this mean that moderation and message alteration 
is actually possible... 

it would only require a few changes to the server and client so they 
support this feature.

Is there a problem with AltME at all? All this that has happened 
seems to not be directly related to AltME, but to the server and 
the backup scheme.

Partially so - but - plain files instead of DB storage of messages. 
If I understand it correctly, e.g. Links group here, got purged, 
and it refers to 153.set - old messages are gone, without the notice.

Henrik, the problem is that the server overwrites the client even 
when the server is wrong (corruption on the server side). It would 
be nice if it recognized the condition and recovered files from the 
clients.


The other problem is: can you think of a backup scheme to avoid this? 
(Note, we're speculating on the base of guesses, but still.)

Backup scheme: I suppose the only way is really some kind of snapshot, 
or by having AltME server pack a single file once an hour, that a 
backup system then can grab and stow somewhere.

or you just run dropbox and have it backup for you automatically 
....

Okay, so.... I just had to reinstall AltME on linux, so here's some 
things I noticed:

*It took me a little over 4 minutes to resync a little over 60 MB 
of data (checked net connection for amount transferred)

*SQLite and Advocacy both got stuck at the end, re-sycing over and 
over until I restarted AltME

*I clicked through everything because I didn't know there was a "Mark 
all messages as read" button, until I re-selected my custom dividers...

I just had a VERY strange problem in altme... its the second time 
this week and its never happened before...


I stopped receiving notifications from altme, but it was still linked 
in, and not reporting Internet busy....

I just did a logout/login and saw that 7 groups had new messages 
in them  !!!

the strange thing is that pn both occasions I remember hearing a 
new message "ping" and there being nothing new... which is a bit 
strange...

Maxim -- I see that regularly....It's the remnants of the old resync 
bug.

Luckily, logging off and back in seems to resync correctly -- not 
like the old days when getting groups to sync properly was capricious.

And sometimes, you'll see the All Groups heading in red, but no group 
heading below it is in red. It's like a phantom group has just received 
a message from Jim Phelps.

That way, we can miss if someone contacts us. So the urge is to check 
if some name is at the top of the list, and better check from time 
to time ....

Yep -- both Max and Petr confirm that 2nd message returns the red. 
So the sync problem seems to resolve itself.

But, as Petr says, we should all check the top few in our lists in 
case there are un-red messages unread.

I think we need to create a dedicated group for Custom Gob Renderers... 
there will be a lot of discussion and support or HW bug tracking 
about it too.

just make it general please. Cyphre used some other method for acceleration, 
and we should consider various possible available solutions, look 
also at how Flash is doing it, etc. methods ...

this is specific for R3 architecture and host-kit related stuff  
its not about using graphics in rebol, its about building the graphics 
engines.

Carl - yes, you simply logged on, and found your pms gone, rewritten 
with new file, or something like that.

Gab has a complete backup of the world, and thought it might be possible 
to merge old and new data, but only RT can say for sure.

Roughly Oct 12-14. Many groups and PMs got lost.

Assuming it can be done, and that it would merge old and new PMs, 
does everyone here want it done?

(R3 Chat uses better method, and can use demand-based virtual msg 
entries to lower the overhead even more.)

+1 for restoring/merging old and new PMs

AltME uses a write/append which for some reason on Linux means: if 
you cannot append, truncate the entire file to zero.


For performance reasons (blame the early optimization guys), the 
default setting for many file systems on Linux is to write metadata 
independently from data, which means that it often gets written first. 
The result is that if writing the data fails (eg. power failure, 
kernel panic, ...), the files end up 0 length.


This can be disabled by setting the filesystem to always write data 
and metadata at the same time (at the price of write performance).

Could one of our world masters please create a "!REBOL3 Proposals" 
group and put it under the R3 divider? Thanks.

Cancel that gripe.....I think it just took a while to resync and 
update the list.

ok... so I am thinking loud.... should we switch  the new and old 
group namesÉ

since, as Brian noted, so far, most of the !REBOL  discussion has 
been about proposals, ideas and stuff?

oh.. I guess it would break up the [web-public] stuff  and any links 
to them.

Carl: http://www.mjmwired.net/kernel/Documentation/filesystems/ext3.txt

data=ordered should be default on ext3 (not so on other fs though), 
but your distribution may be using writeback. or, what you have seen 
may have been due to something else. (personally i've often seen 
the 0 byte files on XFS after crashes, but never with ext3 or ext4)


There's also the fact that the fsync() semantics are not what most 
apps expect it to be, which turned into the filesystems trying to 
work around bad apps... and thus complexity.

I manually removed altme from my Ubuntu netbook today as 1.1.29 would 
neither connect nor uninstall  -  is it worth trying again with a 
more recent release, or should I just use this ole windoze box (with 
1.2.25) and spare myself  the aggravation?

and would refuse to update itself because  ... ?

Likely because there's too much difference with 1.2.x, and 1.2.x 
for Linux was always officially just a beta, which they didn't automatically 
update

I was using 1.1.29 up until a few weeks ago when it stopped connecting, 
and have been using 1.2.25 since

AltME broken for me....


Background: this morning my Windows computer froze and I had to cycle 
the power on it. (AltME was running, but nothing to suggest that 
AltME was the cause of the problem).


After a reboot, AltME would not start -- complained there was a file 
it could not read.


No worries....I deleted the entire REBOL3 folder, and restarted AltME.


After it had downloaded 40+ meg of messages, all should have been 
back to normal. But the resync process got itself stuck, continually 
resyncing the SQLite group.


I have since tried downloading the whole world again; and I have 
uninstalled AltME completely and started again from scratch. Same 
problem.


Now, if I restart AltME, it looks okay at first. But, if I click 
into the SQLite group (or if someone posts a message to that group) 
the endless resync starts up again.


I am not seeing the problem on a different machine where I have not 
reinstalled AltME.

This is like the opposite of the ancient resync bug :)

***


I suspect the SQLite folder is somehow corrupt on the server; but 
the corruption only triggers trouble on a fresh install.....If so, 
we will all see it eventually :(

Can someone copy this to whoever maintains AltME? Thanks!

A new world has several attractions.

But if the problems lie in the codebase and/or the backup strategy, 
then they will recur..

I posted an Altme issue over at Accounts this AM - as I relented 
and put  a 1.2+ (the page said 0.42 ??? )  on my Ubuntu netbook - 
it now connects - but then rejects my account - which is working 
from this Windows box 1.2.25   Must be me.  3-letter word for dolt 
... let's see ... "usr"

If I could reset it HERE, that would not be an issue.  As I say I 
can both count to 3 and recite the alphabet ;-)  Whyever is it not 
in Settings by now?  There must be a good reason ...

Nope.  Just moved in boxes to another country.  And in the Netherlands 
those boxes could easily get soggy wet, No?

Including my Dutch dictionary and Eng-Dutch dictionary

Here it has rained for days, the river is known to flood -  and I 
live down by the river ( but not on the yearly flood plain )

Can you reset it and email me ?

and I am back via LINUX  libc6... so I can count higher than 3  ;-)

That 150,000th [web-public] post took a little longer than I anticipated 
last week (it seems we all stopped posting web-public once the possibility 
was mentioned).


But it has happened now -- congratulations to BrianH and his comment 
on R3's break/return:
    http://www.rebol.org/aga-display-posts.r?post=r3wp824x176


No word yet on whether he'll be flying to Romania to treat himself 
to the prize Kaj wanted :)
    http://www.rebol.org/aga-display-posts.r?post=r3wp151x5092

[So was I:) I could have cheated and posted 150,000th myself.....But 
that would have been cheating)
Nevermind, Maxim, 250,000th is the big one.

Bug:

The old inconsistent bug of failing to copy the text of a message 
by simple right-click prevented me from copying Oldes' message with 
the long referrer link and cookie data in it, so I selected the message 
text first before right-click. That worked, but then all messages 
in the display began disappearing (looks like a shared para scroll 
bug). Even switching groups. Right now I'm looking at !AltME with 
several blank messages - only the dates appear normally.