[REBOL] Re: Library & smart docs => was:{Re: Re: Parsing comment}
From: laurent:giroud:libertysurf at: 28-Sep-2002 12:21
Friday, September 27, 2002, 1:34:21 AM, Graham Chiu wrote :
> On Thu, 26 Sep 2002 23:21:54 +0200
> Christian Langreiter <[chris--langreiter--com]> wrote:
>>> Hmm. I thought that there was not supposed to be a
>>>direct
>>> link anywhere to that archive to prevent spiders getting
>>> to it ... as email addresses are visible on those pages.
>>
>>Well, spiders don't get to sites if they observe
>>robots.txt, which
>>spam address harvesters most certainly don't.
> We had a discussion before about the rebol.org archive,
> and Jeff removed all links to the archive. To get the
> address, you have to use the link on the RT rebsite. So
> even a spam address harvesting robot should not get there.
> But I guess google had it in it's database prior to that
> happening.
It is also possible that people added a link to this page within their own
website. This way google will access it while harvesting those people's sites.
It's hard to keep a useful page from being referenced from somewhere else ;)
Regards,
Laurent
--
Laurent Giroud
