[REBOL] Re: Library & smart docs => was:{Re: Re: Parsing comment}
From: gchiu:compkarori at: 27-Sep-2002 11:34
On Thu, 26 Sep 2002 23:21:54 +0200
Christian Langreiter <[chris--langreiter--com]> wrote:
>> Hmm. I thought that there was not supposed to be a
>>direct
>> link anywhere to that archive to prevent spiders getting
>> to it ... as email addresses are visible on those pages.
>
>Well, spiders don't get to sites if they observe
>robots.txt, which
>spam address harvesters most certainly don't.
We had a discussion before about the rebol.org archive,
and Jeff removed all links to the archive. To get the
address, you have to use the link on the RT rebsite. So
even a spam address harvesting robot should not get there.
But I guess google had it in it's database prior to that
happening.
>> I think that all dynamically created sites tend to be
>> invisible to search engines. Zope is an example.
>
>Not exactly. Search engines have avoided URLs with query
>strings for a
>long time, but how would they distinguish static from
>dynamic content?
>They cannot, it's all bits to them (and us, for that
>matter ;-).
In practise they tend to be invisible.
Here's an article from Paul Graham making that point
http://www.paulgraham.com/mistakes.html
and in my Zope site, I have this in my robots.txt to make
me invisible!
User-agent: *
Disallow: /Shopping/ # This is an infinite virtual URL
space
--
Graham Chiu
