Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Did I do this wrong or is Yahoo fecking ignorant and disrespectful?
03-22-2008, 04:51 AM,
#1
Did I do this wrong or is Yahoo fecking ignorant and disrespectful?
http://conspiracycentral.info/robots.txt
Reply
03-22-2008, 04:54 AM,
#2
Did I do this wrong or is Yahoo fecking ignorant and disrespectful?
:o
Reply
03-22-2008, 04:56 AM,
#3
Did I do this wrong or is Yahoo fecking ignorant and disrespectful?
Huh?
Reply
03-22-2008, 04:57 AM,
#4
Did I do this wrong or is Yahoo fecking ignorant and disrespectful?
AFAIK, the robots.txt in my example means ALL AGENTS need to fuck off. Yahoo just wont go away.

I just sent their feedback link this...


I'm the owner/operator of conspiracycentral.info.
I have a robots.txt file that yahoo seems not to respect.
Does yahoo purposely ignore robots.txt?
I don't want your stupid bots sucking my bandwidth and CPU time.
Please GO AWAY!!!
Reply
03-22-2008, 04:59 AM,
#5
Did I do this wrong or is Yahoo fecking ignorant and disrespectful?
:o
Reply
03-22-2008, 05:00 AM,
#6
Did I do this wrong or is Yahoo fecking ignorant and disrespectful?
Quote:Huh?

A robots.txt file is a file "honourable" bots should seek before indexing a site. IOW, it's a way of asking a crawler to F/O&D. There's no RFC that says it has to be honoured. It's voluntary.

My question is did I make the file wrong or is yahoo just being jerks?
Reply
03-22-2008, 05:23 AM,
#7
Did I do this wrong or is Yahoo fecking ignorant and disrespectful?
Yahoo's just being jerks I think, it shouldn't be probing the site, maybe they are using the probe for other means...
Reply
03-22-2008, 05:26 AM,
#8
Did I do this wrong or is Yahoo fecking ignorant and disrespectful?
Never got into robot.txt except to find the free web page do
not let bots see you page if that is in the root.

Inktomi shows up in peerguardian when I use a link:

http://search.yahoo.com/search?p=teslaandlyne


generated by a html forms window.

So yahoo is using Inktomi... go to Inktomi or wait till Yahoo finds
out who is doing their searches.

Lyne said government suppression and Yahoo ignored his site,
now he shows up number one.
Reply
03-22-2008, 05:27 AM,
#9
Did I do this wrong or is Yahoo fecking ignorant and disrespectful?
Here's some decent info:

How to keep bad robots, spiders and web crawlers away
http://www.fleiner.com/bots/

Bots vs Browsers - database of 154,353 user agents and growing
http://www.botsvsbrowsers.com/

The Web Robot Pages is an information resource dedicated to web robots.
http://www.robotstxt.org (where you prolly got the robots.txt info)

Block Bad Bots Globally In Apache
http://blog.3dpixel.net/block-bad-bots-globally-in-apache/

Bot-trap - A Bad Web-Robot Blocker
http://danielwebb.us/software/bot-trap/

we're beginning to get our share of bad bots over at NASWS
&its just like.. doood ya get the best barrels ever dood..
its just like.. ya pull in and ya just get spit right out of em...
ya just drop in n just smack the lip.. whabap.. drop down..
zibbaaaahhhahahah..
n then after that.. ya drop in.. ride the barrel..
and get pitted.. sooo pitted like that&
- surfer dood

Northern Alberta Surface Water Study
check it out: www.nasws.ca
Reply
03-23-2008, 12:30 PM,
#10
Did I do this wrong or is Yahoo fecking ignorant and disrespectful?
Yours should work

http://help.yahoo.com/l/us/yahoo/search/we...r/slurp-02.html
http://www.ysearchblog.com/archives/000372.html
http://www.archive.org/about/exclude.php
&Having raised the earth's temperature 1 degree Fahrenheit in the last three decades, we're facing another increase of 4 degrees over the next century. That would imply changes that constitute practically a different planet. It's not something we can adapt to. We can't let it go on another 10 years like this.& - NASA's Goddard Space Institute Director James Hansen

ConspiracyCentral Tracker Style Mod
Reply
03-23-2008, 01:00 PM,
#11
Did I do this wrong or is Yahoo fecking ignorant and disrespectful?
Maybe it's in the wrong directory. It should be in the root directory, but also in your document root. What I want to say is: maybe it's in a directory that the bot does not 'see'.

When I resolve the forum IP there is no robots.txt in the root, and maybe that's where the bot looks at:
http://69.89.25.151/robots.txt
Reply
03-23-2008, 04:38 PM,
#12
Did I do this wrong or is Yahoo fecking ignorant and disrespectful?
Quote:Maybe it's in the wrong directory. It should be in the root directory, but also in your document root. What I want to say is: maybe it's in a directory that the bot does not 'see'.

When I resolve the forum IP there is no robots.txt in the root, and maybe that's where the bot looks at:
http://69.89.25.151/robots.txt

Sure, it's in the root.. the forum sits in the root fwiw.
That's why it's http://conspiracycentral.info/index.php? not http://conspiracycentral.info/forum/index.php?

Resolving a domain to an IP wont get you anywhere. The IP you looked for the file on hosts dozens of other sites. How is the server to know who's site you want robots.txt from? Your approach is flawed.
Reply
03-23-2008, 10:43 PM,
#13
Did I do this wrong or is Yahoo fecking ignorant and disrespectful?
Does it hurt to put it in as many places as possible?

Dumb question... but thats the type of solution I do sometimes.
Reply
03-23-2008, 11:14 PM,
#14
Did I do this wrong or is Yahoo fecking ignorant and disrespectful?
No it doesn't hurt, but I only need one for the forum.
Reply


Possibly Related Threads...
Thread Author Replies Views Last Post
  Wrong Color in the Wrong Place psilocybin 2 487 05-20-2013, 03:50 PM
Last Post: SiLVa
  English girl barred from Government job...because she is wrong kind of white Guest 1 511 08-25-2009, 07:33 PM
Last Post: ---
  What's wrong with Transhumanism Anyway? --- 37 3,942 07-26-2009, 04:29 PM
Last Post: ---
  Ignorant twats drummer 2 675 07-08-2009, 12:05 AM
Last Post: ---
  Is it me or is there something wrong with these sites? mastermg 7 1,051 02-24-2008, 09:07 AM
Last Post: harmonius
  When cloning goes wrong blove8. 3 997 02-17-2008, 09:05 PM
Last Post: Dexter
  News Corp 'in secret Yahoo talks' TriWooOx 0 407 02-15-2008, 11:11 AM
Last Post: TriWooOx
  Snail Mail Address is Wrong icosaface 0 382 10-19-2007, 02:05 AM
Last Post: icosaface
  But The Bible Says homosexuality Is Wrong ? drummer 11 1,407 10-11-2007, 04:49 AM
Last Post: hilly7
  You're Wrong byrd 23 1,898 08-02-2007, 01:44 AM
Last Post: MC David

Forum Jump:


Users browsing this thread: 1 Guest(s)