1

Contents of robots.txt for SEO implementation

view story
linux-howto

http://askubuntu.com – Below is a piece of content from robots.txt file which governs that which search engine should follow the site and in which manner. Basically the SEO implementation: User-agent: MSNPTC Crawl-delay: 1 Disallow: /norob/ Can somebody provide clear and exact explain for the last two lines in this fragment. P.S. i am not allowed to post questions on stackoverflow, that's why i posted it here. If you think that it's not relevant here please migrate it and do the needful. (HowTos)

missing's picture
Submitted by missing on 10/16/2012

Stories similar to Contents of robots.txt for SEO implementation

Valid robots.txt? [closed] 25 weeks 4 days ago

I am waiting for Google to crawl my site and display the results in search. Is my robots.txt alright and will it let google, bing etc crawl my site? Thanks!

Wget JPG not working 31 weeks 6 days ago

Hi, I am new to Unix. Actually, I am using Terminal from MAC OSX Snow Leopard. I have installed XCode and wget 1.14 today.

I just want to download all of the jpg files from a specific blog (from blogspot).

Where to put robots.txt? 38 weeks 1 day ago

If I want my main website to on search engines, but none of the subdomains to be, should I just put the "disallow all" robots.txt in the directories of the subdomains? If I do, will my main domain still be crawlable?

Hi everyone!

I'm not sure if this is the best forum to post this in because I'm asking for the opposite of SEO...

I know can create ONE robots.txt file for all domains on an Apache server*, but I want to append to each domain's (if pre-existing) robots.txt. I want some general rules in place for all domains, but I need to allow different domains to have their unique rules.

Is there a way to accomplish this?

(*In my case Apache 2.2.x)

As an example - http://aok.heavengames.com/cgi-bin/aokcgi/display.cgi?action=t&fn=22.

I found a way to get through the robots.txt restrictions, but even then, it just downloads a binary file that's unreadable by anything.

My DialogFragment throws ClassCastException if called from Fragment, while it is working normally if called from an Activity. I have already looked at few other questions with similar problem and basically those are related to imports, but I haven't been able to solve it in my implementation. Here is my implementation for DialogFragment.

I am writing an application that uses NFC to read some data stored on it. My application uses Fragments and Fragment don't come with onNewIntent() method. Since, the data I am reading is done with my separate class which handles NFC related operation, the only thing I need to do is update the TextView inside the Fragment.

Nikki and the Robots 0.4 1 year 35 weeks ago

Nikki and the Robots 0.4 got released and you can view a video showing the changes at Vimeo or YouTube: We released Nikki and the Robots 0.4.0 (0.4.1.1 actually), which you can download here or via in-game updater. The new features are: New robot: Canon! It shoots canon balls that disappear after 10 seconds Players [...]