1. All items from the envato market should now be uploaded with price in $ * 10 = credit price.

    This is just because my server rent is now more expensive.

    Thank you for understanding.

Dismiss Notice

Hello Guest

What happend when you click on ads...

  1. It helps us to stay online
  2. You will get credits if you click on them
  3. We can buy new scripts
  4. Getting a better server in the future

AdSense Crawler

Discussion in 'Trash and/or useless posts' started by EnvatoBot, Nov 9, 2018.

Thread Status:
Not open for further replies.
  1. EnvatoBot

    Member+

    Joined:
    Jan 4, 2016
    Messages:
    6,907
    Likes Received:
    4
    I'm seeing a lot of crawler errors as well, in relation to the pages that require a login. Features like /discover/unread and private forums mostly. Anyway, I don't want crawlers accessing our data behind logins, so, it looks like I need a directive to do this. Here's what Google says about it http://support.google.com/webmasters/answer/6062608?visit_id=636773196575520711-1843243888&rd=1: "You should not use robots.txt as a means to hide your web pages from Google Search results. This is because, if other pages point to your page with descriptive text, your page could still be indexed without visiting the page. If you want to block your page from search results, use another method such as password protection or a noindex directive." I then click on the noindex directive and get this page: http://developers.google.com/search/reference/robots_meta_tag It looks like the best way to hide pages is through a HTTP noindex directive. Do we have more information on how to add this 'noindex directive' to the areas we want to hide from crawlers? PS: I already have these pages in my robots.txt file, but, it seems Google is showing them anyway.

    Continue reading...
     
Thread Status:
Not open for further replies.

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice
Underground Toplist