A webmaster robots.txt personal experience of pain

why did this happen? The reason is that it robots do when the site is completely shielding love Shanghai spiders crawl. We know for a fact! Is that when you wrote the rules; shielding when spiders love Shanghai, love can be found in Shanghai would grab. The reason is very simple, love Shanghai impossible as intelligent. So he will shield you spider to grab. But you also don’t panic! So over time the love of Shanghai will have been caught, but the page shielding page of your robots file; he will be removed from the index database, love Shanghai will also have a process of discovery; to correct and error correction.

at that time, I remember he said a word, why is love Shanghai to shield spider? Because the site just do; and he also learned Shanghai dragon, so also heard Shanghai dragon teacher told some of this content, understand their own to do a new temporary shielding website spider; ready to submit to the search engines and other sites, and the robots file repair, allowing the love Shanghai grab; he may forget the key step to forget. He worked hard to do the site and cause the chain; love Shanghai delete included page, that time is very painful; his whole person is very depressed mood.

today is a familiar things robot file, we believe that this not strange! But I found many webmaster friends forget the importance of this file, and set the time is free. This kind of understanding is a mistake. I own on the side of lectures while self-study, today I met this problem; then, I consulted a Shanghai dragon teacher, I asked him: the teacher robots important? At that time, the teacher of my sincere words and earnest wishes to say; robots can not only achieve your website; it can ruin your site. So it is very important.

now! Come and we talk about robots in some of our site in Shanghai; research center of a dragon and I are good friends, once his website has included thousands, suddenly one afternoon included for 0. He was confused, because the site included no; no flow, working hard for more than a month. In the end they found all in vain. He was downhearted; went to the group chat, I asked everywhere; the site was K, seeking solutions! Worried, then, there are a lot of people to help him; check the reason, after a long period of time to check, we found the reason is his website robots shielding love Shanghai spiders to crawl the web page.

today, I experience through this friend is also to pay attention to the problem of robots? Is very important, please don’t neglect some experience by themselves; talk about my robots, I remember the site in the new time. Is not my own robots this file >

Leave a Reply

Your email address will not be published. Required fields are marked *