Hi,
I recommend a simple improvement that will avoid a lot of CPU overhead when robots access our wiki websites.
At the "sitemap" page include rel="nofollow" in all of namespace links, because robots access each of them thinking to be different pages, causing overhead when there are too many pages in the root namespace.
The situation become worse because each page has a link to index page where the current page name is passed by param. Then, there is one different URL to index page for each page of our wiki and for each of these URLs we have a different link for each namespace.
The robots follow many URLs to read the same data.
Then, I recommend to add rel="nofollow" in all "sitemap" link, except at the main page of the wiki and recommend include rel="nofollow" in all of namespace links at the "sitemap" page.
Example:
Considering the "aves" page, the link is...
http://www.wikiaves.com.br/aves?do=index
for each namespace...
http://www.wikiaves.com.br/aves?idx=areas
http://www.wikiaves.com.br/aves?idx=artigos
....
It can be avoid including some lines at robot.txt too.
Disallow: /*?idx=*
Disallow: /*?do=index
Kind Regards,
Reinaldo César Guedes
WikiAves.com.br