Printable Version of Topic

Click here to view this topic in its original format

_ Forum Information (Readme!) _ Google.com, Yahoo.com etc reading this board?

Posted by: blissyu2

I have bookmarked the main page, and when I click on it sometimes it says that there are e.g. 1 member and 2 guests reading, and it says "Blissyu2, Google.com, Yahoo.com". Why does it say this? What does it mean? Does this mean that Google.com and Yahoo.com spiders are reading the forum or someone is doing a search and finding this page via their caches or what? Does anyone have any idea what it means?

Posted by: Blu Aardvark

I wasn't aware that non-admins could view that, but yes, that's exactly what that means. A search engine bot is indexing the forum.

Google.com got here first. About a week later, Yahoo showed up. And MSN just now started poking around. But yeah, it's just shows the bot activities.

Posted by: Selina

Yeah, it's search engine spiders

Posted by: blissyu2

I wouldn't mind it so much if we could click on them. There was a new one there today. Wisenut.com. I'd really like it if they were clickable, so we could spy on them back. http://www.wisenut.com/

Posted by: Selina













































Bot NameHitsLast Hit
Google.com
1497
11th 3 06, 5:19pm
Yahoo.com
814
11th 3 06, 4:28pm
Wisenut.com
11
11th 3 06, 3:11pm
Cobion.com
1
11th 3 06, 10:28am
MSN.com
40
10th 3 06, 11:18am
Nextopia.com
1
9th 3 06, 5:57pm
Ask Jeeves
2
7th 3 06, 3:28am
Google.com
754
5th 3 06, 7:03pm

Posted by: blissyu2

If Googlebot is hitting us 1497+754 times (2 bots?) = 2251, then how come we seem to be banned from Google? What does this mean?

Posted by: Selina

No idea

http://google.com/search?q=site:wikipediareview.com
http://google.com/search?q=site:wikipediareview.proboards78.com

Maybe it takes a while for it to organise the pages in its database once it's spidered them or something

as for this, http://google.com/search?q=wikipedia+review , I think it's probably because we're a new site we're ranked lower than wikipedia and the other sites

http://google.com/search?q=wikipediareview and http://google.com/search?q="wikipedia+review" still show us tho

Posted by: blissyu2

Yes. I think that we want to be searchable on Google. It'd be good to advertise this site and such and for people who are interested in certain topics to be able to find one that is relevant to them.

Of course, we don't want Google Adsense, or any ads on the pages, but I don't think that the two are connected.

Posted by: Lir

I'd rather not see any ads here; as far as advertising this site, I've linked to it from the top of my kapitalism.net article -- and I think most anyone looking for criticism of wikipedia eventually finds my article. Don't try to rush publicity, with time you'll get better results on search engines.

Posted by: Selina

-- http://wikipediareview.com/indexsitemap.xml.gz
----> http://wikipediareview.com/topicsitemap.xml.gz
----> http://wikipediareview.com/forumsitemap.xml.gz

Those have both been up for a while and the site's been registered on Google Sitemaps for a while

The files also automatically get updated via the forum

Maybe Sitemaps is what's actually causing the problem, I dunno

Posted by: blissyu2

A big time no on Google Adsense, or any advertising on here. It lowers the quality of a site if it has ads. My web page used to be ad-free but now is spammed with ads, so much so that silly Malber suggested it had a virus on it. We don't want the same situation here. If we are going to advertise, do it legitimately. Sell coffee mugs etc if we want to, but no spam ads.

When I clicked on Selina's links, I got "montly". Monthly has an "h" in it. Is that what is causing the problems?

Posted by: Selina

hmmm I think I'd probably be better off just letting the google bot crawl then and removing sitemaps - since the mod for the forum that makes the sitemap obviously doesnt work properly and it'd be a waste of time to add a link to every new topic every time there is one

Posted by: God of War

http://search.yahoo.com/search?p=wikipedia+review&sm=Yahoo%21+Search&fr=FP-tab-web-t&toggle=1&cop=&ei=UTF-8

It seems that we are now number 10 on the Yahoo search page.

Posted by: blissyu2

QUOTE(God of War @ Mon 13th March 2006, 2:33am) *

http://search.yahoo.com/search?p=wikipedia+review&sm=Yahoo%21+Search&fr=FP-tab-web-t&toggle=1&cop=&ei=UTF-8

It seems that we are now number 10 on the Yahoo search page.


Number 5 now, with the old forum at number 4.

We should be number 1 though, at least in searches for the actual site.

Posted by: Selina

Yeah, I think google ranks older sites higher basically - makes sense, since it stops people creating tonnes of spam subdomains and linking each other etc - I've noticed this on google don't know if anyone else has, some things seem to automatically make subdomains with certain keywords and get listed in google but they're nearly always at the very end of searches (unless there's no other results)

Posted by: Blu Aardvark

QUOTE(qwerty @ Sun 12th March 2006, 8:11am) *

No, I guess it's not because of that. I guess it's because of the format of the files. On the Google Sitemaps help it says clearly that the feed file has to be RSS 2.0 if it's RSS. The file also has to have <link> and <pubDate> tags.

The feed isn't RSS, however. It's an XML document using the http://www.google.com/webmasters/sitemaps/docs/en/protocol.html.

QUOTE(qwerty @ Sun 12th March 2006, 8:11am) *
And I don't know if the .gz extension at the end is also affecting the thing.

Possibly. There shouldn't have been a .gz extension present, because gzip'ing of the sitemap documents was disabled. I've enabled that now; we'll see if that helps.

Editted to add Do we have a google sitemaps account? We need one.

Posted by: Selina

We have a google sitemaps account yeah. I disabled sitemapping though to see how it goes - I think quite possibly googlebot will crawl the pages better on it's own than relying on that mod which doesn't make proper sitemap syntax files - when the sitemap was enabled it was set up to read the files and has an account, yeah

Posted by: blissyu2

Well, while popularity isn't everything, I think that we do want to be listed on search engines. We want people who are interested in talking about the pitfalls of Wikipedia to know that there is somewhere that they can go. And this should include journalists and more well known critics as well as just regular users. Ultimately, we don't want this to be mostly filled with people trying to destroy the forum. I don't know if it will always be 50/50, but long term I'd hope that we are mainly working towards the criticism of Wikipedia element rather than criticism of criticism of Wikipedia! LOL.

Posted by: Blu Aardvark

QUOTE(Selina @ Tue 14th March 2006, 1:54am) *

We have a google sitemaps account yeah. I disabled sitemapping though to see how it goes - I think quite possibly googlebot will crawl the pages better on it's own than relying on that mod which doesn't make proper sitemap syntax files - when the sitemap was enabled it was set up to read the files and has an account, yeah


When the sitemap was set up, it didn't list any forums to index, and therefore, none were indexed. Since that has been changed (you have to list them by forum id, comma seperated, in the ACP options for Sitemapping), google is indexing us. Whether or not it was the sitemap config, or something else, I dunno. But we're there now.

http://www.google.com/search?q=site:wikipediareview.com