The death of useful Google searching

[bear]

I've been noticing a change lately in Google, and it's not for the best. I'd been looking for a particular type and design whistling teakettle for my wife, and when I typed the full description into Google, receive about 4 pages of results...none of which were relevant.
The majority of these were sites that placed really well in Google, but had names like "used-computers-and-hardware.com". Curious, I followed all of these results, and 8 out of 10 went to Amazon pages, redirected through this person's site. They get credit for the purchases of anyone that buys after landing there. The other 2 out of 10 (no exaggeration here...actual results) were the same sort of thing, but linked to Ebay instead.

So, for all of you that are working so hard to get good Google rankings, it may be in vain eventually, since all these affiliate folks have long since beat you to it. This just proves that the Google algorythm for the ranking and listing pages and sites is fundamentally flawed.

How long, I wonder, before the general public wakes up and realizes the same thing, and a major shakeup takes place at Google Inc?
Revolution, anyone?

[wera]

I've been noticing a change lately in Google, and it's not for the best. I'd been looking for a particular type and design whistling teakettle for my wife, and when I typed the full description into Google, receive about 4 pages of results...none of which were relevant.
The majority of these were sites that placed really well in Google, but had names like "used-computers-and-hardware.com". Curious, I followed all of these results, and 8 out of 10 went to Amazon pages, redirected through this person's site. They get credit for the purchases of anyone that buys after landing there. The other 2 out of 10 (no exaggeration here...actual results) were the same sort of thing, but linked to Ebay instead.

So, for all of you that are working so hard to get good Google rankings, it may be in vain eventually, since all these affiliate folks have long since beat you to it. This just proves that the Google algorythm for the ranking and listing pages and sites is fundamentally flawed.

How long, I wonder, before the general public wakes up and realizes the same thing, and a major shakeup takes place at Google Inc?
Revolution, anyone?
that is definatley why i use yahoo it always gave me better results

[Zykaz]

Actually, what I've been developing for the past week or two (for the previous owner of this forum, actually) is http://www.crawlr.net - which he pretty much wants to be a Google clone. For one programmer on limited resources, this isn't so easy, but it's coming along to an extent.

If you do visit it, keep in mind that it's still in the works, and by no means has all of the bugs worked out. Some things aren't even working at all.

[wera]

Actually, what I've been developing for the past week or two (for the previous owner of this forum, actually) is http://www.crawlr.net - which he pretty much wants to be a Google clone. For one programmer on limited resources, this isn't so easy, but it's coming along to an extent.

If you do visit it, keep in mind that it's still in the works, and by no means has all of the bugs worked out. Some things aren't even working at all.
ok now i hate to be a bad vibe m,an but.................it..uhhhdont get mad at me............. sucks there i saied it

[bear]

Interesting...has it spidered anything at all, or is it just the front end so far? Will you be grabbing results from other engines, or writing a spider of your own?

I wouldn't agree that "it sucks", it's just new, and there isn't much to see yet.

[Zykaz]

ok now i hate to be a bad vibe m,an but.................it..uhhhdont get mad at me............. sucks there i saied it

No worries. I got a template and a list of features, and have only been doing the programming work. And as I said, it's not near done.

Interesting...has it spidered anything at all, or is it just the front end so far? Will you be grabbing results from other engines, or writing a spider of your own?

As of now, it spiders the meta information of sites on a localized level. Sites are entered into the database manually through the "add a site" link, and those are crawled. I'm making the spider myself, and eventually it will be able to crawl across the internet freely, following links within webpages, storing every bit of code of the pages in the database, and allowing the user to search through those pages. I'm also working on an aspect I call 'smart search', which will contain many features and back-end programming to make the search faster and more efficient.

[bear]

Sounds as thought it's going to need an enormous amount of storage and processing power to do this on any real scale. Will you be clustering servers eventually? I'd have to think that no one server could deal with this once it grew a bit large...

[Zykaz]

Heh, yeah, that's why it currently does it on a local scale, and only runs through the internal pages of sites, rather than branching outwards to external sites. I'm not quite sure what I'll do about the space required, as of yet.

Processing power won't be much of a big deal, as I'll be setting the spider to crawl sites at a slower pace. It may take a while, but that's better than frying the CPU and slowing the server to a halt. In the administration area, the spider also has different crawling options - such as crawling newly added sites, old sites, sites that were found to be down, etc. So, a full crawl won't be done all too often.

Speaking of slowing the server, when I was developing the spider I set an inifinite loop by accident, and didn't realize it. I ran off for a bit before testing it, and when I got back the account was closed and I had an email from the owner saying that I was killing the server with something in my code. :p

[bear]

Never a fun email to receive..:D

I was referring to the user searches regarding processing power. It's one thing to category sites, but the retrieval of this info would become an issue once you began having numerous simultaneous users, no? How to you plan to scale the application/database once you get more than a few users at a time?

[Zykaz]

Retrieving and searching through database information isn't too much of a strain on the processor for large amounts of users. Many, many sites already do this (well, retrieving and then printing/somehow using database information, rather than searching through).

[bear]

Ah, perhaps I misunderstood then. I'd assumed that querying and sorting for large amounts of users would be a burden on the server...

[Zykaz]

Well, this forum (along with others) does it every time a person comes to the site, and it seems to be managing.

[wera]

if you are worried about the server get a free invison free or something and have theam host it lol not eating yore server just theres ;)

[bear]

What, me worry? ;)
I'm not concerned about my server...and I don't think Invisionfree would host a Vbulletin forum...:D

[wera]

What, me worry? ;)
I'm not concerned about my server...and I don't think Invisionfree would host a Vbulletin forum...:D well what about vbulliten :tongue1: