The other night, I couldn’t sleep… as I was lying awake, the phrase “…when you look long into the abyss, the abyss also looks back at you” (F. Nietzsche, Beyond Good and Evil) popped into my head. For some reason, I started thinking about “searching” - search engine ranking and evaluation methods… how its development somewhat mirrors two forms of epistemology (how we “know” something), one old, one new. Stick with me now…

The traditional way of thinking about how we “know” something, or relate to it, is to think of single objects interacting in space: I am in the world, interacting with object x. I know it as separate from me and able to be independently manipulated - like me looking at an apple on a table and noting that there are 3 things in the room:

1. “self”
2. “apple”
3. “table”

This is the type of thinking modern science is based upon. (Roughly speaking, what I’m talking about is empiricism.) Now… think about old school search engines:
1. data driven - 90s model;
2. on-page factors like keyword density, TITLE and META tags are deciding search engine rankings

The search engine assesses websites as they exist in space, categorizes them by tacit, calculable qualities and serves them as non-contextual results pages to the searcher. Yes, this is very comfortable feeling, but is it the best way to deliver quality results… to know what the searcher wants?

Take the next step in thinking about how we know something. Over time, pragmatists and phenomenalists said “Wait a minute, I never know something in space… there are always opinions, environments, relationships, sensations and gradations that aren’t independent.” When I am in a room with an apple on a table, I know those things only by how I perceive them: the feel of the table, how I see the color, the light in the room, my past experience with bad apples, etc. Everything we want to know, need to know is based upon the viewer him or herself bringing influences “to the table” and seeing that apple through particular eyes.

Still with me? Now think about where (I feel) search engines are going… starting with Google’s current algorithm. The traditional way of searching… of knowing the web is acknowledged, but is expanded by:
1. consideration of off-page (”environmental”?) factors like incoming link quality and IP neighborhood in addition to on-page factors;
2. increased importance of website authority or community opinion as determined by quality & number of incoming links;
3. contextual content analysis is used for assessment of quality - more than just keyword density, now we’re talking about grammar and site “theme”;
4. profile of site is used to judge reputation and, therefore, authority - a timeline of site development + ownership information is used to assess (to “know”) the accuracy of results

Sounds a lot like pragmatism and phenomenalism to me; we’re no longer viewing web pages and chunks of information in a vacuum. The goal is to gather and view information in context with history and opinion.

So what’s next??? I definitely think we’ll see the other search engines move more toward the Google model but I also see more contextual & “outward reaching” methods being added to that winning model:
1. social data (like tagging) will be used to help rank results, giving those results increased weight to community opinion;
2. searcher history will increasingly determine what results you are served - data collected through profiles, desktop search, mapping, referrer, etc. will be used to tell what you’re most likely looking for;
3. sites based in and targeting the locality of the searcher will increasingly dominate results, bringing the real world into the mix as the ultimate way of helping the searcher “get to know” what they’re searching for

Yes, I know, all of this is opinion and not as well cross referenced as I’d like it to be but it does paint a general picture of web search mirroring how the West has grown to know and understand the world. While everyone is comfortable with this traditional model of knowing things (and searching the web) the more progressive ways of knowing and understanding seem to be coming to bear on search engine results. It appears the ways of the pragmatists, the phenomenalists and Nietzsche are being used to influence the accuracy of results — one where we may be obsessed with looking into the search abyss, but real answers are found when the abyss looks back into us.

[Note: I originally posted this on one of my blogs, CaseDetails.com]