Thoughts on Search Engines and the Open Web
4 min read

Thoughts on Search Engines and the Open Web

"The medium is the message" is a phrase coined by the Canadian communication thinker Marshall McLuhan. McLuhan proposes that a communication medium itself, not the messages it carries, should be the primary focus of study.

In the context of the open web, it's a fascinating to think how the message (websites & content) is altered to harmonize with its primary communication medium (search engines).

One of its manifestations is the prominence of clickbaity titles in web search. At times, it can feel that the "Best 10 Tips for –insert something– in 2020" or "How to –insert something– in 2020" are the only things left to read on the internet.

Furtunately though, this isn't the case but rather the face of a problem.

This is What Happens When Every Web Page Has a Search Goal

After 20 years of Googling, we learned to optimize our websites and content for web search, similarly to how we optimize our content for amplification. Our focus is the algorithms, likes, comments and shares, instead of our readers and customers. And by focusing on serving search goals with every article we write, we create a web where almost all content is commercialized, and every headline is Associated Press styled titlecase starting with "How to".

There are of course economic imperatives for doing so. Running a website costs money and time – emphasis on the later, and for justifying putting in all these resources, often some degree of monetization is in line. Also, there's nothing wrong with making money on the internet. For internet entrepreneurs (like myself), optimizing a website for web search, curating and contributing information to create helpful content is half passion project and half business goal.

But in the process of SEO and web monetization we've lost something.

Recently I came across to this thought provoking article, about how there are no more blogs left to read:

You could search around and they'd come up even in web search, and you'd find yourself reading someone's blogs. Maybe it was chronicles of their life as they got a job teaching in Japan and how it was leaving America for the first time and all the new things there and skateboarding and meeting people and trying to meet girls, or a photographer working for a while in Minorca or some island off Spain when music hit a rock scene period and all the young people were dressing up in leather and tight jeans and going out dancing to dance rock and writing about his thoughts on where he fit into the scene as he was kind of older but not old, or a compilation of weird and unexplained science and gnostic wisdom, or the things some guy was making out of wood or electronics in his garage, or some Japanese girl who posted pictures of herself looking extremely pink and pneumatic and writing little things with them."When you search for blogs now on Google you see things like 'Top 100 Blogs.' 'How to Make a Successful Blog.' 'Most Powerful 50 Blogs.'

Do you remember this web? I remember reading some random person's blog/ life log about moving from the US to Hong Kong some 10 years ago.

How Did We End Up Here?

Search is a competitive game. You probably have an idea how big is Google's search volume, how many websites are out there competing for traffic, and how competitive the 1st page is. The best think you can do (which I also recommend to my clients), is that every page should have a search goal to optimize it for.

And Google likes it like that. Imagine the resources (and money) involed in crawling and indexing the ever expanding web. The real reason why it's almost impossible now to find a blog that's not on a focused theme, is because if a website doesn't serve a search goal they can't monetize it.
If something doesn't solve a specific problem, offer specific knowledge or advise or contribute to a trend, there is no significant search volume and no ad revenue to be made; only crawling and indexing costs.

This shapes our perception of what web content should look like. Sometime ago I was surprised when I tried to get this on Ad Sense (Google's ad network) and it got disapproved on grounds of low content quality and scrapping (my habit of quoting other people with proper attribution, like above). This made me think, what would Google call popular websites like Daring Fireball, if not of rock bottom content quality.

In the end I didn't want to "fix these issues", and actually decided against using AdSense, Google Analytics or any other Google products on this website. Now it's faster than ever, and also doesn't track you.

Another interesting point from the above article:

There used to seem to be endless search results indexed by Google and the other search engines that were killed by Google. Maybe you'd see 40,000 results for your search. Now it says there are 40,000 but you only get 10 or 20 pages of results you can get to, all basically corporate and lame. You can't get to the 100th page anymore.

This is true, I don't even remember when or how it happened. Most people almost never go beyond page 1, and the majority of clicks go the the top 3 results –if at all when there are rich result features present e.g. knowledge panel, featured snippets etc.

I have the feeling that the reason they limited the search results to 10 pages is to prevent scrapping and reverse engineering, but still, I remember these days where I would visit page 50, or 80 or 100 to just see what's in there.

Where Do We Go from Here?

Unfortunately search engines are today the primary medium within which people interact with websites. I think we need a better way to discover websites that is not a crossbreed between yellow pages and a library.

But until something better comes around, I use bookmarks to save interesting pages that I come across on the web. This is a conscious effort to save interesting sites –many of which are personal websites and blogs, and try to develop a habit of revisiting them occasionally to see what's new.

Social media -and especially Twitter, are also good places to find personal websites, especially if you're part of niche communities (SEO Twitter, Bitcoin Twitter, Naval funs and so on). Nownownow is another good place, it's a directory of personal websites with a now page, a movement started by Derek Sivers.

If you want to share your ideas on website discovery I’m happy to include them here, just DM me @kntoukakis or tweet me if DMs are closed.

Final Thoughts

If you write online, own your distribution and your audience. Companies come and go –AOL, Yahoo, MySpace, and Google one day.
But your own media, built on open protocols and opensource software –a website, or an email list, will endure and be valuable for decades to come.