Working in SEO for the past 7 years has shown me a lot of what the discipline has to offer and made me appreciate the non-commercial Web a lot more. As we’ve all seen, the bigger search engines cater to a part of the Web that is all about selling. Regular verbs and nouns bring up search results for companies unless you add further context to your query. That makes sense in a lot of cases but it still makes me a little sad.
But Google’s sped up proliferation of AI in all of its tools means people are looking to alternative search engines (not you, Bing). A popular alternative is Kagi which I’ve got a higher percentage of traffic from over the last year. You can use it for free for a set amount of searches or pay a subscription fee to get unlimited searches. There’s also smaller search engines that don’t rely on a massive index or crawlers such as Wiby and Marginalia.
I’ve spoken about search engines and how there are heavy caveats and trade-offs to using any now but going back to the main point: what would “optimisation” for smaller search engines look like? What sort of algorithms or information science techniques do they use? I’m trying to be careful with how I approach these questions because these were asked 20–30 years ago as people were trying to exploit the Web 1.0 and 2.0 search engines to try and profit. I’m not suggesting that we go back to cloaking and keyword stuffing but if these smaller search engines focus more on browsing and discoverability, maybe there are new techniques we can learn.
“Optimisation” might not be the right word or approach. We’re writing and having fun with it so there’s no perfect way of doing that. But I love the inner workings of how search and the various NLP techniques at play in a search engine. I think mixing the two together is an interesting concept and it doesn’t have to involve ads, generative AI, or a means to profit.
This is a very half-baked idea that I only conjured this afternoon so if want to delve into it more with me, feel free to email!
Buy me a coffee or a pledge!
