Category: Search

If you’re reading this, chances are you’ve seen our robot visiting your site while looking through your server logs. Our software obeys robots.txt files and robot META tags in HTML. These are the standard mechanisms for webmasters to tell web robots which portions of a site a robot is welcome to access.

Our software obeys the robots.txt exclusion standard, described at https://www.robotstxt.org/orig.html. Our crawlers should respond to the agent name “Jambot”. Thus to ban all Jambot crawlers from your site, place the following in your robots.txt file:

User-agent: Jambot
Disallow: /

Webmasters/Robots META

If you do not have permission to edit the /robots.txt file on your server, you can still tell robots not to index your pages or follow your links. The standard mechanism for this is the robots META tag, as described at https://www.robotstxt.org/meta.html.

If your site has problems or questions about the Jambot crawler, please contact us through our contact form here.

Jeremy Stoppelman: The initial promise of Google was to organize the world’s information. And ultimately that manifested itself in you expecting that the top links, the things that it shows at the top of that page are the best from around the web. The best that the world has to offer. And I could tell you that is not the case. That is not the case anymore.

Instead of doing what’s best for consumers, Stoppelman says Google is doing what’s best for Google.

Jeremy Stoppelman: If I were starting out today, I would have no shot of building Yelp. That opportunity has been closed off by Google and their approach.

Steve Kroft: In what way?

Jeremy Stoppelman: Because if you provide great content in one of these categories that is lucrative to Google, and seen as potentially threatening, they will snuff you out.

https://www.cbsnews.com/news/how-did-google-get-so-big/

Redux (literary term), an adjective meaning “brought back, restored” used in literature, film and video game titles. The term has been adopted by filmmakers to denote a new interpretation of an existing work by the restoration of previously removed material.

What has been removed from traditional search? The very essence of individual endeavor. The character of a highly decentralized, vibrant and diverse internet has been replaced by digital ghettos created by  overly ambitious technocrats with dystopian world views.

We’ll be discussing this more as we go along, rebuilding search.

 

This is where we talk about search.

Thanks for coming by!