My niche little site, http://golfcourse.wiki seems to be very popular with AI bots. They basically become most of my traffic. Most of them follow robots.txt, and that's nice and all, but they are costing me non-trivial amounts of money.
I don't want to block most search engines. I don't want to block legitimate institutions like archive.org. Is there a whitelist that I could crib instead of pretty much having to update my robots file every damn day?
Comments URL: https://news.ycombinator.com/item?id=42861047
Points: 15
# Comments: 8
Login to add comment
Other posts in this group
Article URL: https://mitchhorowitz.substack.com/p/the-crisis-of-professional-skepticism
Comments URL
Hi HN. I built a prototype AI physics tutor that can interpret, draw, and edit free body diagrams.
Lately I've been transfixed with generating diagrams with LLMs. If you pipe generated JSON thro
This is software for my headless, PC based oscilloscope, which is controlled entirely via commands similar to the Vim text editor. I built this because I liked the idea of headless oscilloscopes;
Article URL: https://pixieditor.net/blog/2025/07/30/20-release/
