After striking deals with Google and OpenAI, Reddit CEO Steve Huffman is calling on Microsoft and others to pay if they want to continue scraping the site’s data.
“Without these agreements, we don’t have any say or knowledge of how our data is displayed and what it’s used for, which has put us in a position now of blocking folks who haven’t been willing to come to terms with how we’d like our data to be used or not used,” Huffman said in an interview this week. He specifically named Microsoft, Anthropic, and Perplexity for refusing to negotiate, saying it has been “a real pain in the ass to block these companies.”
Reddit has been escalating its fight against crawlers in recent months. At the beginning of July, its robots.txt file was updated to block web crawlers it doesn’t have agreements with. Then people began noticing that Reddit results were only visible in Google results — where Reddit is paid for its data to be shown — and not other search engines like Bing.
This comemt made me search my reddit name and to my supprise I found comments I wrote years ago. I logged in but nothing showed in my account. I could however delete them when I re found while logg3d in
They only show you a limited number of results. You need to do a GDPR export and delete from that list: https://github.com/mbirth/reddit-cleaner