Maybe. 2nd idea I’ve got is that if no one is replying after say 24hrs and something like 75-80% of your posts are as such and you have at least 100 such posts, you get added to the list?
Main concern I see about something like this is false positives and how someone real could end up getting blocked.
I definitely want to think on this some more but it might have some legs.
…I wonder if there’s a programmatic way to detect these bots? Some sort of analysis on their posting behavior?
If they’re playing nice they’ll have the bot flag checked in their profile, and then maybe build a list of any bot that creates posts? As most of the “good” bots just reply to comments? Anyway just thinking out loud. But I’m thinking I could easily add a public API to my search engine that just returns a list of “posting bots”…
The target folder may be quite large. You can look at the dependencies for my project but my end binary is only a few MB.
I don’t see anything wrong initially. Can you also test with something like postman? It’ll allow you to control the headers etc… that you send.
You can also compare to what I have here and see if that helps: https://github.com/marsara9/lemmy-search/blob/b6c88355aba49abca52862473650526821ee165a/server/src/api/lemmy/fetcher.rs#L123
Thanks for the callout. For anyone using that site, I have a list of bugs I’m hoping to get fixed in the next couple days that should fix a lot of the major issues (broken links, etc…).
Problem is finding the difference between repost bots and bots that are helpful like automod and link redirectors.