ickplant@lemmy.world to Fuck AI@lemmy.world · 4 days agoLOLlemmy.worldimagemessage-square98linkfedilinkarrow-up1817arrow-down14file-text
arrow-up1813arrow-down1imageLOLlemmy.worldickplant@lemmy.world to Fuck AI@lemmy.world · 4 days agomessage-square98linkfedilinkfile-text
His article about it: https://www.bbc.com/future/article/20260218-i-hacked-chatgpt-and-googles-ai-and-it-only-took-20-minutes
minus-squarecron@feddit.orglinkfedilinkarrow-up3·4 days agoDoes robots.txt really work in the fediverse? At least on lemmy, the content can be retrieved on different hosts, all of which have different robots.txt files. Unless it is somehow “baked” into the protocol.
minus-squarepkjqpg1h@lemmy.ziplinkfedilinkEnglisharrow-up2·3 days agoMajor search engines respect robots.txt, but as you said some instances allow them but this is not a scalable way
Does robots.txt really work in the fediverse? At least on lemmy, the content can be retrieved on different hosts, all of which have different robots.txt files. Unless it is somehow “baked” into the protocol.
Major search engines respect robots.txt, but as you said some instances allow them but this is not a scalable way