• snooggums@lemmy.world
    link
    fedilink
    English
    arrow-up
    57
    arrow-down
    2
    ·
    3 days ago

    AI crawlers tend to overwhelm websites by doing the least efficient scraping of data possible, basically DDOSing a huge portion of the internet. Perplexity already scraped the net for training data and is now hammering it inefficiently for searches.

    Cloudflare is just trying to keep the bots from overwhelming everything.