Hi, I’m building a personal website and I don’t want it to be used to train AI. In my robots.txt file I blocked:

  • ChatGPT-User
  • GPTBot
  • Google-Extended
  • FacebookBot

What bots should I also add? Are there any other ways to block AI bots?

IMPORTANT: I don’t want to block search engine crawlers, only bots that are used to train AI.

  • Luke@lemmy.ml
    571·
    1 年前

    FYI, bots and crawlers can simply ignore your robots.txt entirely. This is probably common knowledge around these parts, but I’ve run into clients at work who thought it was a law or something.

    I do like the idea of intentionally polluting the data robots will see, as suggested by this comment. There’s no reliable way to block them without also blocking humans, so making the crawled data as useless as possible is a good option.

    Just be careful not to also confuse screen readers with that tactic, so that accessibility is maintained for humans. It should be easy enough if you keep your aria attributes filled out appropriately, I imagine.

  • hperrin@lemmy.world
    392·
    1 年前

    Pollute your site with nonsense that’s invisible to users. Things like pages that are linked to with invisible links that are just walls and walls of random text.

    • chevy9294@monero.townOP
      13·
      1 年前

      Good idea. I will made a invisible link to “traps for bots”. One trap will show random text, one will be redirect loop and one would be random link generator that will link to itself. I will also make every response randomly slow, for example 0,5 to 1,5 seconds.

      Good thing is that I can also block search engine crawlers from accessing only the traps.

        • Pantherina@feddit.de
          2·
          1 年前

          I dont think thats really a big problem. Like simply make every key word useless, somehow automate the process.

          There should be a tool for this damn, there is at least one Unicode character that doesnt even display a blank in a damn Terminal.

          Like… modern web crap doesnt even load without Javascript or animations. So dont bother a bit more HTML

    • folkrav@lemmy.ca
      9·
      1 年前

      OP still wants search indexing, in which case it’s a big no-no - it can be perceived as spam by search engines, and links your pages to tons of unrelated keywords.

    • stewsters@lemmy.world
      3·
      1 年前

      As long as you do not rely on SEO to get traffic. This has a good chance of affecting how Google sees your site as well.

  • inspxtr@lemmy.world
    13·
    1 年前

    I’m curious about how to verify that these bots respect the rules. I don’t doubt that they do, since it might be a PR nightmare for these big tech companies if they don’t, but I don’t know how to verify them. Asking because I’m also doing this for my website.

    By the way, LLMs are usually also trained by common crawl, (not sure to what extent), but I’m not sure whether you want to block common crawl.

    Another thing to consider is whether your website is indexed and crawled by web archive, and whether web archive has some policy on AI bot crawlers and scrapers.

  • Mubelotix@jlai.lu
    7·
    1 年前

    Block everyone but the crawlers you like. Blacklists are less reliable than whitelists

      • bbbhltz@beehaw.org
        62·
        1 年前

        Mostly the hype and because artists and creators are being hurt by its existence.

        I feel as though using AI is a cop-out. If I want to do something good, I also want to be proud of it. So I would rather not take that away from myself by doing it with AI. However, progress marches on, and I am neither an expert nor an authority on the subject. Asking someone like myself that question is nearly a trap. If I tell you that Generative AI is a bubble, like cryptocurrency and the Metaverse, that is just my gut feeling.

      • folkrav@lemmy.ca
        11·
        1 年前

        I was about to ask the same question. It’s one thing to think of the potential impacts of AI technology, but to be “against AI” in the most general sense is, to me, a weird concept, especially considering AI is so many things.

  • Oliver Lowe@lemmy.sdf.org
    5·
    1 年前

    Maybe there’s some IP address ranges to try block?

    It’s difficult because, for example, blocking the addresses OpenAI’s crawlers use may inadvertently block addresses from Azure used by Bing or whatever.

  • ⚡⚡⚡@feddit.de
    41·
    1 年前

    If you want this to work reliably for future bots BUT also want to allow search engines, you’ll loose this game.

    BTW: What makes you sure, that the search engine bot of Google does not crawl your website, store it in a cloud and AI is then used to later allow the search engine users to ask questions about your website and get AI generated answers. I think, that’s the goal of the search engines to improve results with AI…

  • jlow (he/him)@beehaw.org
    32·
    1 年前

    I don’t really understand the reasoning behind doing any of this, they didn’t give a fuck about stealing clearly copyrighted content in the first place, why would they care about you (not OP specifically) begging them not to steal your stuff. (As long as theres no laws about this which afaik there aren’t).

    • wagoner@infosec.pub
      1·
      1 年前

      So that leaves two options then. Leave the front door wide open, don’t bother with any locks. Or shut down the web site. I’m for at least closing the door with the right robots.txt

      • jlow (he/him)@beehaw.org
        1·
        1 年前

        The analogy should be either having the door open or having the door open but putting a note on the door saying to please not steal anything. I’m not saying you shouldn’t do it, I just don’t think it’s gonna do anything, so I’m not going to bother.

  • Xirup@lemmy.dbzer0.com
    1·
    1 年前

    Pehaps the user (or in this case the bot) will not go directly to your website, but first to some method of captcha verification or something like that, or like those pages (SteamDB for example) that do not open directly but first open a blank page to verify your network and browser with a captcha.