1 post
Two levels: ask crawlers not to via robots.txt, and enforce it by never serving gated content to unauthenticated bots. Only the second is real protection.