What is robots.txt?
robots.txt is a text file in which a website tells search robots and data-collection programs which areas they may or may not visit. It has no legal force and works more like a convention that crawlers agree to follow.
Terms seen alongside this one
- Computer Use 1× together
- reinforcement learning 1× together
- civil investigative demands 1× together
- unified memory 1× together
Get it in your inbox every day
Daily headlines and summaries, a weekly synthesis every Sunday, and a monthly report at the start of each month. Sent at 8AM KST, which is the evening before in the US.
Free · no ads · one-click unsubscribe