What is crawler?
A crawler is a program that visits websites and automatically collects their content. Search engines and AI companies use them to gather material.
Terms seen alongside this one
- robots.txt 1× together
- reinforcement learning 1× together
- bridge round 1× together
- sandbox 1× together
Get it in your inbox every day
Daily headlines and summaries, a weekly synthesis every Sunday, and a monthly report at the start of each month. Sent at 8AM KST, which is the evening before in the US.
Free · no ads · one-click unsubscribe