Crawl rate limiting refers to controls, either set by a website owner through Search Console or determined by a search engine’s own systems, that restrict how quickly and frequently a crawler accesses a website’s pages, preventing server overload from excessive crawling.
Setting an overly restrictive crawl rate can inadvertently slow down how quickly new or updated content gets discovered and indexed, while too aggressive a crawl rate on a server with limited resources can cause performance issues for real visitors.
Most modern search engines, particularly Google, are generally sophisticated at automatically adjusting crawl rate based on how a server responds, reducing the need for manual intervention in most cases.
We monitor crawl rate behavior as part of our Technical SEO service.