Repository navigation
Memory is critically overloaded WARNING #550
Description
I'm running the quickstart code in WSL2 on Windows 11. Crawlee is not using the full available memory in WSL2. I have 15Gi total memory in WSL2
$ free -mh
total used free shared buff/cache available
Mem: 15Gi 2.7Gi 8.9Gi 3.0Mi 3.9Gi 12Gi
Swap: 4.0Gi 0B 4.0GiCrawlee is only using 3.88GB for some reason.
[crawlee._autoscaling.snapshotter] WARN Memory is critically overloaded. Using 5.24 GB of 3.88 GB (135%). Consider increasing available memory.How can I make Crawlee use a larger percentage of available memory?
Code (same code as the quickstart):
import asyncio
from crawlee.playwright_crawler import PlaywrightCrawler, PlaywrightCrawlingContext
async def main() -> None:
# PlaywrightCrawler crawls the web using a headless browser controlled by the Playwright library.
crawler = PlaywrightCrawler()
# Define a request handler to process each crawled page and attach it to the crawler using a decorator.
@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
# Extract relevant data from the page context.
data = {
'url': context.request.url,
'title': await context.page.title(),
}
# Store the extracted data.
await context.push_data(data)
# Extract links from the current page and add them to the crawling queue.
await context.enqueue_links()
# Add first URL to the queue and start the crawl.
await crawler.run(['https://crawlee.dev'])
if __name__ == '__main__':
asyncio.run(main())
Activity