Scrapling 🕷️

🕷️ Scrapling tip: Paginating a JSON API with POST? Keep the spider's duplicate filter on. It checks the JSON body as well as the URL. Send page 1, page 2, then page 2 again to the same endpoint. Only two requests reach the server. 🎯

🕷️ Scrapling tip: You can use a regex inside XPath. Bind the EXSLT regex namespace and use re:test() to match a SKU on each row. One query gets Lamp and Desk and skips the Flash sale row. 🎯

🕷️ Scrapling tip: Two GraphQL calls hit /graphql. Which reply has the price? Turn on record_requests=True. Each saved reply carries its POST body in meta["request_body"]. Match the operationName and read the price without sending the query again 🎯

🕷️ Scrapling tip: RemoteSession(new_context=False) uses an open browser's login state. Connect to a browser with remote debugging enabled, then scrape without exporting cookies. When the session ends, your existing tabs stay open 🎯

🕷️ Scrapling tip: stream() has a 100-item buffer. If your writer falls behind, item producers wait when it fills. Save each result as it arrives instead of keeping the full result list in RAM. This example writes one JSON record per line ⚡

🕷️ Scrapling tip: AutoThrottle accepts a target below 1. Set autothrottle_target_concurrency=0.5 to move each site's delay toward twice its response time, within your delay limits. Slow sites get more breathing room without a fixed sleep. ⚡

🕷️ Scrapling tip: retry_blocked_request() can switch a blocked HTTP request to a stealth browser. Add that browser with lazy=True and it stays closed until needed. Fast pages use HTTP; blocked pages get one browser retry in this example ⚡

🕷️ Scrapling tip: on_scraped_item() can remove duplicate products from different URLs. Track each SKU in a set and return None for repeats. They stay out of results and stream(), while items_dropped counts them. This set lasts for one run 🎯

🕷️ Scrapling tip: sitemap_alternate_links=True makes SitemapSpider follow language URLs inside sitemap entries too. By default, it reads only <loc> URLs. Find pages listed only as language alternatives, with no custom XML parser 🎯

🕷️ Scrapling tip: CSVFeedSpider reads .csv.gz feeds and passes each row to parse_row() as a dict. No header row? Set headers and delimiter=";" to keep the first product as data. Gzip and CSV parsing are handled for you ⚡

🕷️ Scrapling tip: keep_comments=True preserves data tucked inside HTML comments. A table inside a comment is still invisible to CSS. Parse comment.text with a second Selector to get its rows, without running a browser. Try the demo below 🎯

🕷️ Scrapling tip: Response.markdown() removes scripts, templates, inline-hidden text and aria-hidden elements before conversion. Keep this hidden noise out of your LLM input without writing a cleanup pipeline. Try the example below ⚡

🕷️ Scrapling tip: adaptive CSS saves each part of ".price, .stock" separately. A broad selector saves only its first match. Use commas to save both fields, then recover price and stock after their classes change ⚡

🕷️ Scrapling tip: sitemap_follow filters child sitemaps before they are fetched. Point SitemapSpider at an index and keep only product maps. Blog and category maps are never downloaded, so you skip whole crawl branches with one LinkExtractor ⚡