Leaderboard Ad728 × 90AdSense placeholder — will activate after approval

ChatGPT prompt for building a Python web scraper with error handling and rate limiting

Coding web scraping Intermediate 🤖 ChatGPT 👁 2 views

📝 The Prompt

Act as a senior Python developer specializing in web scraping and data collection. Build a production-quality web scraper with proper error handling and ethical rate limiting. Target details: - Website to scrape: [URL or type of site — e.g., e-commerce product listings, job board, news articles] - Data to extract: [list specific fields — title, price, date, URL, description, etc.] - Number of pages/items: [approximate volume] - Frequency: [one-time / daily / weekly] - Output format: [CSV / JSON / database] The scraper should include: 1. HTTP client setup: - Use requests or httpx with session management - Custom User-Agent header - Retry logic with exponential backoff (3 retries) - Configurable rate limiting (requests per second) - Respect robots.txt 2. Parsing: - BeautifulSoup or lxml for HTML parsing - Robust selectors that handle missing elements gracefully - Data cleaning and normalization 3. Error handling: - HTTP errors (403, 404, 429, 500) - Network timeouts - Parsing errors (missing elements, changed HTML structure) - Logging all errors with context 4. Data pipeline: - Progress tracking (tqdm or similar) - Incremental saving (don't lose data if script crashes) - Deduplication - Output validation 5. Configuration: - Config file for URLs, selectors, rate limits - Command-line arguments for common options - Dry-run mode Provide the complete code, requirements.txt, and a README with usage instructions. Include ethical scraping guidelines.

⚙️ Replace 5 placeholders: [URL or type of site — e.g., e-commerce product listings, job board, news articles] [list specific fields — title, price, date, URL, description, etc.] [approximate volume] [one-time / daily / weekly] [CSV / JSON / database]

🎯 What this prompt does

This AI prompt helps you chatgpt prompt for building a python web scraper with error handling and rate limiting. Designed for web scraping workflows in the coding category, it's a intermediate-level prompt you can copy directly into ChatGPT to get instant, production-ready results.

Use it when you need a intermediate prompt that produces clear, actionable output without wrestling with trial-and-error wording. Just copy, customize, and run.

In-article Ad #1336 × 280AdSense placeholder — will activate after approval

🚀 How to use this prompt

  1. Copy the prompt using the 📋 button above.
  2. Open ChatGPT (or Claude, Gemini, Perplexity, or your preferred LLM).
  3. Paste the prompt into a new chat. Replace 5 bracketed placeholders ([URL or type of site — e.g., e-commerce product listings, job board, news articles] [list specific fields — title, price, date, URL, description, etc.] [approximate volume] ) with your own details.
  4. Run the prompt and review the AI's response. Most outputs are usable immediately.
  5. Iterate if needed — if the tone, length, or structure isn't quite right, reply with "make it shorter", "use bullet points", or "make it more formal" and the AI will refine it.

💡 Tips for better results

  • Replace the bracketed placeholders ([URL or type of site — e.g., e-commerce product listings, job board, news articles], [list specific fields — title, price, date, URL, description, etc.], [approximate volume], [one-time / daily / weekly]) with your own specifics before sending.
  • If the first output isn't quite right, ask the AI to refine, rewrite, or add more detail — iteration is key.
  • For long outputs, ask for a section at a time (e.g. 'start with the introduction only') to keep quality high.
  • Combine this with other coding prompts to build an end-to-end workflow.
  • Save your favorite variations — small wording tweaks often produce noticeably different results.
In-article Ad #2336 × 280AdSense placeholder — will activate after approval

✨ What you'll get

When you run this prompt, expect ChatGPT to return:

  • A directly usable web scraping output tailored to the details you provided
  • Clear structure (headings, bullets, or numbered sections) that you can drop into your workflow
  • Content that matches your specified tone and context
  • Results in under 30 seconds — no manual drafting required

Need a different angle? Just ask follow-up questions. The AI will adjust without you starting over.

🔄 3 variations to try

1

Make it more formal

Add "Use a formal, professional tone suitable for enterprise clients" at the start of the prompt.

2

Ask for multiple options

Append "Give me 5 alternative versions, each with a different angle or approach." after the main instruction.

3

Request structured output

Add "Return the response as a markdown table (or bullet list, or JSON)" so you can paste the result directly into your docs or code.

🏷 Tags

🔎 Find more prompts like this

Browse 76 more coding prompts or search the full library.

End-of-content Ad728 × 90AdSense placeholder — will activate after approval
Mobile Sticky320 × 50AdSense placeholder — will activate after approval