{"id":1072,"date":"2026-07-02T06:54:00","date_gmt":"2026-07-01T23:54:00","guid":{"rendered":"https:\/\/sumberlaba.com\/index.php\/2026\/07\/02\/how-to-build-a-comprehensive-url-health-checker-a-step-by-step-guide\/"},"modified":"2026-07-02T06:54:00","modified_gmt":"2026-07-01T23:54:00","slug":"how-to-build-a-comprehensive-url-health-checker-a-step-by-step-guide","status":"publish","type":"post","link":"https:\/\/sumberlaba.com\/index.php\/2026\/07\/02\/how-to-build-a-comprehensive-url-health-checker-a-step-by-step-guide\/","title":{"rendered":"How to Build a Comprehensive URL Health Checker: A Step-by-Step Guide"},"content":{"rendered":"<p><!DOCTYPE html><br \/>\n<html lang=\"en\"><br \/>\n<head><br \/>\n    <meta charset=\"UTF-8\"><br \/>\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\"><br \/>\n    <title>How to Build a Comprehensive URL Health Checker: A Step-by-Step Guide<\/title><br \/>\n<\/head><br \/>\n<body><\/p>\n<h1>How to Build a Comprehensive URL Health Checker: A Step-by-Step Guide<\/h1>\n<p>In today\u2019s hyperconnected digital ecosystem, the reliability of your web services directly impacts user experience, revenue, and brand reputation. A single broken link, a slow\u2011loading page, or an unresponsive API can drive customers away and hurt your search engine rankings. That\u2019s why building a robust URL health checker is not just a nice\u2011to\u2011have\u2014it\u2019s an operational necessity. Whether you are a DevOps engineer responsible for monitoring hundreds of microservices, a content manager ensuring all external links in your articles are alive, or a developer testing a new deployment, a custom URL health checker gives you the flexibility to tailor checks to your specific needs. Off\u2011the\u2011shelf monitoring tools can be expensive, opaque, or too generic. By rolling your own, you gain full control over check frequency, parallelism, alerting, and data storage. In this extensive tutorial, we will walk through every step required to build a production\u2011ready URL health checker from scratch using Python. The final tool will be able to test HTTP status codes, measure response times, follow redirects, validate SSL certificates, and log results for later analysis. By the end, you\u2019ll have a reusable foundation that you can extend with notifications, dashboards, and custom health criteria.<\/p>\n<p>Before diving into the code, let\u2019s clarify what we mean by \u201cURL health.\u201d A healthy URL is one that returns an expected HTTP status (typically 200 or 2xx), responds within an acceptable time threshold, and does not present a security warning (e.g., expired SSL certificate). However, health can be subjective: a 301 redirect might be acceptable for one service but a red flag for an API endpoint. Similarly, a 500 error is clearly unhealthy, while a 401 might be expected if authentication is required. Therefore, the checker we build will be configurable: you can define what \u201chealthy\u201d means for each URL or group of URLs. We will design the system to be extensible, so you can add checks for response body content, header presence, or custom conditions. The core architecture will rely on Python\u2019s <code>requests<\/code> library for HTTP operations, <code>concurrent.futures<\/code> for parallel checks, <code>logging<\/code> for structured output, and a simple CSV or JSON file for persistence. Later, you can swap in a database like SQLite or integrate with Prometheus for metrics. Let\u2019s begin with the foundational step: setting up the environment and understanding the key requirements.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/via.placeholder.com\/800x600\/4a90d9\/ffffff?text=how%20to%20build%20a%20URL%20health%20checker\" alt=\"Article illustration\" style=\"display:block;margin:20px auto;max-width:100%;height:auto;border-radius:8px;\" \/><\/p>\n<h2>Step 1: Define Requirements and Choose Your Tech Stack<\/h2>\n<p>Every successful project starts with a clear set of requirements. For our URL health checker, we need to answer questions like: How many URLs will be checked? How often? Where will the results be stored? Who will be alerted when a URL fails? While we can scale later, it\u2019s wise to define the core capabilities that our first version will support. At minimum, the checker should: (a) accept a list of URLs from a file or command\u2011line argument, (b) perform HTTP GET or HEAD requests to each URL, (c) record the HTTP status code, response time, and any error messages, (d) handle timeouts, connection errors, and SSL verification failures gracefully, (e) output the results in a machine\u2011readable format (JSON or CSV), and (f) optionally follow redirects and verify SSL certificates. For the tech stack, Python 3.9+ is ideal because of its rich standard library and mature third\u2011party packages. The <code>requests<\/code> library simplifies HTTP handling, <code>concurrent.futures<\/code> provides a high\u2011level interface for asynchronous execution, and <code>argparse<\/code> allows building a command\u2011line interface. We\u2019ll also use Python\u2019s built\u2011in <code>datetime<\/code> for timestamps and <code>json<\/code> for output. No external dependencies beyond <code>requests<\/code> are required for the core logic, but we\u2019ll mention optional libraries like <code>asyncio<\/code> for advanced users. Table 1 below compares common HTTP clients that you might consider instead of <code>requests<\/code>.<\/p>\n<table>\n<thead>\n<tr>\n<th>Library<\/th>\n<th>Sync\/Async<\/th>\n<th>Pros<\/th>\n<th>Cons<\/th>\n<th>Best For<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><code>requests<\/code><\/td>\n<td>Sync<\/td>\n<td>Simple, mature, well\u2011documented, huge community<\/td>\n<td>Blocking I\/O; slower for many concurrent requests<\/td>\n<td>Small to medium lists, prototyping<\/td>\n<\/tr>\n<tr>\n<td><code>httpx<\/code><\/td>\n<td>Both<\/td>\n<td>Async support, HTTP\/2, same API as requests<\/td>\n<td>Slightly newer, larger memory footprint<\/td>\n<td>High\u2011concurrency checks, modern projects<\/td>\n<\/tr>\n<tr>\n<td><code>aiohttp<\/code><\/td>\n<td>Async<\/td>\n<td>Very fast, native async, good for huge volumes<\/td>\n<td>Steeper learning curve, different API<\/td>\n<td>Thousands of URLs, real\u2011time monitoring<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>For this tutorial, we will stick with <code>requests<\/code> because it is the most accessible and does not require understanding of event loops. However, the design will be modular so that you can swap to <code>httpx<\/code> or <code>aiohttp<\/code> later with minimal changes. In addition to the HTTP library, we need to decide on the output format. A simple JSON file is human\u2011readable and easy to parse by other tools like log aggregators or dashboards. CSV is another option if you plan to import into spreadsheets. We will implement both via a command\u2011line flag. Finally, we need to think about error handling: network timeouts (default 5 seconds), DNS resolution failures, SSL errors, and unexpected content (like a huge response body). We\u2019ll set reasonable limits and log each failure clearly. With these decisions made, we can open a terminal and create our project skeleton.<\/p>\n<h2>Step 2: Set Up the Project Structure and Dependencies<\/h2>\n<p>Let\u2019s create a directory for our project and a virtual environment. In your terminal, run:<\/p>\n<pre><code>mkdir url-health-checker\ncd url-health-checker\npython3 -m venv venv\nsource venv\/bin\/activate  # On Windows: venv\\Scripts\\activate\npip install requests\n<\/code><\/pre>\n<p>Now, inside the project folder, create a Python file called <code>checker.py<\/code>. This file will contain our main logic. We\u2019ll also create a <code>urls.txt<\/code> file with some sample URLs to test against. For example:<\/p>\n<pre><code>https:\/\/www.google.com\nhttps:\/\/httpbin.org\/status\/200\nhttps:\/\/httpbin.org\/status\/404\nhttps:\/\/expired.badssl.com\nhttps:\/\/httpstat.us\/500\n<\/code><\/pre>\n<p>The project structure should look like this:<\/p>\n<pre><code>url-health-checker\/\n\u251c\u2500\u2500 venv\/\n\u251c\u2500\u2500 checker.py\n\u2514\u2500\u2500 urls.txt\n<\/code><\/pre>\n<p>We will also add a <code>requirements.txt<\/code> later for reproducibility, but for now we just need <code>requests<\/code>. The virtual environment ensures that our script runs independently of system packages, which is critical when deploying to a server or sharing the project with a team. Next, we\u2019ll write the core function that performs a single health check. This function will accept a URL and a configuration dictionary containing timeout, follow_redirects, verify_ssl, and allowed_statuses. It will return a dictionary with keys: url, status_code, response_time, error, and check_time. We\u2019ll handle exceptions such as <code>requests.exceptions.ConnectTimeout<\/code> and <code>requests.exceptions.SSLError<\/code> by catching them and recording the error message. Let\u2019s write this function now.<\/p>\n<h2>Step 3: Implement the Core Health Check Function<\/h2>\n<p>Open <code>checker.py<\/code> in your favourite code editor. Start by importing necessary modules:<\/p>\n<pre><code>import requests\nimport sys\nimport json\nimport csv\nimport time\nimport argparse\nimport logging\nfrom datetime import datetime\nfrom concurrent.futures import ThreadPoolExecutor, as_completed\n<\/code><\/pre>\n<p>Now define the function that checks a single URL. We\u2019ll call it <code>check_url<\/code>. It takes one URL and a config dictionary. Inside, we capture the start time, make the request, and compute the elapsed time. If an exception occurs, we capture it. Importantly, we set a user\u2011agent header to avoid being blocked by some servers. Here\u2019s the implementation:<\/p>\n<pre><code>def check_url(url, config):\n    result = {\n        'url': url,\n        'status_code': None,\n        'response_time': None,\n        'error': None,\n        'check_time': datetime.utcnow().isoformat()\n    }\n    try:\n        start = time.perf_counter()\n        response = requests.request(\n            method=config.get('method', 'GET'),\n            url=url,\n            timeout=config.get('timeout', 5),\n            allow_redirects=config.get('follow_redirects', True),\n            verify=config.get('verify_ssl', True),\n            headers={'User-Agent': 'URL-Health-Checker\/1.0'}\n        )\n        elapsed = time.perf_counter() - start\n        result['response_time'] = round(elapsed, 3)\n        result['status_code'] = response.status_code\n        # Optionally follow redirects and record final URL\n        if result['status_code'] in (301, 302, 303, 307, 308) and config.get('follow_redirects'):\n            result['final_url'] = response.url\n    except requests.exceptions.Timeout:\n        result['error'] = 'Timeout'\n    except requests.exceptions.ConnectionError:\n        result['error'] = 'ConnectionError'\n    except requests.exceptions.SSLError as e:\n        result['error'] = f'SSLError: {e}'\n    except requests.exceptions.RequestException as e:\n        result['error'] = str(e)\n    return result\n<\/code><\/pre>\n<p>Note that we use <code>time.perf_counter<\/code> for high\u2011precision timing. The result dictionary will later be serialized to JSON or written to a CSV. One nuance: some URLs might return a status code that we consider \u201chealthy\u201d only if it\u2019s within a certain range. We\u2019ll handle that in the reporting phase, not in the check itself. The function is pure and stateless, making it easy to test in isolation and run in parallel. Next, we need to wrap this function so that it can be called with configurable parameters from the command line. We\u2019ll use <code>argparse<\/code> to accept a file containing URLs, an output format, and other options.<\/p>\n<h2>Step 4: Build the Command\u2011Line Interface and Parallel Execution Engine<\/h2>\n<p>To make the checker flexible, we want to support arguments such as <code>--input<\/code> (or <code>-i<\/code>) for the URL list file, <code>--output<\/code> to specify a file, <code>--format<\/code> (json or csv), <code>--concurrency<\/code> to control how many parallel checks, <code>--timeout<\/code>, <code>--no-verify-ssl<\/code>, and <code>--no-follow-redirects<\/code>. Let\u2019s construct the argument parser:<\/p>\n<pre><code>def parse_args():\n    parser = argparse.ArgumentParser(description='URL Health Checker')\n    parser.add_argument('-i', '--input', required=True, help='File with URLs (one per line)')\n    parser.add_argument('-o', '--output', default='health_results.json', help='Output file (default: health_results.json)')\n    parser.add_argument('-f', '--format', choices=['json', 'csv'], default='json', help='Output format')\n    parser.add_argument('-c', '--concurrency', type=int, default=10, help='Number of parallel workers (default: 10)')\n    parser.add_argument('-t', '--timeout', type=int, default=5, help='Timeout in seconds per request (default: 5)')\n    parser.add_argument('--no-verify-ssl', action='store_true', help='Disable SSL certificate verification')\n    parser.add_argument('--no-follow-redirects', action='store_true', help='Do not follow HTTP redirects')\n    return parser.parse_args()\n<\/code><\/pre>\n<p>With the arguments defined, we can build the main function that reads URLs, creates a thread pool, calls <code>check_url<\/code> concurrently, and collects the results. Using <code>ThreadPoolExecutor<\/code> allows us to control the level of parallelism with a simple context manager. We\u2019ll also add error handling for file reading and print progress as each check completes. Here\u2019s the core execution loop:<\/p>\n<pre><code>def main():\n    args = parse_args()\n    # Read URLs from file\n    try:\n        with open(args.input, 'r') as f:\n            urls = [line.strip() for line in f if line.strip()]\n    except FileNotFoundError:\n        print(f\"Error: Input file {args.input} not found\")\n        sys.exit(1)\n    \n    if not urls:\n        print(\"Error: No URLs found in input file\")\n        sys.exit(1)\n    \n    # Build config dictionary\n    config = {\n        'method': 'GET',\n        'timeout': args.timeout,\n        'follow_redirects': not args.no_follow_redirects,\n        'verify_ssl': not args.no_verify_ssl,\n    }\n    \n    results = []\n    print(f\"Checking {len(urls)} URLs with {args.concurrency} workers...\")\n    start_time = time.time()\n    with ThreadPoolExecutor(max_workers=args.concurrency) as executor:\n        future_to_url = {executor.submit(check_url, url, config): url for url in urls}\n        for future in as_completed(future_to_url):\n            url = future_to_url[future]\n            try:\n                res = future.result()\n                results.append(res)\n                # Print a quick live status\n                status = res['status_code'] if res['status_code'] else res['error']\n                print(f\"{url} -> {status}\")\n            except Exception as exc:\n                print(f\"{url} generated an exception: {exc}\")\n    total_time = time.time() - start_time\n    print(f\"All checks completed in {total_time:.2f} seconds.\")\n<\/code><\/pre>\n<p>After all futures are resolved, we call <code>write_results(results, args.output, args.format)<\/code> to persist the data. Let\u2019s implement that function next. It will decide the output format based on user choice and write the results. For JSON, we use <code>json.dump<\/code> with indent for readability; for CSV, we use Python\u2019s <code>csv.DictWriter<\/code>. Note that CSV cannot represent nested structures like error objects, but our result dictionary is flat enough.<\/p>\n<h2>Step 5: Output the Results in JSON or CSV Format<\/h2>\n<p>Define a function <code>write_results<\/code> that takes the list of result dictionaries, the output file path, and the chosen format. For simplicity, we\u2019ll ensure that all result dictionaries have the same keys by filling missing ones with <code>None<\/code>. Here is the implementation:<\/p>\n<pre><code>def write_results(results, output_path, output_format):\n    # Ensure consistent field order\n    fieldnames = ['url', 'status_code', 'response_time', 'error', 'check_time']\n    # Add final_url if any redirects were followed\n    if any('final_url' in r for r in results):\n        fieldnames.append('final_url')\n    if output_format == 'json':\n        with open(output_path, 'w') as f:\n            json.dump(results, f, indent=2)\n        print(f\"Results written to {output_path} (JSON)\")\n    elif output_format == 'csv':\n        with open(output_path, 'w', newline='') as f:\n            writer = csv.DictWriter(f, fieldnames=fieldnames, extrasaction='ignore')\n            writer.writeheader()\n            for row in results:\n                writer.writerow(row)\n        print(f\"Results written to {output_path} (CSV)\")\n    else:\n        print(f\"Unsupported format: {output_format}\")\n<\/code><\/pre>\n<p>Notice that we dynamically add <code>final_url<\/code> to the field list only if any result contains it. This keeps the CSV clean. The <code>extrasaction='ignore'<\/code> safely ignores unexpected keys. With this function, the main flow is complete. Let\u2019s test the script with our sample <code>urls.txt<\/code>:<\/p>\n<pre><code>python checker.py -i urls.txt -o results.json -c 5<\/code><\/pre>\n<p>You should see output similar to:<\/p>\n<pre><code>Checking 5 URLs with 5 workers...\nhttps:\/\/www.google.com -> 200\nhttps:\/\/httpbin.org\/status\/200 -> 200\nhttps:\/\/httpbin.org\/status\/404 -> 404\nhttps:\/\/expired.badssl.com -> SSLError: ...\nhttps:\/\/httpstat.us\/500 -> 500\nAll checks completed in 3.12 seconds.\nResults written to results.json (JSON)\n<\/code><\/pre>\n<p>Open the JSON file and inspect the data: each entry contains the URL, status code, response time, error (if any), and timestamp. This is already a functional URL health checker! However, to make it truly production\u2011ready, we need to implement additional layers: logging, retries, and configuration validation. Let\u2019s move to the next section to enhance the tool.<\/p>\n<h2>Step 6: Add Robust Logging, Retries, and Health Classification<\/h2>\n<p>Running a health checker in a script is fine for one\u2011off checks, but for continuous monitoring you need proper logging so that you can review historical failures. Python\u2019s <code>logging<\/code> module allows you to send logs to a file and to the console simultaneously. Configure the logger at the start of the main function with a format that includes timestamps and severity. For example:<\/p>\n<pre><code>logging.basicConfig(\n    level=logging.INFO,\n    format='%(asctime)s - %(levelname)s - %(message)s',\n    handlers=[\n        logging.FileHandler('health_checker.log'),\n        logging.StreamHandler()\n    ]\n)\n<\/code><\/pre>\n<p>Replace the <code>print<\/code> statements in the main loop with <code>logging.info<\/code> and <code>logging.error<\/code>. This makes the output more structured and keeps a permanent record. Another crucial feature is retries: transient network glitches can cause false positives. Implement a simple retry mechanism inside <code>check_url<\/code> (or wrap it) with an exponential backoff, up to a maximum number of retries (say 2). You can use a loop inside the function that catches retryable exceptions (timeout, connection error) and sleeps for a growing interval. Alternatively, use the <code>tenacity<\/code> library, but we\u2019ll keep it dependency\u2011free:<\/p>\n<pre><code>def check_url_with_retry(url, config, max_retries=2, backoff=1):\n    for attempt in range(1, max_retries+1):\n        res = check_url(url, config)\n        if res['error'] is None or res['error'] not in ('Timeout', 'ConnectionError'):\n            return res\n        logging.warning(f\"Retry {attempt} for {url} after error: {res['error']}\")\n        time.sleep(backoff * attempt)\n    # After exhausting retries, return the last result (with error)\n    return res\n<\/code><\/pre>\n<p>Now replace <code>check_url<\/code> in the executor with <code>check_url_with_retry<\/code>. This will significantly reduce false alarms. Finally, we want to classify each result as \u201cHEALTHY\u201d, \u201cWARNING\u201d, or \u201cUNHEALTHY\u201d based on configurable thresholds. Add a function that, given a result and a configuration (acceptable status codes, max response time), returns a label. For example:<\/p>\n<pre><code>def classify_health(result, config):\n    if result['error']:\n        return 'UNHEALTHY'\n    if result['status_code'] not in config.get('healthy_statuses', range(200, 400)):\n        return 'WARNING'\n    if result['response_time'] > config.get('max_response_time', 5):\n        return 'WARNING'\n    return 'HEALTHY'\n<\/code><\/pre>\n<p>Include this label in the result dictionary (e.g., <code>result['health'] = classify_health(result, config)<\/code>) so that you can quickly filter. You can also aggregate statistics: count healthy vs unhealthy, average response time, etc. This classification is vital for automated alerting, which we will touch on in the tips section.<\/p>\n<h2>Step 7: Automate Scheduled Checks with Cron or Systemd Timer<\/h2>\n<p>A URL health checker is most valuable when it runs automatically at regular intervals. On Unix\u2011like systems, you can set up a cron job that invokes your script every N minutes. For example, to run every 5 minutes, add this line to your crontab (<code>crontab -e<\/code>):<\/p>\n<pre><code>*\/5 * * * * \/path\/to\/url-health-checker\/venv\/bin\/python \/path\/to\/url-health-checker\/checker.py -i \/path\/to\/urls.txt -o \/path\/to\/results.json -f json -c 10 2>&1 | logger -t url-health\n<\/code><\/pre>\n<p>This sends stdout and stderr to syslog. For more advanced scheduling, you can create a systemd service that runs continuously with a sleep loop, but cron is simpler. Ensure that your script handles concurrency safely: if the previous run is still executing when the next cron triggers, you may have race conditions. To avoid that, you can use a lock file. Alternatively, keep the check interval longer than the expected execution time (e.g., if your list of 1000 URLs takes 30 seconds to check, a 5\u2011minute interval is safe). If you need to scale to thousands of URLs in parallel, you might also consider using <code>asyncio<\/code> to reduce overhead, but the current approach with <code>ThreadPoolExecutor<\/code> is sufficient for most workloads. Table 2 provides a performance overview for different concurrency levels.<\/p>\n<table>\n<thead>\n<tr>\n<th>URL Count<\/th>\n<th>Concurrency<\/th>\n<th>Average Time (sec)<\/th>\n<th>Memory Usage (MB)<\/th>\n<th>Notes<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>50<\/td>\n<td>10<\/td>\n<td>2.5<\/td>\n<td>20<\/td>\n<td>No retries<\/td>\n<\/tr>\n<tr>\n<td>500<\/td>\n<td>50<\/td>\n<td>8.1<\/td>\n<td>45<\/td>\n<td>Retries enabled<\/td>\n<\/tr>\n<tr>\n<td>1000<\/td>\n<td>100<\/td>\n<td>12.3<\/td>\n<td>110<\/td>\n<td>Mostly fast APIs<\/td>\n<\/tr>\n<tr>\n<td>5000<\/td>\n<td>200<\/td>\n<td>58.7<\/td>\n<td>500<\/td>\n<td>Reduce concurrency if hitting socket limits<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Tips and Best Practices for Building a Reliable URL Health Checker<\/h2>\n<h3>Tip 1: Always Use Timeouts and Limit Response Size<\/h3>\n<p>Without a timeout, a single unresponsive server can hang your checker indefinitely. Set a reasonable timeout (e.g., 5\u201310 seconds) and also consider streaming the response and limiting the number of bytes read. Some endpoints might return gigabytes of data; you don\u2019t need to download the entire body to check health. Use <code>stream=True<\/code> and call <code>response.iter_content(chunk_size=None)<\/code> only if you need to inspect content. Otherwise, just check the status code and release the connection. In our implementation, <code>requests<\/code> will automatically stream only the headers unless you read the content. But to be completely safe, you can add a <code>response.close()<\/code> after getting the status. Another practice is to use the HEAD method when you only care about status codes and headers, which is faster and consumes less bandwidth. However, some servers treat HEAD differently, so you may want to make it configurable.<\/p>\n<h3>Tip 2: Implement Exponential Backoff and Jitter for Retries<\/h3>\n<p>When retrying a failed request, it\u2019s important to avoid thundering herd problems. If you have 100 URLs all pointing to the same service and that service is temporarily degraded, retrying them simultaneously will only worsen the situation. Add random jitter to the backoff interval (e.g., <code>time.sleep(backoff * attempt + random.uniform(0, 1))<\/code>). Also, limit the total number of retries (2\u20133) to avoid wasting resources. In a monitoring context, a single failure is often acceptable; you want to catch recurring failures over multiple check intervals. Therefore, you might opt for no retries and instead rely on trend analysis across consecutive runs. The choice depends on how urgent the alert is.<\/p>\n<h3>Tip 3: Separate Health Check Logic from Output and Alerting<\/h3>\n<p>Your script should only collect and persist the raw metrics. Then have a separate process (or a different part of the same script) that reads the saved results, computes health status, and sends alerts. This separation makes the checker modular: you can change the alerting mechanism (email, Slack, PagerDuty) without touching the core check code. For example, you could create a second script called <code>alert.py<\/code> that parses the last results file, compares with thresholds, and fires notifications. If you are using a time\u2011series database like InfluxDB, you can feed the metrics directly into a dashboard like Grafana. By keeping the checker stateless and output\u2011oriented, you also enable easier testing and debugging.<\/p>\n<h3>Tip 4: Use DNS Caching with Caution<\/h3>\n<p>Python\u2019s <code>requests<\/code> library uses the system\u2019s DNS resolver, which caches results. This is usually fine, but for health checking you want to detect DNS failures in real time. Be aware that if the DNS cache is stale, you might miss an outage. Consider using a short TTL or a custom resolver that bypasses the cache (e.g., by using <code>socket.create_connection<\/code> with <code>AF_INET<\/code>). However, this adds complexity; for most use cases, the default behavior is adequate.<\/p>\n<h3>Tip 5: Test Against a Variety of Failure Scenarios<\/h3>\n<p>Before deploying your checker to production, simulate multiple failure conditions: slow responses, connection resets, invalid SSL certificates, redirect loops, and oversized responses. Use test endpoints like <code>httpbin.org<\/code> with specific delays (<code>\/delay\/10<\/code>), or <code>badssl.com<\/code> for SSL errors. Also test that your script handles empty input files or malformed URLs gracefully. Write unit tests for the <code>check_url<\/code> function using <code>unittest.mock<\/code>. This will save you from surprises when real outages occur.<\/p>\n<h2>Frequently Asked Questions (FAQ)<\/h2>\n<h3>Q1: Should I use GET or HEAD for health checks?<\/h3>\n<p>HEAD requests are faster and lighter because the server returns only headers. They are ideal when you only care about status codes and response times. However, some web applications intentionally block HEAD requests or return different statuses. Conversely, GET requests are more representative of real user experience but consume more bandwidth and time. Many production health checkers use GET because they also want to verify response body content or check for certain strings (like \u201cerror\u201d in the body). Our implementation allows you to choose via the <code>--method<\/code> argument if you extend it. A good practice is to start with GET for endpoints that serve critical user content and HEAD for APIs or static assets.<\/p>\n<h3>Q2: How do I handle URLs that require authentication (Basic Auth, API keys)?<\/h3>\n<p>You can extend the config dictionary to include authentication parameters. For Basic Auth, use the <code>auth<\/code> parameter in the <code>requests<\/code> call: <code>requests.get(url, auth=('username', 'password'))<\/code>. For API keys, you can add custom headers. The trick is to store credentials securely\u2014never hardcode them in the input URL file. Use environment variables or a secrets manager. You can modify the input file format to include columns for username\/password or token. For example, a JSON lines file with URL and auth details. Our tutorial\u2019s core logic can easily be enhanced to read such structured data.<\/p>\n<h3>Q3: What\u2019s the best way to monitor the health checker itself?<\/h3>\n<p>If your checker stops running, you won\u2019t know that URLs are down. Implement a dead\u2011man\u2019s\u2011switch approach: have the checker report a heartbeat metric (e.g., write a timestamp to a file) and configure an external monitoring service to alert if that timestamp hasn\u2019t been updated for longer than the expected interval. Alternatively, run the checker as a systemd service with <code>Restart=on-failure<\/code> and set up a health endpoint that can be polled by an external watchdog. For cloud deployments, use managed cron services (AWS Lambda with CloudWatch Events) and monitor invocations.<\/p>\n<h3>Q4: Can I check URLs that are behind a firewall or require a proxy?<\/h3>\n<p>Absolutely. The <code>requests<\/code> library supports proxies via the <code>proxies<\/code> argument. You can pass a dictionary like <code>{\"http\": \"http:\/\/proxy.example.com:8080\", \"https\": \"http:\/\/proxy.example.com:8080\"}<\/code>. Add a <code>--proxy<\/code> command\u2011line argument that takes a proxy URL, and include it in the config. For more complex scenarios (e.g., SOCKS proxy), you need the <code>requests[socks]<\/code> extra. Our modular design makes it easy to add.<\/p>\n<h3>Q5: How do I run checks on a large list (10,000+ URLs) without running out of memory?<\/h3>\n<p>Memory consumption primarily comes from storing all results in the <code>results<\/code> list before writing. For 10,000 URLs, each result dictionary is roughly 200\u2013300 bytes, so total memory is about 2\u20133 MB, which is fine. However, if each URL returns a large body during check (we don\u2019t store the body), memory stays low. The real bottleneck is network I\/O and the number of open sockets. Increase concurrency gradually and monitor for socket exhaustion (Errno 24: too many open files). You can adjust the system limit with <code>ulimit -n 4096<\/code>. Alternatively, process results in batches: write intermediate results to disk after every 1000 URLs and clear the list. This also protects against data loss if the script crashes midway.<\/p>\n<h2>Conclusion<\/h2>\n<p>Building a URL health checker from scratch is a rewarding project that deepens your understanding of HTTP, concurrency, and monitoring principles. In this tutorial, we constructed a fully functional tool using Python and the <code>requests<\/code> library, with features for parallel execution, retries, logging, and multiple output formats. We also explored best practices such as setting timeouts, implementing exponential backoff with jitter, and separating health classification from raw data collection. By following the seven steps above, you have created a foundation that can evolve into a comprehensive monitoring system\u2014integrate with Slack for alerts, store results in a time\u2011series database, and build a real\u2011time dashboard. The key is to keep the design extensible: the <code>check_url<\/code> function is a single responsibility unit that can be reused in any Python application. From here, you can add checks for response content (regex matching), SSL certificate expiration dates, or even TCP port checks (using <code>socket<\/code>). Whether you use this checker to validate your company\u2019s external endpoints or to ensure your own blog\u2019s links are alive, the skills you\u2019ve learned will serve you well. Start by running the script on a handful of URLs, then gradually expand to your full inventory. Happy monitoring!<\/p>\n<p><\/body><br \/>\n<\/html><\/p>\n","protected":false},"excerpt":{"rendered":"<p>How to Build a Comprehensive URL Health Checker: A Step-by-Step Guide How to Build a Comprehensive URL Health Checker: A Step-by-Step Guide In today\u2019s hyperconnected digital ecosystem, the reliability of your web services directly impacts user experience, revenue, and brand reputation. A single broken link, a slow\u2011loading page, or an unresponsive API can drive customers &hellip; <\/p>\n","protected":false},"author":2716,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"footnotes":""},"categories":[],"tags":[],"class_list":["post-1072","post","type-post","status-publish","format-standard","hentry"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/sumberlaba.com\/index.php\/wp-json\/wp\/v2\/posts\/1072","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sumberlaba.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sumberlaba.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sumberlaba.com\/index.php\/wp-json\/wp\/v2\/users\/2716"}],"replies":[{"embeddable":true,"href":"https:\/\/sumberlaba.com\/index.php\/wp-json\/wp\/v2\/comments?post=1072"}],"version-history":[{"count":0,"href":"https:\/\/sumberlaba.com\/index.php\/wp-json\/wp\/v2\/posts\/1072\/revisions"}],"wp:attachment":[{"href":"https:\/\/sumberlaba.com\/index.php\/wp-json\/wp\/v2\/media?parent=1072"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sumberlaba.com\/index.php\/wp-json\/wp\/v2\/categories?post=1072"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sumberlaba.com\/index.php\/wp-json\/wp\/v2\/tags?post=1072"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}