crawler.go in Go

Fetch several URLs concurrently and report each result.

// Fetch several URLs concurrently and report status and size.
package main

import (
    "fmt"
    "io"
    "net/http"
    "sync"
    "time"
)

type result struct {
    url    string
    status int
    size   int
    err    error
}

func fetch(url string, client *http.Client) result {
    resp, err := client.Get(url)
    if err != nil {
        return result{url: url, err: err}
    }
    defer resp.Body.Close()

    body, err := io.ReadAll(resp.Body)
    return result{
        url:    url,
        status: resp.StatusCode,
        size:   len(body),
        err:    err,
    }
}

func main() {
    urls := []string{
        "https://example.com",
        "https://example.org",
        "https://example.net",
    }
    client := &http.Client{Timeout: 10 * time.Second}

    results := make(chan result, len(urls))
    var wg sync.WaitGroup
    start := time.Now()

    for _, url := range urls {
        wg.Add(1)
        go func(u string) {
            defer wg.Done()
            results <- fetch(u, client)
        }(url)
    }

    wg.Wait()
    close(results)

    for r := range results {
        if r.err != nil {
            fmt.Printf("%-24s error: %v\n", r.url, r.err)
            continue
        }
        fmt.Printf("%-24s %d  %d bytes\n", r.url, r.status, r.size)
    }
    elapsed := time.Since(start).Round(time.Millisecond)
    fmt.Printf("fetched %d urls in %s\n", len(urls), elapsed)
}

How it works

1. One goroutine per URL sends into a buffered channel.
2. A result struct carries status, size, or error.
3. WaitGroup then close lets the reader drain it.

Keywords and builtins used here

• chan
• close
• continue
• defer
• error
• for
• func
• go
• if
• int
• len
• make
• range
• return
• string
• struct
• type
• var

The run, in numbers

Lines

67

Characters to type

1130

Tokens

294

Three-star pace

110 tpm

At the three-star pace of 110 tokens a minute, this run takes about 160 seconds.

Type this snippet

Step 1 of 2 in Encore, step 24 of 25 in Concurrency.

← Previous Next →