A data engineer writes a nightly job that pulls 40,000 PDF filings from a regulator's document portal. It works perfectly on a laptop. In production, behind a proxy pool, it silently collapses: half the files are 2 KB HTML error pages, the process list on the build server is leaking proxy credentials, and nobody notices for eleven days because the exit code was zero the whole time.
That failure has nothing to do with the portal and everything to do with a tooling decision made in thirty seconds. curl and wget look interchangeable at the command line. Once you put a proxy in the path, add authentication, and run the job at volume, they behave like genuinely different products with different failure modes.
This is a practical comparison for teams that download data on a schedule: how each tool handles proxy configuration, what auth schemes it actually supports, how each performs under concurrency, and which mistakes cost the most bandwidth and the most debugging time.
Two Tools, Two Design Philosophies
The distinction that explains almost every behavioural difference is this: curl is a transfer tool, and wget is a retrieval agent.
curl is a thin command-line wrapper over libcurl. It speaks an unusually wide range of protocols, exposes nearly every knob of the transfer, writes to stdout by default, and treats each URL as a transfer to be executed. It does not crawl, does not build a local mirror, and does not assume it knows what you want done with the bytes.
wget is built around the idea of fetching things and keeping them. It writes to files by default, follows links recursively, respects robots.txt during recursive operations, retries persistently on its own initiative, supports timestamping so it only pulls changed files, and resumes partial transfers with a single flag.
Neither is better. They are optimised for different jobs. The problem is that proxy support, credential handling, and concurrency all fall on the curl side of that split, which is why proxy-dependent pipelines usually end up standardised on curl even when wget would be the nicer fetcher.
Proxy Compatibility Compared
How cURL Handles Proxies
curl treats the proxy as a first-class part of the transfer configuration. You can set it per invocation with -x or --proxy, and you can specify the proxy protocol explicitly in the scheme:
curl -x http://gateway.example.net:8000 https://target.example/data.jsoncurl -x socks5h://gateway.example.net:1080 https://target.example/data.json
The scheme matters more than most people realise. socks5 resolves the hostname locally and sends the resolved IP to the proxy. socks5h sends the hostname to the proxy and lets the exit node resolve it. If you are running geo-targeted collection, socks5h is usually what you want: local DNS resolution can return a CDN edge in your own region, which quietly defeats the point of routing through an exit node in another country.
Beyond the basics, curl gives you options that matter in real deployments: HTTPS proxy support with --proxy-cacert and --proxy-insecure for encrypted proxy connections, --proxytunnel to force CONNECT tunnelling for non-HTTPS requests, --preproxy to chain a SOCKS proxy in front of an HTTP proxy, --noproxy for bypass lists, and per-protocol environment variables (http_proxy, https_proxy, all_proxy, no_proxy) when you would rather not touch the command line.
How Wget Handles Proxies
wget supports proxies, but only HTTP-style ones, and almost entirely through configuration rather than flags. There is no --proxy= argument that takes a URL. You set environment variables or .wgetrc entries:
export http_proxy="http://gateway.example.net:8000"export https_proxy="http://gateway.example.net:8000"export no_proxy="localhost,127.0.0.1,.internal"wget https://target.example/data.json
The flags that do exist are narrow: --no-proxy to disable proxying for a run, and --proxy-user / --proxy-password for credentials. -e use_proxy=yes lets you inject config inline.
The critical limitation in the widely deployed GNU Wget 1.x line is the absence of SOCKS support. If your provider or your local tooling exposes a SOCKS5 endpoint, wget cannot use it natively. Teams work around this with proxychains or tsocks, which wrap the process and intercept its socket calls. That works, but you have now introduced an LD_PRELOAD shim into a scheduled production job, with all the debugging joy that implies. GNU Wget2 modernises a lot of the engine, including HTTP/2 and multi-threaded transfers, but you should verify what your specific build supports rather than assuming parity with curl.
What This Means in Practice
If your proxy access is HTTP or HTTPS with user:pass or IP whitelisting, both tools work and the choice comes down to auth and performance. If you need SOCKS5, remote DNS resolution through the exit node, chained proxies, or a custom CA for an encrypted proxy connection, curl is effectively the only option without wrapper hacks.
One more subtlety that bites people: the classic http_proxy / https_proxy split. Setting only http_proxy and then requesting an HTTPS URL means the request goes out directly, from your own datacenter IP, at full speed, looking exactly like what it is. Both tools behave this way. It is the single most common reason a "proxied" job produces clean data in staging and a block wall in production, because the staging target was HTTP and the real one was not.
Authentication Handling Compared
Proxy Authentication
curl supports Basic, Digest, NTLM, and Negotiate for proxy auth, selectable with --proxy-basic, --proxy-digest, --proxy-ntlm, --proxy-negotiate, or --proxy-anyauth to negotiate automatically. Credentials go in --proxy-user user:pass or inline in the proxy URL.
wget supports Basic proxy authentication in practice, via --proxy-user and --proxy-password or the proxy_user and proxy_password entries in .wgetrc. For the overwhelming majority of commercial proxy gateways, which use Basic over a CONNECT tunnel, that is sufficient. For corporate egress proxies running NTLM or Kerberos, it is not, and that is a real dividing line in enterprise environments.
Target-Site Authentication
This is where curl pulls clearly ahead. It handles Basic, Digest, NTLM, Negotiate, AWS SigV4 signing (--aws-sigv4), and bearer tokens (--oauth2-bearer), plus arbitrary header injection for custom schemes and API keys. Cookie jars work in both directions with -c and -b, which matters when a portal requires a login POST before the download endpoint becomes reachable.
wget offers --http-user and --http-password, with --auth-no-challenge to send Basic credentials preemptively instead of waiting for a 401. It handles cookies via --load-cookies and --save-cookies, and it can carry custom headers with --header. For token-based APIs you end up hand-rolling headers, and for anything involving a signed request you are writing the signature yourself.
For a portal that needs a session cookie plus a bearer token plus a proxy credential, curl expresses that in one line. wget expresses it in three flags and a comment explaining the workaround.
Credential Hygiene Is the Real Issue
Putting --proxy-user user:pass on a command line means the credentials are visible in ps, in /proc, in shell history, and in CI logs if the runner echoes commands. On a shared build agent, that is a credential leak with a long tail.
Safer patterns for both tools:
- Use a config file with restricted permissions.
curl --config /etc/curl/job.confreads flags from a file you canchmod 600.wgetreads.wgetrc, andWGETRClets you point at a job-specific file. - Use
.netrcfor target-site credentials. Both tools support it (curl --netrc-file,wgethonours it for HTTP auth), which keeps secrets out of argv. - Prefer environment injection from a secrets manager over literals in a crontab or a YAML pipeline definition.
- Issue per-job sub-user credentials rather than sharing one proxy account across every pipeline. When a credential leaks or a job misbehaves, you rotate one identity instead of auditing everything.
IP whitelisting removes the credential problem entirely, which is attractive for fixed-egress servers. It becomes fragile the moment your jobs run on ephemeral cloud runners with rotating public IPs, so most teams end up with whitelisting for stable infrastructure and user:pass for everything elastic.
Performance Compared
Single-Transfer Throughput
For one large file over one connection, the two are close enough that the difference is noise. Both saturate the link; the bottleneck is the proxy path, the origin server, and TLS overhead, not the client. Anyone reporting a dramatic single-file difference is usually measuring compression, HTTP version, or DNS behaviour rather than the tool.
Concurrency
This is the biggest practical gap. curl has native parallelism: -Z / --parallel with --parallel-max to cap simultaneous transfers, fed by --config files or multiple URL arguments with brace and bracket globbing.
curl -Z --parallel-max 20 -x http://user:[email protected]:8000 \ --remote-name-all "https://target.example/docs/file-[1-500].pdf"
wget 1.x has no parallel mode. You parallelise it externally, typically xargs -P or GNU parallel over a URL list. That works, but every worker is a fresh process: fresh TLS handshake, fresh proxy CONNECT, no connection reuse across the batch.
Connection Reuse Through a Proxy
Connection reuse is where proxied downloads differ most from direct ones. Every new HTTPS transfer through an HTTP proxy costs a CONNECT round trip plus a full TLS handshake to the origin. Through a residential exit node with 80 ms of added latency, that setup can easily exceed the transfer time of a small JSON file.
One curl process fetching 500 URLs from the same host reuses the tunnel and the TLS session. Five hundred wget invocations do not. At small file sizes, this single factor can change wall-clock time by an order of magnitude, and it also changes how your traffic looks to the target: a burst of hundreds of fresh handshakes is a more distinctive pattern than a sustained keep-alive session.
If you must use wget, at least pass it a URL list with -i urls.txt so a single process can reuse connections, rather than spawning one process per file.
Retries, Resumes, and Correctness
wget is genuinely better out of the box here. --tries, --waitretry, --retry-connrefused, --continue for resume, and -N for timestamp-based skipping give you a robust fetcher with almost no configuration. --mirror plus -np and -A pdf will walk a document tree and pull only what changed, which is exactly what a nightly regulatory-filings job needs.
curl can do all of it, but you have to ask. Crucially, plain --retry only retries transient network errors and a narrow set of HTTP responses. Add --retry-all-errors so that 429 and 5xx responses actually trigger a retry with backoff, and add --fail or --fail-with-body so an HTTP error is treated as a failure rather than a successful download of an error page. Without --fail, curl exits 0 after writing a CAPTCHA page to disk, which is precisely how the 2 KB PDF problem happens.
Bandwidth Is the Cost Centre
On metered proxy plans, bytes are money, so efficiency flags are budget decisions. Use curl --compressed or wget --compression=auto so text payloads travel gzipped. Use conditional requests (curl -z, wget -N) so unchanged files are not re-downloaded every night. Use curl -I or wget --spider to check size and modification headers before committing to a large transfer.
For measurement, curl has a tool wget cannot match: -w with the write-out variables.
curl -o /dev/null -s -w "dns:%{time_namelookup} connect:%{time_connect} tls:%{time_appconnect} ttfb:%{time_starttransfer} total:%{time_total}\n" \ -x http://user:[email protected]:8000 https://target.example/data.json
That breakdown tells you whether your latency is DNS, the proxy handshake, the origin, or the payload. It is the fastest way to prove that a slow job is an exit-node problem rather than a target problem. When you want to validate candidate endpoints before wiring them into a pipeline, pairing that timing output with a proxy tester gives you both the per-request detail and a quick sanity check on geolocation and reachability.
Recursive Mirroring: Where Wget Earns Its Place
If the job is "keep a local copy of this document tree in sync", wget remains the better tool, and no amount of curl flags changes that. wget -m -np -A pdf,csv --wait=2 --random-wait https://portal.example/filings/ expresses a polite, resumable, incremental mirror in one line.
Two cautions. First, wget honours robots.txt during recursive fetches, and while -e robots=off disables that, turning it off is a decision with legal and ethical weight, not a performance tweak. Make it deliberately, document it, and check the site's terms.
Second, recursion plus a rotating proxy gateway is an awkward pairing. A crawl that hops exit IPs every request looks like a distributed scrape of the same document tree, which is a strong anti-bot signal. Recursive mirroring wants session persistence: one exit IP, or a small set, held for the duration of the crawl, with deliberate delays.
Common Mistakes That Cost the Most
- Assuming
wgetspeaks SOCKS. It does not, in the common 1.x builds. Test before you architect around it. - Setting
http_proxybut nothttps_proxy. Your HTTPS traffic leaves unproxied and your real egress IP gets burned. - Resolving DNS locally for geo-targeted work. Use
socks5hor let the HTTP proxy resolve, otherwise you hit an edge node in the wrong region. - Missing
--failand--retry-all-errorsincurl. Silent success on error pages is the most expensive bug in this entire category. - Ignoring exit codes.
wgetreturns 8 for server error responses;curlreturns 22 with--fail. Wire both into your alerting. - Rotating mid-transfer. Ranged resume requests landing on a different exit IP can fail validation or trip heuristics. Large files need sticky sessions.
- Credentials in argv. Config files and
.netrc, not command lines, on any shared host. - One process per file. You pay a full proxy CONNECT and TLS handshake for every download and make your traffic pattern noisier at the same time.
Where Proxies Fit In
Neither tool controls the variable that most determines whether a scheduled download job succeeds: the identity of the IP making the request. curl and wget decide how efficiently bytes move and how errors are handled. The proxy layer decides whether the target answers at all, whether the content you get is the regional variant you intended, and whether the request survives rate limiting.
That is why pool composition matters as much as flag selection. Bulk document retrieval from tolerant public portals runs perfectly well and far more cheaply on datacenter IPs. Consumer-facing targets with real anti-bot stacks need residential or ISP addresses to get a normal response. Geo-specific catalogues, pricing, or filings require exit nodes physically located in the market you are measuring. Long mirroring sessions need sticky sessions so a crawl does not appear to come from forty different households simultaneously.
This is the gap providers like EnigmaProxy are built to fill: multiple pool types (residential, ISP, datacenter, and mobile) reachable from the same account, so a single pipeline can route cheap bulk fetches through datacenter IPs and sensitive geo-targeted requests through residential exits without maintaining separate vendor integrations.
The operational details are what make the difference in a scheduled job. Sub-user credentials let each pipeline carry its own identity, so a misbehaving job can be throttled or rotated without touching the rest. Session control decides whether you can hold an exit IP long enough for a resumable multi-gigabyte download. Broad geo-coverage across ethically sourced residential and ISP pools means regional data collection reflects what local users actually see, and predictable pricing lets you reason about the cost of a nightly mirror before you commit to it. Ethical sourcing is not a soft concern either: if your data feeds analytics, models, or regulatory reporting, the provenance of the network carrying that traffic is part of your compliance story.
Strategic Insights and Where This Is Heading
HTTP/3 changes the proxy path. QUIC runs over UDP, which the classic CONNECT tunnel was never designed for. curl already supports HTTP/3 in appropriately built binaries, and targets are increasingly offering it. Expect proxy support for UDP-based transport to become a real selection criterion rather than a footnote, particularly for latency-sensitive collection.
Shell-level fetching is being absorbed into orchestrators. More teams are moving from cron plus shell scripts to workflow engines with retry semantics, observability, and per-task credentials built in. The value of curl in that world is libcurl underneath the abstraction, and the value of a proxy provider shifts toward the quality of its API: programmatic session control, usage analytics, and sub-user provisioning matter more than a copy-paste endpoint string.
Detection is moving down the stack. TLS and TCP fingerprinting means a default curl handshake is identifiable regardless of which IP it comes from. Clients that let you tune TLS behaviour, and proxy exits whose network characteristics match the identity being presented, will keep working on targets where a stock fetch through a clean IP no longer does.
Cost discipline becomes an engineering discipline. As metered bandwidth becomes a visible line item, conditional requests, compression, and deduplication stop being micro-optimisations and start appearing in budget reviews. The teams that instrument bytes per useful record are the ones that scale collection without scaling spend proportionally.
Conclusion
Choose curl when the job involves proxies with anything beyond plain HTTP, non-Basic authentication, token or signature-based APIs, parallel transfers, or precise timing instrumentation. It is the better fit for pipelines, and its native SOCKS and per-request control make it the default for proxy-heavy work.
Choose wget when the job is a recursive, incremental mirror of a file tree over a simple HTTP proxy with Basic auth. Its timestamping, resume, and retry defaults are excellent, and rewriting them in curl is busy work.
Whatever the client, the flags that prevent silent failure are not optional: fail loudly on HTTP errors, retry on 429 and 5xx with backoff, reuse connections, and verify that every request is actually leaving through the proxy you think it is. Once that foundation is solid, the remaining variable is the network underneath, and pairing a well-configured client with a provider such as EnigmaProxy that offers pool diversity, geo-coverage, and business-grade reliability is what turns a script that works on a laptop into a download pipeline you can leave running.