Block SSRF to private and reserved addresses (fixes #573)

[Vitorinox]

Summary

Fixes #573.

This builds on ideas from #578 by @markfullmer (pinned DNS, HTTP(S)-only protocols, and redirects followed by the library rather than by libcurl). Thank you for that work. The check lives at request time instead of in URI resolution, which is what made relative URLs fail in that approach.

Validation runs on the absolute URI when a request is about to be sent: in CurlDispatcher / CurlClient, and in Crawler before a custom PSR-18 client. isHttp() and Extractor::resolveUri() are unchanged, except that resolveUri() now keeps the base URL’s port.

A 4.5.0 bump would match the size of the behavior change. The version is yours to pick.

Threat model

An attacker who can choose a URL Embed fetches, or who can place a URL in a page Embed will fetch next, can otherwise make the server request loopback, RFC 1918, link-local, CGNAT, and other non-public addresses. The vectors covered here are:

• the URL passed to get() / getMulti()
• HTTP redirects (301, 302, 303, 307, 308), including multi-hop, relative Location (port preserved), and protocol-relative Location
• file:// and gopher:// (and any scheme other than http / https), including as a redirect target
• DNS rebinding, by resolving once and pinning the result with CURLOPT_RESOLVE
• alternative IP spellings libcurl accepts (2130706433, 127.1, octal, hex), which CURLOPT_RESOLVE does not override
• IPv4-mapped IPv6, NAT64 (dotted and hex), and 6to4, by classifying the embedded IPv4 address
• Teredo, by rejecting the whole 2001::/32 prefix rather than reading an address out of it
• page-declared oEmbed endpoints, adapter APIs, and meta-refresh targets
• credential forwarding: Authorization and Cookie are removed when a redirect changes host

A remote JSON-LD @context is not covered. ML\JsonLD still loads it with file_get_contents and follows redirects. That is a separate change; see Follow-ups.

Design

UrlPolicy is immutable and on by default. It allows only http and https, rejects userinfo, backslashes, empty hosts, and control characters, and requires every resolved IPv4 address to be public. One non-public address rejects the lookup. IP literals, including inet_aton spellings, are classified without DNS. Internationalized names are converted with idn_to_ascii(); without ext-intl they are rejected. Failure messages stay generic so a resolved internal address is not leaked. BlockedRequestException extends InvalidArgumentException and implements PSR-18 RequestExceptionInterface. DNS failure remains NetworkException code 6.

IpClassifier matches explicit CIDR tables with inet_pton. It does not use FILTER_FLAG_GLOBAL_RANGE, which does not exist before PHP 8.2 and is a fatal error on PHP 8.0 and 8.1.

CurlDispatcher turns CURLOPT_FOLLOWLOCATION off and follows redirects itself, so every hop is validated. follow_location set to false still returns the 3xx response. max_redirs is still the number of redirects to follow (10 by default). A negative value, which curl treated as unlimited, follows at most 50, so a redirect loop cannot run without a bound. Reaching the limit is still curl error 47. The message is now curl's generic Number of redirects hit maximum amount, not Maximum (N) redirects followed.

timeout is the total time for the request and every redirect; each hop receives only the time still left. connect_timeout stays per connection. ssl_verify_host of 0 disables hostname checks. Any other value, including true, 1, and '1', is sent as 2. Passing 1 straight through used to make PHP warn and then verify; mapping it to 2 keeps the check and drops the notice. ssl_verify_peer is disabled only for a falsy value (0, false, '0', ''). Any other value, including 1, 2, 'true', and 'yes', keeps certificate verification on.

The handle is reused, so cookies set on a redirect are still sent on the next hop. CURLOPT_PROTOCOLS and CURLOPT_REDIR_PROTOCOLS are limited to HTTP and HTTPS (*_STR when the PHP build has them). The resolve entry uses the punycode host, because that is the name libcurl looks up. On PHP 8.4+, CURLOPT_PREREQFUNCTION aborts before the bytes are sent if the connected address is not one that was validated. Every version also checks CURLINFO_PRIMARY_IP and refuses the response when it does not match. curl_multi validates the whole batch before any transfer, and curl_multi_info_read is read in a loop. A redirect found after the batch has started is validated on that same thread, including DNS, so the other transfers wait until the check returns.

Content-Location remains the final URL.

A blocked oEmbed endpoint, adapter API, or meta-refresh target is skipped. Only the URL passed to get() / getMulti() throws. One blocked URL in getMulti() fails the batch before any request is sent.

Why this is not in resolveUri()

Relative references (/images/pic.jpg, //cdn.example/favicon.ico, a relative oEmbed href) are resolved long before anything is fetched, and many of them are never fetched. Checking them there rejects valid relative URLs and runs DNS for URLs the library only returns. Doing it on the absolute URI at request time leaves those results the same and checks the request that would actually leave the process.

Configuration

Private and loopback URLs throw Embed\Http\BlockedRequestException. Intranet installs can opt back in:

$embed->getCrawler()->setUrlPolicy(
    UrlPolicy::default()->allowPrivateNetworks()
);

Crawler::setUrlPolicy() copies the policy onto the client only when that client is exactly Embed\Http\CurlClient. A decorator around CurlClient is not updated. Set the policy on the inner client as well. Otherwise the wrapper is only pre-checked, and the inner client keeps the policy it already had.

withAllowedHosts('wiki.corp', '*.corp') allows named hosts (and an explicit address) without opening every private network. *.corp matches wiki.corp and a.b.corp, not the apex corp. Allowing one resolved address does not allow another address returned for the same name. withAllowedPorts([80, 443]) is optional and off by default.

If http_proxy, https_proxy, or all_proxy is set, curl connects to the proxy and the proxy resolves the origin. Embed still validates each URL, and still checks redirects, but it cannot pin DNS or see the address the proxy used.

Backward compatibility

Public signatures of Embed, Crawler, CurlClient, isHttp(), and resolveUri() are unchanged. The new API is additive: UrlPolicy, BlockedRequestException, Crawler::setUrlPolicy() / getUrlPolicy(), and the same on CurlClient.

The behavior change is intentional: non-public targets are rejected unless the caller opts out. follow_location, ignored_errors, and CURL_IPRESOLVE_V4 are kept. resolveUri() now preserves the base port (/x against http://h:8080/a is http://h:8080/x).

Differences from the previous curl follower:

• max_redirs < 0 is capped at 50 instead of being truly unlimited.
• Error 47's message is Number of redirects hit maximum amount.
• timeout still bounds the whole chain, not each hop.
• ssl_verify_host values other than 0 enable hostname verification, including true and 1.
• ssl_verify_peer is disabled only for falsy values (0, false, '0', ''). 1, 2, 'true', and 'yes' keep certificate verification on.

Custom PSR-18 clients

Crawler validates before sendRequest / sendRequests and does not validate again inside that client. It also does not double-resolve when the client is CurlClient.

What a custom client still has to handle:

• DNS can change between the check and the connection.
• Redirects the client follows are not checked, unless the client exposes a hook. The README has a Guzzle on_redirect example. That hook still cannot see the connected address.

Prefer the built-in client for untrusted pages. Extracted URLs (image, favicon, and the rest) are not downloaded; call getUrlPolicy()->validate() before fetching one yourself.

Tests

Offline and deterministic. The classifier and the policy assert both the exception type and getReason(). Integration tests run against a local php -S for redirects, relative and protocol-relative Location, forbidden schemes, the redirect limit, a negative max_redirs, a timeout that spans hops, hostname verification (0 / false / '0' versus true / 1 / '1' / 2, accepting both curl wordings of the hostname mismatch), certificate verification for falsy and non-falsy ssl_verify_peer, cookies across hops, Authorization stripped on a host change, parallel requests, a batch rejected before any request, and DNS rebinding (the resolver’s second answer is never used). A stub PSR-18 client checks that relative og:image, a protocol-relative favicon, hreflang, and a relative oEmbed endpoint match the previous results. PagesTest, DocumentTest, and MultipleRequestsTest use a fake resolver that returns a public address, so they do not depend on live DNS.

Tested locally on PHP 7.4, 8.3, 8.4 and 8.5; 8.0–8.2 rely on CI.

Follow-ups

• A separate PR should stop ML\JsonLD from fetching remote @context documents (including a page-supplied URL that redirects internally). That change also explains the existing Tumblr publishedTime fixture, which is null because the schema.org fetch failed when the fixture was recorded. This PR does not change LinkedData or that fixture.

[Vitorinox]

Hi @markfullmer, can you try this?

[markfullmer]

Testing and reviewing now, @Vitorinox ....

[markfullmer]

This approach is comprehensive, intervening in the correct location in the code to preemptively catch any look-ups. In addition to feeling that the test coverage added here is comprehensive, I also ran this change through an integration I have of embed/embed, where I set up integration tests to check for the SSRF vulnerability. As expected, those tests would fail on the current release of embed/embed, while they pass using the code from this pull request.

This approach has my full approval!

[markfullmer]

Just a suggestion, too: I think this should be included in a semver patch-level release (i.e., 4.4.20). This will ensure that applications using this library that have a patch-level floating version constraint in Composer will be able to update automatically, without having to manually modify their requirement on embed/embed.