Block SSRF to private and reserved addresses (fixes #573)

CHANGELOG.md

@@ -4,6 +4,21 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/)
 and this project adheres to [Semantic Versioning](http://semver.org/).
 
+## [Unreleased]
+### Security
+- Block requests to non-public addresses before they are sent, including redirects, page-declared oEmbed endpoints, adapter APIs, and meta-refresh targets. Private and loopback URLs are rejected unless `UrlPolicy::allowPrivateNetworks()` or `withAllowedHosts()` is used [#573]
+
+### Added
+- `Embed\Http\UrlPolicy` and `Embed\Http\BlockedRequestException`, configurable from `Crawler::setUrlPolicy()`
+
+### Changed
+- A negative `max_redirs`, which curl treated as unlimited, now follows at most 50 redirects
+- Reaching that limit is still curl error 47. The message is `Number of redirects hit maximum amount` rather than curl's `Maximum (N) redirects followed`
+
+### Fixed
+- `resolveUri()` dropped the port of the base URL
+- Read every `curl_multi_info_read` message in the parallel client instead of only one per iteration
+
 ## [4.4.19] - 2026-07-08
 ### Changed
 - Access tokens are now optional with Meta (Facebook & Instagram) [#577]
@@ -299,6 +314,7 @@ Full library refactoring.
 [#569]: https://github.com/oscarotero/Embed/issues/569
 [#571]: https://github.com/oscarotero/Embed/issues/571
 [#572]: https://github.com/oscarotero/Embed/issues/572
+[#573]: https://github.com/oscarotero/Embed/issues/573
 [#574]: https://github.com/oscarotero/Embed/issues/574
 [#577]: https://github.com/oscarotero/Embed/issues/577
 

README.md

@@ -329,12 +329,12 @@ $client = new CurlClient();
 $client->setSettings([
     'cookies_path' => $cookies_path,
     'ignored_errors' => [18],
-    'max_redirs' => 3,               // see CURLOPT_MAXREDIRS
-    'connect_timeout' => 2,          // see CURLOPT_CONNECTTIMEOUT
-    'timeout' => 2,                  // see CURLOPT_TIMEOUT
-    'ssl_verify_host' => 2,          // see CURLOPT_SSL_VERIFYHOST
-    'ssl_verify_peer' => 1,          // see CURLOPT_SSL_VERIFYPEER
-    'follow_location' => true,       // see CURLOPT_FOLLOWLOCATION
+    'max_redirs' => 3,               // redirects to follow; a negative value means up to 50
+    'connect_timeout' => 2,          // per connection, see CURLOPT_CONNECTTIMEOUT
+    'timeout' => 2,                  // total seconds for the request, including redirects
+    'ssl_verify_host' => 2,          // 0 disables hostname checks; any other value enables them
+    'ssl_verify_peer' => 1,          // falsy disables certificate checks; any other value enables them
+    'follow_location' => true,       // false returns the 3xx; each hop is checked
     'user_agent' => 'Mozilla',       // see CURLOPT_USERAGENT
 ]);
 
@@ -359,6 +359,89 @@ $info = $embed->get($url);
 
 Note: The built-in detectors does not require settings. This feature is only for convenience if you create a specific detector that requires settings.
 
+## Security / SSRF protection
+
+The default HTTP client refuses to connect to non-public addresses. That covers the URL you pass to `get()` / `getMulti()` and every later request Embed makes itself (redirects, oEmbed endpoints declared by the page, adapter APIs, and meta-refresh targets). A hostname is allowed only when every IPv4 address it resolves to is public. Loopback, private, link-local, CGNAT, documentation, and multicast ranges are rejected. IPv4-mapped IPv6, NAT64, and 6to4 are rejected when the embedded IPv4 address is not public. The Teredo prefix `2001::/32` is rejected as a whole, not by reading an address out of it. Only `http` and `https` are allowed, and redirects are followed inside Embed so each hop is checked the same way.
+
+`follow_location` set to false still returns the 3xx response. `max_redirs` is still how many redirects are followed (10 by default). A negative value, which curl treated as unlimited, follows at most 50. `timeout` is the total time for the request and its redirects; `connect_timeout` is still per connection. `ssl_verify_host` of `0` disables hostname checks, and any other value enables them, as curl does when it receives `1` or `true`. `ssl_verify_peer` is off only for a falsy value (`0`, `false`, `'0'`, `''`); `1`, `2`, `'true'` and `'yes'` keep certificate checks on. Reaching the redirect limit still fails with curl error 47. The message is `Number of redirects hit maximum amount`.
+
+This is a behavior change: a URL that used to be fetched from an intranet now throws `Embed\Http\BlockedRequestException`. To opt back in:
+
+```php
+use Embed\Http\UrlPolicy;
+
+$embed->getCrawler()->setUrlPolicy(
+    UrlPolicy::default()->allowPrivateNetworks()
+);
+```
+
+You can allow specific names without opening every private network. `*.corp` matches any host that ends in `.corp`. Allowing one resolved address does not allow the other addresses returned for that name:
+
+```php
+$embed->getCrawler()->setUrlPolicy(
+    UrlPolicy::default()->withAllowedHosts('wiki.corp', '*.corp')
+);
+```
+
+`withAllowedPorts([80, 443])` is optional. When it is set, an omitted port counts as 80 or 443. Internationalized host names are converted with `idn_to_ascii()`, which needs the `intl` extension; without it those hosts are rejected. The built-in client pins the punycode form, because that is the name libcurl looks up.
+
+Values such as `image` or `favicon` are URLs extracted from the page. Embed does not download them, so it does not apply this policy to them. Check one before you fetch it yourself:
+
+```php
+$policy = $embed->getCrawler()->getUrlPolicy();
+$image = $info->image;
+
+if ($image !== null) {
+    $policy->validate($embed->getCrawler()->createRequest('GET', (string) $image));
+}
+```
+
+A page-declared oEmbed endpoint, adapter API, or meta-refresh target that the policy rejects is skipped, and extraction continues with the rest of the page. The URL passed to `get()` or `getMulti()` is not skipped: that request throws. In `getMulti()`, one rejected URL rejects the whole batch before any request is sent. A redirect discovered after the batch has started is checked on the same thread, including its DNS lookup, so the other transfers in that batch wait until the check returns.
+
+`Crawler::setUrlPolicy()` copies the policy onto the client only when that client is exactly `Embed\Http\CurlClient`. A decorator or other wrapper around `CurlClient` is not updated. Call `setUrlPolicy()` on the inner client as well. Otherwise the wrapper is pre-checked with the crawler's policy, and the inner client keeps the policy it already had.
+
+### Custom PSR-18 clients
+
+If you pass your own PSR-18 client, Embed validates the URI before calling it and does not validate again inside that client. Two things are then outside Embed's control:
+
+* DNS can change between the check and the connection (rebinding).
+* Redirects followed by the client are not checked, unless the client gives you a hook.
+
+The built-in curl client pins the validated addresses with `CURLOPT_RESOLVE` and follows redirects itself, so those gaps do not apply to it. With Guzzle you can repeat the check on each redirect:
+
+```php
+use Embed\Http\UrlPolicy;
+use GuzzleHttp\Client;
+use GuzzleHttp\HandlerStack;
+use GuzzleHttp\Middleware;
+use Psr\Http\Message\RequestInterface;
+use Psr\Http\Message\ResponseInterface;
+use Psr\Http\Message\UriInterface;
+
+$policy = UrlPolicy::default();
+$stack = HandlerStack::create();
+$stack->push(Middleware::mapRequest(function (RequestInterface $request) use ($policy) {
+    $policy->validate($request);
+
+    return $request;
+}));
+
+$client = new Client([
+    'handler' => $stack,
+    'allow_redirects' => [
+        'on_redirect' => function (RequestInterface $request, ResponseInterface $response, UriInterface $uri) use ($policy): void {
+            $policy->validate($request->withUri($uri));
+        },
+    ],
+]);
+```
+
+Even that hook cannot see the address the client actually connected to. Prefer the built-in client when the pages are untrusted.
+
+### Proxies
+
+If `http_proxy`, `https_proxy`, or `all_proxy` is set, curl connects to the proxy and the proxy resolves the origin. Embed still validates each URL before requesting it, and still checks redirects, but it cannot pin DNS or verify the address the proxy used. Point those variables only at a proxy you trust.
+
 ---
 
 ## Testing

composer.json

@@ -47,6 +47,7 @@
         "phpstan/phpstan-strict-rules": "^2.0"
     },
     "suggest": {
+        "ext-intl": "To resolve internationalized domain names when protecting against SSRF",
         "symfony/css-selector": "If you want to get elements using css selectors"
     },
     "autoload": {

phpunit.xml.dist

@@ -2,7 +2,7 @@
 <phpunit
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/9.4/phpunit.xsd"
-    bootstrap="vendor/autoload.php"
+    bootstrap="tests/bootstrap.php"
     backupGlobals="false"
     backupStaticAttributes="false"
     colors="true"

src/Embed.php

@@ -3,6 +3,7 @@
 
 namespace Embed;
 
+use Embed\Http\BlockedRequestException;
 use Embed\Http\Crawler;
 use Psr\Http\Message\RequestInterface;
 use Psr\Http\Message\ResponseInterface;
@@ -88,7 +89,11 @@ private function extract(RequestInterface $request, ResponseInterface $response,
         }
 
         $request = $this->crawler->createRequest('GET', (string) $redirectUri);
-        $response = $this->crawler->sendRequest($request);
+        try {
+            $response = $this->crawler->sendRequest($request);
+        } catch (BlockedRequestException $exception) {
+            return $extractor;
+        }
 
         return $this->extract($request, $response, false);
     }

src/Http/BlockedRequestException.php

@@ -0,0 +1,45 @@
+<?php
+declare(strict_types = 1);
+
+namespace Embed\Http;
+
+use InvalidArgumentException;
+use Psr\Http\Client\RequestExceptionInterface;
+use Psr\Http\Message\RequestInterface;
+
+/**
+ * The request was refused by the URL policy before a connection was used.
+ *
+ * The message is generic on purpose: it must not include a resolved address,
+ * so a caller cannot use the failure as a DNS oracle.
+ */
+final class BlockedRequestException extends InvalidArgumentException implements RequestExceptionInterface
+{
+    private RequestInterface $request;
+
+    /** @var 'scheme'|'host'|'non_public_address'|'port'|'redirect' */
+    private string $reason;
+
+    /**
+     * @param 'scheme'|'host'|'non_public_address'|'port'|'redirect' $reason
+     */
+    public function __construct(string $message, RequestInterface $request, string $reason)
+    {
+        parent::__construct($message);
+        $this->request = $request;
+        $this->reason = $reason;
+    }
+
+    public function getRequest(): RequestInterface
+    {
+        return $this->request;
+    }
+
+    /**
+     * @return 'scheme'|'host'|'non_public_address'|'port'|'redirect'
+     */
+    public function getReason(): string
+    {
+        return $this->reason;
+    }
+}

src/Http/Crawler.php

@@ -15,6 +15,7 @@ class Crawler implements ClientInterface, RequestFactoryInterface, UriFactoryInt
     private RequestFactoryInterface $requestFactory;
     private UriFactoryInterface $uriFactory;
     private ClientInterface $client;
+    private UrlPolicy $urlPolicy;
     /** @var array<string, string> */
     private array $defaultHeaders = [
         'User-Agent' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:73.0) Gecko/20100101 Firefox/73.0',
@@ -26,6 +27,25 @@ public function __construct(?ClientInterface $client = null, ?RequestFactoryInte
         $this->client = $client !== null ? $client : new CurlClient();
         $this->requestFactory = $requestFactory !== null ? $requestFactory : FactoryDiscovery::getRequestFactory();
         $this->uriFactory = $uriFactory !== null ? $uriFactory : FactoryDiscovery::getUriFactory();
+        $this->urlPolicy = $this->client instanceof CurlClient ? $this->client->getUrlPolicy() : UrlPolicy::default();
+    }
+
+    /**
+     * The policy is copied onto the HTTP client only when that client is
+     * exactly CurlClient. A decorator around CurlClient is not detected;
+     * set the policy on the inner client as well.
+     */
+    public function setUrlPolicy(UrlPolicy $urlPolicy): void
+    {
+        $this->urlPolicy = $urlPolicy;
+        if ($this->client instanceof CurlClient) {
+            $this->client->setUrlPolicy($urlPolicy);
+        }
+    }
+
+    public function getUrlPolicy(): UrlPolicy
+    {
+        return $this->urlPolicy;
     }
 
     /**
@@ -57,6 +77,8 @@ public function createUri(string $uri = ''): UriInterface
 
     public function sendRequest(RequestInterface $request): ResponseInterface
     {
+        $this->validateCustomClient($request);
+
         return $this->client->sendRequest($request);
     }
 
@@ -69,12 +91,25 @@ public function sendRequests(RequestInterface ...$requests): array
             return $this->client->sendRequests(...$requests);
         }
 
+        foreach ($requests as $request) {
+            $this->urlPolicy->validate($request);
+        }
+
         return array_map(
             fn ($request) => $this->client->sendRequest($request),
             $requests
         );
     }
 
+    private function validateCustomClient(RequestInterface $request): void
+    {
+        if ($this->client instanceof CurlClient) {
+            return;
+        }
+
+        $this->urlPolicy->validate($request);
+    }
+
     public function getResponseUri(ResponseInterface $response): ?UriInterface
     {
         $location = $response->getHeaderLine('Content-Location');

src/Http/CurlClient.php

@@ -14,12 +14,14 @@
 final class CurlClient implements ClientInterface
 {
     private ResponseFactoryInterface $responseFactory;
+    private UrlPolicy $urlPolicy;
     /** @var array<string, mixed> */
     private array $settings = [];
 
     public function __construct(?ResponseFactoryInterface $responseFactory = null)
     {
         $this->responseFactory = $responseFactory !== null ? $responseFactory : FactoryDiscovery::getResponseFactory();
+        $this->urlPolicy = UrlPolicy::default();
     }
 
     /**
@@ -30,9 +32,19 @@ public function setSettings(array $settings): void
         $this->settings = $settings + $this->settings;
     }
 
+    public function setUrlPolicy(UrlPolicy $urlPolicy): void
+    {
+        $this->urlPolicy = $urlPolicy;
+    }
+
+    public function getUrlPolicy(): UrlPolicy
+    {
+        return $this->urlPolicy;
+    }
+
     public function sendRequest(RequestInterface $request): ResponseInterface
     {
-        $responses = CurlDispatcher::fetch($this->settings, $this->responseFactory, $request);
+        $responses = CurlDispatcher::fetch($this->dispatchSettings(), $this->responseFactory, $request);
 
         return $responses[0];
     }
@@ -42,6 +54,17 @@ public function sendRequest(RequestInterface $request): ResponseInterface
      */
     public function sendRequests(RequestInterface ...$request): array
     {
-        return CurlDispatcher::fetch($this->settings, $this->responseFactory, ...$request);
+        return CurlDispatcher::fetch($this->dispatchSettings(), $this->responseFactory, ...$request);
+    }
+
+    /**
+     * @return array<string, mixed>
+     */
+    private function dispatchSettings(): array
+    {
+        $settings = $this->settings;
+        $settings['url_policy'] = $this->urlPolicy;
+
+        return $settings;
     }
 }