[RaphaelFakhri]

Description

• RobotsTxtFile.load decodes the response body with utf-8, which keeps a leading UTF-8 byte order mark (BOM) as \ufeff. Protego then does not recognize the first line (\ufeffUser-agent: *) as a User-agent directive, so the rules of the first group are ignored and is_allowed returns True for disallowed paths. With respect_robots_txt_file=True, the crawler fetches pages the site disallows.
• The body is now decoded with utf-8-sig, which drops the BOM if present and is otherwise identical to utf-8.
• Adds a regression test that loads a robots.txt with a BOM through the test server.

Issues

• Closes: robots.txt rules are ignored when the file starts with a UTF-8 byte order mark #2350

Testing

• New test test_load_robots_txt_with_utf8_bom in tests/unit/_utils/test_robots.py. It fails on master (assert not robots.is_allowed(...) gets True) and passes with the fix, for all HTTP client parametrizations.
• uv run pytest tests/unit/_utils/test_robots.py: 20 passed.
• ruff check, ruff format --check, and ty check pass on the changed files.

Checklist

• CI passed

[Mantisus]

LGTM

[apify-approval-mirror]

Mirroring approval from @Mantisus.