Repository navigation
Vulnerablecode useragent #410
Description
Activity
@Hritik14 thanks!
I am split on this one.- on the one hand, I want to be well behaved and polite with our own agent string
- on the hand hand, condoning web sites that prohibits certain agents like mattermost without a good reason would go entirely against our mission
I think we should have this may:
- a default of our own defined user agent such as
VulnerableCode - an alternative way to provide a config with a list of other user agents we could use (and rotate through)
So may be the best way is to have a config TBD that provides a list of user agent strings and would default to
VulnerableCodecondoning web sites that prohibits certain agents
I think we should contact the providers and have their concent about the scraping (maybe we can even maintain a document as proofs of consents for later use). I'm confident most of them would agree and would not blacklist our own defined user agent.
best way is to have a config TBD that provides a list of user agent strings and would default to VulnerableCode
This sounds good to me too.
Hi @pombredanne 👋
I’d like to work on this issue.I will start by adding a default User-Agent for all web requests
and make it configurable via environment variables.Thanks!
Hi @pombredanne, I would like to take this issue and work on it. I will analyze everything and comes with a PR ready soon.
Much of what we're doing now is scraping the webpages for the required information. Doing such activity without even declaring who we are might not be seen as a very good practice.
We need to settle on a useragent choice. Some websites even block regular scraper useragents. There are two solutions I can think of:
There could exist benefit of either methods but I'm not so sure.
References:
https://forum.mattermost.org/t/mattermost-website-returning-403-when-headers-contain-the-word-python/11412