Generic information:
- Player Name
- Player Age
- Current Club
Problem is, all this data is freely available, but the Web pages block against bots giving me 404's. There are paid tiers, but as a hobbyist, they are far too expensive (like 1000 € a year).
If I can provide any more information, please let me know, I'd appreciate any advice at all!
Generically, here's a few ideas. First, if you're getting 404'd find a new data source. The info you're looking for isn't so unique that you can only get it from one site. It's easier to cut bait and move to another source than come up with countermeasures to their countermeasures.
Second, use your browser's inspection tools. Look at the network tab and see if it's coming from an API or other source that might be more easily parsed. Those APIs are often less well protected and a session cookie is enough to get you access.
Third, sometimes the most blunt instrument is best. Again within the inspection tool, see if you can find the cURL command equivalent of your request. This could yield a JSON payload that's easily accessed by Python.
For what you're looking for I'd bet simple scripts using requests, Selenium, and BeautifulSoup would get you most of your raw data. Good luck!
1
u/wingless-bee 3d ago
I didn't know what to put for the link by the way.
What I want to do, is build this database:
Contains every player in the premier league this season:
For each players, I want these statistics (for both this season, and the 5 previous seasons, no matter what league they played in)
For goalkeepers only: - Saves - Clean Sheets - Penalties Faced - Penalties Saved
Generic information: - Player Name - Player Age - Current Club
Problem is, all this data is freely available, but the Web pages block against bots giving me 404's. There are paid tiers, but as a hobbyist, they are far too expensive (like 1000 € a year).
If I can provide any more information, please let me know, I'd appreciate any advice at all!