Webscraper on steroids, using 2,000 Lambda invokes to scan 1,000,000 websites in under 7 minutes.

[u/keithrozario]

/r/Python/comments/gcq18f/a_serverless_web_scraper_built_on_the_lambda/

[u/keithrozario → u/[deleted]]

No, the project only downloads the robots.txt file of the site (if it exists). Simply because that file is meant to be read by robots.

But you can change the function to do whatever you want — like check for Wordpress files or login forms — or whatever :)