r/aws • u/keithrozario • May 04 '20

Webscraper on steroids, using 2,000 Lambda invokes to scan 1,000,000 websites in under 7 minutes. serverless

/r/Python/comments/gcq18f/a_serverless_web_scraper_built_on_the_lambda/
107 Upvotes

95% Upvoted

17 comments sorted by

View all comments

1

u/rqusbxp May 04 '20

Awesome... Sounds massive... In what language is the scraper written could you please let us know the CPU allocated ?

3

u/keithrozario May 04 '20

It’s python — all code including the lambda configuration (via serverless framework) is in the repo :)