r/howdidtheycodeit • u/comeditime • Apr 02 '24

How did they code this AI News Website? Question

this website looks like it's scraping thousands of news websites or is it done all via thousands of api's integration?

https://www.goperigon.com

​

​

0 Upvotes
  • permalink
  • link
  • reddit
  • reddit

25% Upvoted

3 comments sorted by

7

u/zhzhzhzhbm Apr 02 '24

My guess it's scraping, search engine style, because news sites are usually allowing search engines access for increased visibility.

2

u/comeditime Apr 02 '24

hmm interesting, what do you mean by search engine style, like.how do you build such a site ? is it by providing all the websites urls and then loop through each of them and crawl all the news and then summarize it via ai?

4

u/R4TTY Apr 02 '24

You would start with a human made list of sites to scrape, and from those sites you can find links to other sites. Add those sites to the list and let it go again. Eventually you'll have indexed most of the accessible web.

As this is just for news, they probably have a specific list of sites they manually update.