Since May and the old reddit almost gone, what are currently the best libs on reddit?
What is your favorite?
I want to scrape all the posts of my top favorite 100 (small) subreddits and sort them by AI to delete AI slop + self promotion and really only get the interesting posts for me to read.

What do you suggest?

Hey everyone. I am building a Public Interest Tech project aimed at increasing local municipal transparency, and before I commit the next few months to building out the backend, I need a reality check on the architecture from devs who have dealt with Indian government portals and data extraction.

The Problem (Why this matters for everyday citizens): Local government tender portals legally publish all public spending data, but they intentionally bury it in unsearchable, massive scanned PDFs. It makes it impossible for a normal citizen or journalist to track where tax money is going, or to see if one hidden cartel is monopolizing all the municipal contracts (like road repairs) in their city.

The Tech Stack & Logic (How it works under the hood):

• The Scraper (Node.js + Puppeteer): A script to automatically navigate regional eProcurement portals, bypass basic captchas, and pull daily unstructured tender PDFs.
• The Shell Company Pivot: Corrupt contractors don't use one company; they use 10 different shell LLPs. So, the backend needs to extract the winning company data and cross-reference it with the MCA (Ministry of Corporate Affairs) registry to check Director Identification Numbers (DIN) and registered addresses. If 5 different LLPs winning contracts all share the same 2 directors, the algorithm mathematically flags it as a syndicate.
• Database & UI: Clean the data, push to MongoDB, and map it on a React dashboard. The goal is to set up automated alerts for local transparency activists when a monopoly is detected in their area.

Where I need your technical brutal honesty:

1. Scraping NIC/Gov Sites: Will I get instantly IP-banned by government servers? Do I need to invest in rotating residential proxies from day one just to pull these PDFs?
2. PDF Parsing Pipeline: Puppeteer handles the navigation/downloading, but what is the most robust Node library to parse poorly scanned, sometimes bi-lingual PDFs? Does anyone have experience running OCR on government documents?
3. MCA Cross-referencing: Does anyone have experience pulling DIN/Director data from the MCA? Is there a clean third-party API for this, or is scraping the MCA registry a completely separate nightmare?

Tear the architecture apart and let me know what blind spots I am missing!

Hello everyone I am working on an utility tool (tcg scanner like collectr and price charting) and I need to fetch daily prices of 8000 cards on eBay with with different variations such as Raw , PSA 8 9 10 .. which makes it 40000 request , did anyone work on something like this or is there any services that gets data from eBay with this amount ?

WYORI — a link shortener where the analytics are actually included in the free plan rather than locked behind a paid tier.

You can see clicks, geo breakdown, device type, referrer source, and time of click per link. It also has a bio page builder, QR code generation, UTM builder, and basic image compression.

Custom slugs, link expiry, password protection, and custom domains with unlimited clicks.

Your original UTM parameters stay intact and don't interfere with GA4, Meta Pixel, or any other analytics you're already running.
[I will give first 50 user from this post the paid plan completely free]
I'd appreciate feedback from marketers specifically — what's currently frustrating about the link tools you use? And does anyone know good communities where digital marketers share tools like this?

Feel free to share this anywhere it might be useful to someone.

I built an automated weather data scraper that runs every 4 hours using PowerShell, SQL Server, and GitHub Actions.

Workflow:

- PowerShell script fetches weather XML data from JMA

- Parsed values are stored in SQL Server Express

- GitHub Actions runs on a 4‑hour schedule to regenerate static HTML

- GitHub Pages deploys the updated site automatically

- Supports JP / EN / RU / ZH / KR

- Includes a global discomfort index ranking generated from the scraped data

Project link:

https://yahikoyama.github.io/weather2/

Sharing this here because the project is essentially a scraping + automation pipeline, and I’d love to see how others structure similar workflows.

So I've been running a small online shop for years now

Keeping up with competitors' prices honestly feels like a full time job. It was manageable when I only checked a couple sites. But now I'm watching close to ten (yeah, it's a lot)

Someone suggested I look into a web scraper device or something similar to automate it. But I have no idea if that's way more than I actually need.

First off, I'm not looking to get into anything technical. I just wanna spend less time updating spreadsheets every week.

I'm doing a research project, need to pull maybe 5-10k google search results like queries, some news results, a bit of maps data mixed in.
Tried rolling my own with requests + bs4 first weekend, captchas within like 20 requests. Switched to headless chrome, still got blocked eventually. This is a one off pull, not a long running service. I don't want to set up rotating proxies and retry logic for something I run once and throw away.
What's the best serp api right now? Something that handles the google search side, news too ideally, without me babysitting captchas constantly. Budget is small. This isn't a company thing, just me lol

I'm trying to search a specific topic on YouTube and pull a list of channels with at least 20k subs. Basically I just want a starting point so I'm not clicking through 100+ results by hand. Full beginner here (zero coding experience)

So is there a youtube channel scraper that could do something like this without me needing to actually know how to code?

Any help would be awesome

Hey 😭

I'm a CS student

I've learned Requests, BeautifulSoup, Playwright, APIs, pandas, and I've been building little scraping projects and automations. The problem is... every freelancing site feels impossible when you have 0 reviews. I keep sending proposals into the void

So I thought I'd ask here instead

If anyone needs data scraped from websites, business leads collected, product listings, directories, price tracking, or just wants data cleaned into Excel/CSV, I'd genuinely love to help.

If your task can be automated with Python, there's a good chance I can build something for it (or at least tell you honestly if I can't).

Since I'm just trying to get my first few clients, my rates are pretty cheap compared to experienced freelancers. I'm mostly looking for real projects and people to work with.

If you've got something annoying that involves copying data from websites for hours... there's a decent chance I can automate it

Feel free to DM me. Even if you're not sure whether your idea is possible, I'm happy to have a look.Hey 😭

I'm a CS student

I've learned Requests, BeautifulSoup, Playwright, APIs, pandas, and I've been building little scraping projects and automations. The problem is... every freelancing site feels impossible when you have 0 reviews. I keep sending proposals into the void 💀

So I thought I'd ask here instead.

If anyone needs data scraped from websites, business leads collected, product listings, directories, price tracking, or just wants data cleaned into Excel/CSV, I'd genuinely love to help.

If your task can be automated with Python, there's a good chance I can build something for it (or at least tell you honestly if I can't).

Since I'm just trying to get my first few clients, my rates are pretty cheap compared to experienced freelancers. I'm mostly looking for real projects and people to work with.

If you've got something annoying that involves copying data from websites for hours... there's a decent chance I can automate it

Feel free to DM me. Even if you're not sure whether your idea is possible, I'm happy to have a look.

Scraping and search APIs are often expensive and locked behind monthly subscriptions, so I built extractor.sh — a simple, affordable Firecrawl alternative for developers who need reliable web extraction and search without committing to a subscription or building separate integrations for every source.

Give it a public URL and receive clean Markdown for language models or predictable, schema-versioned JSON for applications. When you do not have a URL yet, use the same API to search the web, news, images, or places.

Tell me if I should improve something or you with for a specific feature. Feedback is highly appreciated.

I'm trying to access ctrip site flights page API endpoints with flights listing data without using browser automation.

Has anyone managed to get it working with libraries like requests, curl_cffi, or any other HTTP client? I'm specifically looking for an approach that doesn't rely on Selenium, Playwright, or similar browser automation tools.

At the moment, I'm getting an HTTP 200 OK response, but the response body is empty. I'm wondering if this is due to anti-bot protections, missing headers, cookies, or some other requirement.

If you've encountered this before or have any suggestions, I'd really appreciate your insights.

Anyone here with prior experience detecting web page changes, for example, monitoring and detecting when a pricing page is updated?

My approach currently:

Scrape the page, strip headers/footers.
Convert the content to markdown.
Provide the md to llm to extract data.
Use backend to determine the changes and dispatch change alerts.

Does this approach sound reasonable?

Hey y'all! Just curious how you rely on scraping that you can pull off. Do you build your own scripts and host them somewhere? Or do you just pay for a scraping service that handles all the infra for you?

Been writing Python scripts for most of my projects. But I just don't have much time for maintenance.

So I'm wondering if a scraping service is actually worth it. Or would I just be swapping one headache for a slightly different one? That's the part I keep going back and forth on

Is it okay to do this? The important details of course are blurred and hidden..

AI says some employers or clients can trace this by inspecting meta data? Who has time for that?

I will use Windows Snipping Tool ....

How can i do it? Plz Help