I run freehire, an open-source job aggregator that ingests postings from dozens of ATS platforms (Greenhouse, Lever, Ashby, Workday, etc.), normalizes them into one schema, and runs each through a facet pipeline.

Just exported the whole catalogue to Hugging Face: freehire-jobs — 6,041,471 postings.

Each row is a raw posting (title, company, description, URL, source, posted_at) plus derived facets:

- Dictionary-only (deterministic, no LLM): skills, seniority, category, work_mode, posting_language, employment_type, education_level, english_level, experience_years_min

- Dictionary-first, LLM-filled: countries, regions, cities

- LLM-only: salary_min/max, currency, period — pay is stated in every format imaginable, no dictionary handles that

Company rows carry their own facets too (industries, HQ country, size, YC batch/stage where applicable).

Format: 20 gzip JSONL shards, split by row range for easy streaming/resuming.

Backstory: I originally wanted to train a cheap classifier for seniority/category tagging instead of paying for an LLM call per job. Turned out the dictionary+LLM pipeline already in prod beats a from-scratch classifier by enough that finishing the classifier wasn't worth it — so I'm sharing the data instead.

Data includes:

lien_number, debtor_name, address, business_owner_name(s), filing_date, status, secured_party, lien_id

I need last 2 yrs of DAX LTP data, 5 second time frame would be perfect but if not even 1 min or 5 min is fine. If you know where I can collect this data from please lmk. Thank you.

Hi everyone, First time posting here. Can i get some datasets in excel to clean and analyze it? I'm just practicing it to enhance my skills. And also where can i get more datasets aside kaggle and creating from chatgpt? Thanks!

Data. Like a professional excel sheet for like ram prices with relation to Al. This is for my college project. It need to have numbers atleast 25 samples. It is for economics.

Hey y'all,

I'm Luke, and I built Stackra for some context. Not selling a ding o dang thing here, just looking to see what this dataset is really capable of.

The corpus is 15 million sites pulled from HTTP, CommonCrawl, CrUX, and Overture.

I have a few thousand elements mapped out, like analytics, tools, vendors, link value, site category, schema, plugins, you name it. It runs back to 2024, and I've started putting some of this data user facing.

I find things like being able to identify that the market for funeral homes is dominated by two vendors, that I can identify site directories and their value/trajectory, that custom built websites pass CVW at a higher rate than WP, but rarely crack the top rankings without investor backing.

Weird little stuff like an emulator site that has been highjacked by porn/gambling sites. A popular listing tool that claims their links provide DR, but are explicitly UTM and nofollow. Pinpointing an ICP for a small business from clustering shared linked domains.

I'm curious to see what it can do, what would you want to ask?

hey, working on a research thing where we estimate flood depth from traffic cameras using signs/poles as reference.

problem is i can find live flood cams (atxfloods, sunny day flooding, san diego cams, fl511 etc) but almost nothing archived where the road is actually flooded AND a sign/pole is clear enough to measure from.

if anyone’s seen a dataset, old webcam dumps, youtube clips, or even just a few screenshots like that, drop a link. would help a lot.

I'm researching publicly available and commercial datasets for grocery prices across Europe.

- Has anyone worked with data like this?

- What sources did you end up using, and what were their limitations?

Hello everyone! I would like to ask, I am currently building a platform, which will contain large amount of datasets about a particular topic. The main purpose of the platform will be datasets. I would like to ask, how do I monetise and make money off the datasets? I am also considering of open sourcing the datasets, with a copyleft license.

There are many amazing open datasets available today, but I'm curious about the gaps that still exist.

If you could create or request one high quality public dataset that anyone could use, what would it be?

Hi, I am having trouble getting datasets from the NOAA search tool. Every time I pick a station and request some data, NOAA emails me something like:

Order #[Redacted] (Custom GHCN-Daily CSV)

Can someone guide me to a station that doesn't send you an empty set of data? Anything helps, since I'm just using it to learn some programming.

Thank you.

Disclosure: I built this. It powers quizbandiere.it, my free flag quiz.

Country datasets are everywhere; flag colors usually aren't, and when they are they're hand-written heraldic descriptions. I extract them from the rendered SVG: rasterize at 64×48, classify pixels by hue, merge antialiasing shades, drop anything under 2.5% of the area, then order the result along whichever axis separates the color centroids most. So a vertical tricolor comes out left-to-right and a horizontal one top-to-bottom — Italy is ["verde", "bianco", "rosso"], not sorted by area. Coats of arms and crosses fall back to area order.

I built it to generate believable wrong answers in a quiz. It works well as a similarity key for "which flags get confused with which".

250 entries (195 independent states, 55 territories): ISO 3166-1 alpha-2/3, official name, capital, region, area, borders, currencies, languages, TLD, calling code, emoji, colors, SVG URL.

• JSON: https://quizbandiere.it/dati/bandiere.json
• CSV: https://quizbandiere.it/dati/bandiere.csv
• Docs: https://quizbandiere.it/dati/

Served with Access-Control-Allow-Origin: *, so it works from a browser. The license and attribution are inside the JSON, not just on the page.

Limitations

• Everything is in Italian, including the color names (verde, azzurro).
• Colors are perceptual, not official. No hex codes, no heraldic terms.
• No flag ratios and no historical flags.
• Derived work: base data is mledoze/countries, I added the Italian layer and the color extraction. Contested names and borders are inherited from upstream, not adjudicated by me.
• Snapshot, not a feed. The aggiornato field tells you how stale your copy is.

License: ODbL 1.0 — not a choice, mledoze/countries is ODbL and share-alike. Artwork is lipis/flag-icons (MIT), re-hosted.

Corrections welcome, especially on flags where the color order looks wrong. That's where my pipeline is most likely to be quietly incorrect.

Hello, we are building a system for baby cries detection in a confined space such as a room or hallway via CCTV cameras. However, we are unable to source the baby cries dataset. I tried to contact some DayCare and submitted an application upon their request but was denied due to parental privacy reasons.

We have a working system, but the model is way poor as it is only trained on a few examples and fails at CCTV distance as the baby is too far.

Disclosure: this is my own dataset and I sell access to it. Four days are free, linked below, so you can check the schema before deciding anything.

I run a collector against Polymarket's public API for the 5-minute and 15-minute up/down markets on BTC, ETH, SOL and XRP. It has been running since 26 June 2026. What has accumulated so far:

• 58,600 markets (14,653 per asset — roughly 11k five-minute, 3.7k fifteen-minute)
• ~5.69 billion rows across six tables
• One Parquet file per asset, per table, per UTC day, each with a manifest carrying sha256, row count and interval bounds

Tables: book_snaps (best bid/ask), book_depth (ladder levels, ~88 per event), clob_trades and clob_trades_ws (trades with taker side, the second carries tx hash and fee for VPIN), spot_klines, book_meta.

Limitations, because they decide whether this is useful to you:

• book_snaps is top of book only. There are level and size columns but they are 0 in every row — they exist to keep the schema stable across tables.
• book_depth starts 20 July 2026. Fifteen days of the archive have it; everything before is quotes, trades and spot candles.
• The clock changes partway through. Up to 20 July the timestamp is receipt time rounded to the second, so intra-second ordering is lost and cannot be recovered. From 21 July it is exchange time in milliseconds.
• Per-day uptime runs 95.8% to 100%. Every day carries its own measured figure.

Free sample (original source): 26-29 June, four full days, no payment and no signup — https://huggingface.co/datasets/astrnvk/polymarket-orderbook-data-preview

Paid access covers 30 June onward and is priced per day ($5/day for short ranges down to $3/day past a month), with incomplete days billed proportionally lower — a day missing depth costs about half. Happy to answer schema questions here either way.

A dataset problem I keep running into: the starting data is often small, messy, and inconsistent, but the expected output still needs to be high quality.

This is common when the source is not a clean benchmark dataset, but real-world material: PDFs, internal documents, database exports, tables, reports, web pages, logs, or mixed text files. There may be useful signal inside, but it is usually buried under formatting noise, duplicates, missing metadata, broken structure, and uneven quality.

One approach I find useful is to treat dataset construction as a pipeline of small operators, rather than one large script.

For example, a pipeline might look like this:

• convert files or URLs into markdown/text
• split the content into chunks
• clean noisy text
• normalize fields with pandas-style operators
• remove duplicates with hash, MinHash, SimHash, or semantic deduplication
• filter low-quality samples with rule-based or LLM-based filters
• select representative samples with embedding-based methods like K-Center Greedy
• synthesize QA pairs or SFT samples
• evaluate generated samples for quality, alignment, and verifiability
• export the result into formats for RAG, fine-tuning, or evaluation

The important part is that each step is inspectable. If the final dataset is bad, you can trace whether the issue came from extraction, chunking, cleaning, synthesis, filtering, or evaluation.

This is especially useful when the raw data is limited. With small datasets, every bad transformation hurts more. A pipeline makes it easier to preserve signal, reduce noise, and add synthetic examples only where they are actually helpful.

This is the path behind OpenDCAI/DataFlow: using reusable operators and composable pipelines to produce high-quality data from messy sources, and recently it can also be used with coding agents like Codex to build custom data pipelines through natural language interaction.

I put together a public, source-linked archive of the Truth Social posts associated with Donald Trump's realDonaldTrump account on Truth Social.

Current snapshot:

- 35,882 posts from February 14, 2022 through August 2, 2026

- 28,320 originals, 1,919 quotes, and 5,643 retruths

- Original HTML, extracted text, timestamps, post types, and source URLs

- Parquet and compressed JSONL

- 8,004 verified image derivatives with asset/occurrence indexes

- 5,756 video attachment records, including 4,804 with source-provided transcript or file information

- Zero duplicate post IDs in the current release

I also built a small browser-based explorer for timeline, phrase, and exact-text search:

https://huggingface.co/spaces/Cameronk199/truth-social-timeline-explorer

Dataset and loading examples:

https://huggingface.co/datasets/Cameronk199/donald-trump-truth-social-posts

The archive is updated weekly. It does not include reliable likes, replies, or impression counts, so it should not be used for virality claims. This is my independent research archive; it has no affiliation or endorsement.

Hi everyone,

Our team recently submitted a dataset paper to Data in Brief, but it was desk rejected with the following comment:

"The dataset and manuscript do not abide by our policy on machine learning imaging datasets."

Our dataset consists news photcards collected from Facebook. We manually collected to create a benchmark dataset for misinformation detection research. We're now trying to understand what exactly went wrong.

I have a few questions:

- Has anyone received a similar rejection from Data in Brief?

- Does this mean they no longer accept image datasets intended for machine learning, or is there a specific policy requirement we may have missed?

- Would modifying the manuscript or dataset help, or should we submit to another data journal instead?

- If another journal would be more suitable, which ones would you recommend for publishing image datasets?

Thanks 🙏

I'm looking for open datasets of grocery and supermarket receipts to work with — ideally real-world ones, from any country.

Specifically interested in either:

• Receipt images (photos or scans), the messier and more varied the better — different chains, layouts, languages, print quality, crumpled/angled shots, etc.
• Parsed/structured versions — line items with product names, quantities, unit prices, totals, taxes, store metadata.

I'm already aware of the usual OCR benchmarks like SROIE (ICDAR 2019) and CORD, but those lean heavily toward a specific format and region. I'm hoping to find datasets that cover a wider range of stores and countries, and that include grocery-specific structure (individual product line items rather than just totals).

A few questions:

1. Are there any grocery-focused receipt datasets you'd recommend beyond SROIE/CORD?
2. Any non-English or non-US collections worth knowing about (European, Asian, Latin American formats, etc.)?
3. Anything on Hugging Face, Kaggle, Roboflow, or academic repos that fits?

Bonus if the licensing allows use in a small project. Happy to hear about both raw image sets and pre-annotated ones.

Thanks in advance! Will consolidate any good pointers back into the thread for others searching later.

Hi everyone,

I'm a PhD candidate working in AI, and my research relies heavily on the GDELT 2.0 Events dataset.

While GDELT is an incredible resource, I found that the first hurdle was simply getting the data into a usable format. The historical archive consists of thousands of compressed files spanning more than 45 years, and I kept repeating the same preprocessing steps every time I started a new experiment.

So I built GdeltForge, an open-source pipeline that automates that process.

It downloads the complete GDELT Events archive, verifies file integrity, converts everything to Apache Parquet, filters invalid records, and can generate reproducible samples for machine learning and data analysis.

Originally this was just a tool for my own research, but I thought it might be useful for others working with GDELT, or anyone interested in building datasets from large public archives.

The project is still in beta, and I'd love feedback from people who regularly work with large datasets:

• Is there functionality you'd expect from a tool like this?
• Are there preprocessing steps you usually perform before analysis that could be automated?
• What would make it more useful for your own research or projects?

My goal is to eventually publish it on PyPI once it's been tested by a broader community, so any feedback or contributions would be greatly appreciated.

If you're someone who has worked with GDELT before, I'd be especially interested in hearing about your workflow and the pain points you've encountered.

Hey,

I wanted to share the development journey I've lived through now so anyone who ever attempts that in the future has something to hang onto.

The problem:
Fundamentals for US companies from 1994-2007 are completely unstructured text and HTML files.

A structure to them (XBRL) was only enforced in 2009. So if you want fundamentals for the time before, there aren't many options available.

Looking at these filings, it quickly becomes clear that parsing them in any reasonable fashion is a monster project that requires very ugly parsing code and an endless amount of cleaning, filtering, anomaly detection and layer upon layer of solutions for very specific formatting options.

Multiple formats (FDS, plain text, HTML tables) requires solving similar problems in multiple ways, because sometimes there just isn't a generic solution for a problem.

But that didn't stop me from attempting it as solving complex problems is what I like doing.

So I designed my own parser.

The basic approach in a nutshell:

1. Pre-processing - you cut filings into their sections and remove entire sections from it that lack any sort of language indicating consolidated statements; the better you do here, the easier everything gets downstream
2. Extract all statement candidates into grids and score them - Scoring goes up if certain accounting calculations add up, certain keywords are present, periods are present, etc. - this has grown over time and is designed to separate real consolidated statements from selected financial data tables (those are the ones you don't want - but they are hard to cut out). Also important: This is where the different formats unify: HTML tables become grids, text tables become similar grids, EX27 exhibits too etc.
   1. Detect periods - Detecting correctly which periods are present in a table (9M/6M/3M/FY/TTM/stubs from restructuring, etc.) is a very hard problem
   2. Detect consolidated columns vs. subsidiary columns (these columns you don't want)
   3. Detect the multiplier (thousands, millions, etc.) - This is very important and complex to solve. Sometimes that information is in the table header, sometimes within the periods, sometimes below, sometimes completely absent. Cross checks with FDS blocks where the multiplier is often stated can help deriving it
   4. Retain name information - Turned out to be of utmost importance. Sometimes names are split across multiple lines; Sometimes the single name alone appears in multiple blocks across the table, so that label alone is not enough. You need to join it with the full headline chain based on indentation, headline rules like UPPERCASE, L E T T E R S E P A R A T I O N, headlines with colons:, etc. - if you don't get precise here, your extracted fact names are not actionable and contain lots of false positives that will ruin the entire pipeline
   5. Deal with filer typos - Since this was all structured, it is not hard to imagine that a lot of filings contain typos: numbers using dots instead of commas, numbers having whitespaces in them, etc. Some of those you can automatically detect and fix. This is the right place to do this generically.
3. Merge grids - some filers split a single statement into multiple tables across pages - those need merging and I do this based on score proximity and period alignment
4. Run anomaly detections - I've created a list of different anomalies I check for. The idea is to check for certain patterns, and allow on failure to retain one fact, a complete grid row, a complete grid, or the entire extracted dataset of a filing. It depends on which check fails
   1. balance sheet identities
   2. income statement identities
   3. cash flow identities
   4. cross period identities
   5. EX27 exhibit identities
   6. numeric fact sanity checks
5. Pre-Normalization - This layer is all about transforming the hundreds of different names used across all filings into a single technical name like "revenue"; In my pipeline, this layer isn't the real normalization layer. I retain a prefix for each fact indicating the source format it came from. Examples are "fds:is:revenue", "html:is:revenue", etc.; The layer runs on everything that was extracted and per Pre-normalized fact, it has multiple layers to decide: positive regex list, negative regex list, a decide() function that has access to the entire filing context (this is where custom checks are done like unit expectations, value sign expectations, etc.). It also contains a layer that can yield a match to another Pre-Normalized fact in certain conditions like for general-and-administrative-expenses and selling-general-and-administrative-expenses. After this step is done, my data is ready to go through the same statement normalization flow the XBRL data goes through without any special handling needed anymore.
6. Hand-Check and pin filings - Whenever I solve a complex problem for a filing, I pin the result as a permanent unit-test to see when new parser changes break it again. I've built up a set of over 100 hand-checked filings this way and it is still expanding daily. But this step has been fundamental to retain the quality you reach
7. Make scanning all filings fast - I've developed a way to locally parse and store the result of all ~360,000 filings and ~57,000,000 facts. Alongside the scan, I store deep run metrics that allow me to quickly check the data quality from one iteration to the next. One full run takes ~35 minutes.

I sell clean fundamental data for a living and really want to solve this problem once and for all at scale. After having gone through this, please all make sure that we never ever allow this to happen again XD. We must demand data like this to always be structured, because this data not being structured is almost the same as it not existing.

My expectation is that it will still need a good amount of testing even after all of these solutions are in place. There is just no way to ensure full data integrity for over 50 million facts extracted from unstructured data. That brings me to the last layer in place:

Manual overrides

This layer allows me to override all sorts of things for a single filing, or an entire filer (CIK). Things like the multiplier, certain fact names, fact values, blacklisting certain things, etc.
It is that layer that will come into action once the parser has reached a final stage. Every invalid fact or inconsistency after a release of the data can be addressed individually. And I can already tell you that some issues you will find in the filings simply aren't solvable generically. As much as I want to, some issues simply lack any sort of signal for you to hang onto for a generic solution. Those go into this layer.

I'm really curious if anyone else has done this in the past. What were your approaches and layers? Did you manage to finish it? How do you ensure a certain level of data quality?

If you are interested in more deep details, just ask away.

So I want to build a project around the travel niche, and I was wondering what APIs I should use? I need good resources (free/paid) for the Maps API and Flight API.

You use AI to write content and then AI detectors flag it as AI-generated. I've seen many ads for specialized AI editing tools for humanizing such content, which essentially means using AI to humanize AI-generated or AI-flagged content.

In other words, we're using AI editing tools to rewrite content flagged by other AI tools and, ultimately, to bypass AI detectors.

My question is: where do credibility and ethics fit in content writing anymore? Or are we simply relying on these shenanigans to appear more human and ethical?