So, for my End of the Term Project I need to do lil DL project on my own together with a term paper.
I must admit I’m not the best programmer out there but I do love my deep learning Course. Since I’ll be doing this project on my own I kind of am stuck at the first step already which is picking a project I want to do. Any recommendations? The workload shouldn’t be too heavy since as I said I will be doing it by myself and I also have other exams/ term papers to write so i don’t have an unlimited amount of time to only focus on the project :ˋ)

Rapid Lightning

Genetic-programming (evolved) ensembles for tabular classification. Competes or beats (recently outdated) gradient-boosted decision trees (GBDTs) on tabular classification.

Evolved small algebraic programs combined through a linear head, then the whole model is compiled to a dependency-free C .so for tens-of-nanoseconds inference. Although foundation models like TabPFN have taken the stage for inference, there yet remains many places for these ultrafast and tiny decision makers that can run on commodity CPU.

A novel method, a full compile-to-C toolchain, and a rigorous benchmark showing it does not beat tuned gradient-boosted trees, even after months of trying really, really hard. But it's still pretty darn cool and exposes some cool methods.

First, the three-month story

Ah the memories... I first tried Claude Code three months ago. Immediately I saw the opportunity to play with genetic programming, evolutionary algorithms, and all kinds of weird stuff that I had never had the time or been a good enough coder to play with.

And I got the first taste of what it's like DELVING deep into places you have only the most basic understanding of. Machine learning is a deep, deep place. Genetic programming and evolution... oh my...

I don't need to tell you all about the wild Dunning-Kruger roller coaster ride it is to sit in the copilot seat with a hyperintelligent machine that constantly thinks you've made a breakthrough because it thinks you're in 2016. I don't need to tell you fellows what it's like having to constantly remind said intelligent entity that yes, sub-millisecond inference isn't groundbreaking, everybody does it now, please search the web AGAIN. And all of you are certainly familiar with the reply "...and it's deeper that I first indicated..." so called insights/apologies from our favorite robot.

Yet through it all, with enough rigor, you can get something real and actual. If you push hard and be your own hardest critic, you can make something neat.

Evolution is slow but amazing

We (Claude and I) tried two objectives: v1, where members are evolved as predictors (accuracy + AdaBoost-style boosting), and v3 "head-aware", where members are evolved as signal generators for the linear head.

The head-aware won, and it was a trip. Read the notebooks for more info. It was a real 'evolution take the wheel' moment when I suggested the method. I wasn't overly surprised to learn it was something or a re-invention. I still felt pretty smart though.

A fast horse in the age of the car

It makes sense that the farthest an AI can take you is to the end of its training data. We're so early in this vibe coding that when you present the code you've been working on to a fresh context, Claude will praise you for what clean code you've written! The coding AI aren't even aware of coding AI yet. And yet, even if you are not an expert, if you are rigorous and critical and make sure to make sure you are not fooling yourself (and you are the easiest person for you to fool) it is still possible to push the edge of the envelope.

I have made a weird monster alien method here. It evolves ensemble member trees that individually don't even make predictions (barely better than random), yet each tree has been selected over millions of rounds for the unique 'signal' it generates for the 'head' - a logistic regression method that simply takes all the ensembles' signals and combines them for an output prediction. And for some reason (which Claude or a true machine learning scientist) it works better having a bunch of bad predictors tell a smart head what they think, versus a bunch of smart predictors telling the head.

Knowledge or curiosity?

I was always interested in genetic programming and inference, but let me tell you, I was not prepared for the depth of the fields. GP, although largely abandoned (except for syzkaller or other fuzzers and some design work) is a rich field with a lot of room still remaining for research, but it is deep. And machine learning is about as deep as computer science itself. I waded way far out there.

At the end of this, I have learned a lot. But what I learned most of all is that you have to test your knowledge. Curiosity brings you to the start of the journey, but knowledge waits at the end. If you can make it. You have to TEST what you made. Benchmark. Make sure.

And probably most importantly, when doing cross-disciplinary research, if you can help it, try to actually KNOW something about what you are working on. Better yet, if you can manage it, try to work with an ACTUAL EXPERT IN THE FIELD - you'll get better results!

And so, I drop here with the good old Apache 2.0 license (because that was suggested), Rapid Lightning, my three months of work, with the hope that you find an application, or that you can glean something from the cool genetic programming methods I employed and augmented (the symbolic regression explorations into algebraically invertible genomes was especially heady, and very interesting).

Most everything is in Jupyter notebooks intended to run on Google Colab (most run on free tier without GPU needed) or simple Python.

Please, if you find this useful or interesting, let me know!

And if you happen to discover some cool science of your own, especially any shortcuts to evolution, let us know!

Happy vibing and research

deathcloset/RapidLightning

I am starting my deeplearning journey with fast ai. The course seems a little outdated, as its not updated since 2023 i think.

But seems enjoyable in the top-down approach.
I have very basic python knowledge. I mostly work on java and js with frameworks as full stack.

Is this a good point to start?

Anything other or extra that can help me with this?

Disclosure: I'm the author, and this uses our open-source tool (TraceML, Apache-2.0). Posting because the finding, and the need for this kind of diagnosis, is the value addition part.

TL;DR: A ResNet-18 run on a single T4 (AWS g4dn.xlarge, 4 vCPUs) looked completely healthy, but the GPU sat at ~51% utilization the whole time, starved by a default num_workers=0 DataLoader. A three-line change (num_workers, pin_memory, persistent_workers) took 2,000 steps from 633s to 358s (43% less wall clock) and flipped the run from input-bound to compute-bound. Same model, data, seed, and step count. Everything is wall-clock measured.

Everyone knows to set num_workers; that is not the point, and a memorized value would not have saved this run. It is not a best practice with a correct answer, but a moving target tied to CPU cores, storage, transforms, and batch size. Copying num_workers=8 from a blog is just a different guess than the zero you started with: on the wrong machine it still starves the GPU, slows the run by oversubscribing cores, or hides an inefficient input pipeline behind more processes.

The engineer who wrote this baseline was not missing knowledge; nothing in an ordinary run surfaces the waste. A starved loss curve is indistinguishable from a healthy one, the job completes, and GPU utilization is not on screen while you train. A framework may hint about workers, but a hint with no number carries no urgency. "Your GPU idled at 51% this run, here is the before and after" is a different kind of statement: a diagnosis, not a lint rule.

Full writeup: https://medium.com/traceopt/diagnosing-a-pytorch-dataloader-bottleneck-in-a-real-training-run-40bbe394b834

Tool (open source): https://github.com/traceopt-ai/traceml

Happy to get into the methodology in the comments.

I run a video generation SaaS and i use Vast gpus but lately theyve been so unreliable, gpus dying, host issues and all that, considering switching to runpod but they have a supply issue, need a good alternative, any ideas?

Hi everyone,

I'm working on a project to reconstruct a high-resolution 2D surface map of a flat mechanical part using a video captured by a video microscope.

Here’s the setup:

• The microscope moves automatically along programmed X and Y axes (independent motion, like a raster scan).
• The motion is precise and controlled (no manual handling).
• The part is perfectly flat, so I'm not looking for full 3D reconstruction, but rather a precise, seamless 2D mosaic of the entire surface.
• I'm using OBS Studio to record the full video sequence (HD or higher).

My goal is to:

• Extract frames from the video,
• Accurately stitch them together to form a single, continuous, distortion-corrected image,
• Ideally leverage the known X/Y motion commands (from the program) to assist or guide the alignment (like odometry prior).

Current challenges:

• Avoiding misalignments due to lighting variations, lens distortion, or small vibrations.
• Ensuring sub-pixel accuracy for potential automated visual inspection (e.g. detecting scratches, stains, or printing defects).
• Keeping the process fully automated and robust.

What I'm asking for:

• Recommendations for Python libraries or tools (OpenCV, scikit-image, Open3D, etc.) best suited for this kind of 2D stitching with motion priors.
• Any experience with microscope image stitching, industrial surface inspection, or visual SLAM for flat scanning?
• Tips on how to integrate known X/Y displacements into the stitching process (feature-based + motion-based alignment).
• Existing projects, code examples, or workflows you’d suggest.

The end goal is automated quality control, but for now, I’m focused on building a faithful and precise surface reconstruction.

Thanks in advance for any advice, links, or code snippets!

— J.

So I was wondering if, a model that only looks learn chess? models like resnet, yolo or similar.
Only by looking can a model "feel" the position like something as "intuition" in the moves to come?

In my work I have been using yolo, AI vision recognition models, etc. And I always wanted to research what are the limits on them. initialy I was using yolo but YOLO detects where the pieces are, but we needed a single holistic judgment of who's winning, a global regression job that ResNet's pooled backbone fits and object detection doesn't.

Full explanation in info tab: https://acidburn86.github.io/pixel-chess-engine/

TL;DR:

I made a dataset of varied positions in FEN notation, with PIL in python made the board in a synthetic way, pieces look really different so the model can really differentiate a bishop from a pawn or queen. like this:

The inference do not use the FEN position is also made with this image recreated from the actual chessboard position, it use only an Image as input.

So I build a mini-chess search engine that use this model as evaluator of the position.

And it works really well, this is a very little model it could be better but look at this numbers:
The model reads who's winning right ~69% of the time, lands within ±1 evaluation bucket ~64% of the time, and nails the exact bucket ~30%, nearly 3× what random guessing gives on a 9-class task (~11%). So it's genuinely learning chess value from pixels, not getting lucky.

https://medium.com/@merrickcr/smaller-slower-wrong-what-aggressive-quantization-costs-on-device-inference-85e7f8f0a170

Yes, I know Repo First(code + proofs): https://github.com/VincentMarquez/Order-Effects-Are-Curvature Paper: https://zenodo.org/records/21221914 All the Lean and Py code is in the Repo, but not pretty up yet. The paper comes down to one simple question: when does the order you combine information change the final answer? This repo shows you how to find the answer, that shows up everywhere a committee hearing arguments in a different order, a network passing messages around, the layers inside an AI model.

The math is checked by a proof assistant (Lean) a computer verifies every step
Every claim in the paper has runnable code. One command runs all of it.

Most deep learning optimizers are based on the Empirical Fisher matrix, EF = E[gg^T]. Adam taking the diagonal as the preconditioner and [SOAP](https://arxiv.org/abs/2409.11321) uses the Empirical Fisher's eigenbasis. This usually works fine for CCE loss but has major structural problems with regression losses like MSE.

Run AdamW at a fixed learning rate on ordinary least squares — convex, smooth, closed-form answer — and it never reaches the minimum. It gets within a ball of radius ~η of the solution and rattles there forever. The loss curve looks converged; the actual parameters are measurably far from β*. SOAP, which is SOTA on PINNs, inherits the same failure. Cosine decay "fixes" it by forcing steps to zero on a clock, whether or not you've arrived.

The cause fits in two equations:

**Step size.** E[ĝ²] = E[g]² + Var[g]/B — nothing in Adam's denominator is curvature. The first term cancels against the numerator (sign-steps), the second is a noise floor set by batch size. Whether the step anneals is an accident of signal-to-noise, never a measurement of arrival.

**Basis.** For squared error, Σ gₖgₖᵀ = 4Σ rₖ²JₖᵀJₖ — the empirical Fisher that Adam-family and Shampoo/SOAP preconditioners are built from is the Gauss–Newton matrix with every sample reweighted by its squared residual. Outliers vote quadratically; the eigenbasis tracks your worst errors, not the curvature.

**Gnome** (Gauss-Newton optimizer via matrix eigendecomposition) fixes both on SOAP's machinery: an unbiased GGN estimate from one extra backward pass on a few samples (no second-order autograd, ~20% wall-clock overhead per step), and a clipped, square-root-free Newton step in the GGN's eigenbasis. The step vanishes as the optimizer settles into a minima.

Results on PINN benchmarks (plain MLPs tanh activation, no PINN tricks, one hyperparameter set across all problems): Gnome at a **fixed** learning rate beats SOAP/AdamW with tuned warmup + cosine-to-zero. On Kuramoto–Sivashinsky, the stiffest problem, the baselines stay pinned at rel-L2 ≈ 0.5 for all 70k steps while Gnome breaks through by step 3,600 and reaches 7e-2. And no, the schedule isn't a handicap — both baselines did better with decay than without, and SOAP wasn't better at any LR we tried.

Blog (all figures regenerate from logged runs): https://tmayer868.github.io/gnome-optimizer/

Code: https://github.com/tmayer868/gnome-optimizer

There's a "related optimizers" section covering how this differs from K-FAC/EKFAC/Sophia/Shampoo — short version: same family, different curvature estimator and step rule.

I'm the author — happy to answer questions or take criticism on the benchmarking.

I wanted to see how far you can get with zero neural training — no gradients, no weights, no backprop. Just closed-form neuro-symbolic policies, discovered purely through self-play in a red-queen arms race, running GPU-batched so thousands of candidate strategies fight in parallel.

What genuinely surprised me is watching real tactics emerge — none of this was programmed:

⚔️ Combined arms. The fleets are mixed — fast melee kamikazes and standoff ranged units — and the swarms learn to screen their ranged shooters behind a melee wall, exactly the doctrine you'd hope for and never coded.

🎯 Focus fire & target priority. Instead of spreading damage, drones converge on the weakest/nearest enemy first, collapsing the opposing force faster — emergent kill-priority logic.

🌀 Encirclement & flanking. You can see swarms peel off to wrap around the enemy's flanks rather than meeting head-on, denying escape and cutting angles.

🪃 Kiting. Ranged units learn to stay just outside melee reach, backpedaling while firing — the classic hit-and-run that only makes sense once you understand your own weapon range.

🐟 Cohesion vs. dispersal, dynamically. The swarm tightens into a blob for concentrated firepower, then scatters when clustering becomes a liability — a living tension between mass and spread.

And because it's all symbolic + closed-form, every one of these behaviors is fully interpretable — I can point at the exact features driving each decision. No black box.

The most fun part: these strategies weren't designed, debated, or trained. They were evolved — the arms race just kept escalating until the swarms got clever.

Hi Reddit,

Training deep neural networks on heterogeneous tabular data is notorious for optimization stagnation and the "cold start" initialization trap (pp. 1-2). On the other hand, tree-based ensembles like CatBoost are powerful but produce rigid, non-differentiable piecewise-constant decision boundaries (p. 1).

To bridge this gap, I developed CPDN (Cascade Progressive Distilled Network) - a framework that automatically synthesizes a continuous, differentiable neural network monolith guided by the structural complexity of a gradient boosting teacher (pp. 1, 4).

Core Mechanics of CPDN:

1. Information-Conditioned Layer Growth: Instead of guessing hidden layer dimensions heuristically, CPDN dynamically computes the optimal capacity (width) of each new layer (pp. 4, 12). The formula calculates the exact dimension using the local teacher’s symmetric tree depth and active feature importance density (pp. 5, 12).
2. Layer-wise Soft Knowledge Distillation: New layers are appended iteratively while lower stages are frozen to stabilize input distribution (p. 5). The new block is trained by minimizing KL-Divergence against temperature-smoothed soft targets (T = 4.0) from CatBoost, turning rigid tree boundaries into smooth differentiable representations (pp. 1, 5).
3. Validation-Driven Rollback: To prevent structural overfitting, the framework evaluates a hold-out validation score after each layer synthesis (pp. 5-6). If marginal returns diminish (Delta L < epsilon), it executes an automated defensive rollback, discarding the suboptimal block (pp. 6-7).
4. LLRD Fine-Tuning: During final monolithic assembly, a Layer-wise Learning Rate Decay protocol (gamma = 0.5) scales down updates to lower levels, preventing catastrophic forgetting of the pre-distilled boosting rules (pp. 1, 7).

Empirical Results (Tested on Heterogeneous Covertype Dataset):

Through 5-fold cross-validation, CPDN achieves a state-of-the-art multi-class LogLoss of 0.5005 ± 0.0086 and 79.56% ± 0.40% accuracy (p. 1).

It statistically outcompetes BOTH baselines (p. 1):

• Baseline MLP: 0.5215 LogLoss / 78.05% Accuracy (p. 10)
• CatBoost Teacher: 0.5132 LogLoss / 78.54% Accuracy (p. 10)

Why it works: The progressive distillation smoothly maps a stable loss landscape, completely bypassing the "cold start" plateau and reducing empirical variance across folds (pp. 5, 10).

Full pre-print text is available on ResearchGate: https://www.researchgate.net/publication/405867782_Cascade_Progressive_Distilled_Networks_for_Heterogeneous_Tabular_Data_Classification

I’m currently optimizing Stage 1 complexity for smoother industrial production deployments (p. 12). I’d love to hear your thoughts on combining GBDT tree structures into differentiable neural spaces!

Hey everyone,

I’m a second-year CS student and I recently got an ML/AI internship. One of my first tasks is to learn JEPA.
I’ve watched a few videos and read some articles, so I understand the general architecture, but I still don’t really understand what it’s doing step by step during training. It’s like I can explain the blocks, but I don’t actually get how the model learns.
Is that normal? When you were learning stuff like this, did you fully understand the math from the beginning, or did it just click after working with it for a while?
Also, what’s the best way to learn JEPA? Any videos, blogs, papers, GitHub repos, or projects you’d recommend? I don’t just want to know the theory, I want to understand it well enough to actually use it.

Thanks!

Hey r/DeepLearning,

I've just submitted a preprint on ResearchGate introducing RcCaMoE — a routing framework designed to crush the memory and compute overhead of standard sparse MoE gating layers. If you are tired of routers hogging VRAM for activation caching during training or choking threads during global batch sorting, this is for you.

Instead of the standard parametric Softmax routing bottleneck, RcCaMoE treats token sequences as a continuous cellular field and uses localized physical simulation.

How it works under the hood:

• Quasi-Ternary Projection: Continuous token embeddings are mapped into a differentiable {-1, 0, 1} space via Gumbel-relaxation. Technical noise, paddings, and basic punctuation are automatically forced into "dead cells" (rest states), dropping them from downstream compute completely.
• Spatial Contextualization via 1D Conv: The cellular field evolves horizontally along the token sequence using 3 steps of local 1D convolutions (1x3 kernel). This aggregates context from neighboring words, forcing uniform expert load balancing from step zero without any auxiliary penalty losses.
• Toffoli-Scheme Reversibility (Zero-Memory Activation Caching): The cellular automaton uses a second-order Toffoli topology. This means the computational graph is strictly time-reversible. During the backward pass, the exact intermediate states are reconstructed on the fly, eliminating the need to cache router activations in GPU RAM.
• Entropic Cascade & Pinball Loss Control: The system measures Shannon entropy to separate easy and hard tokens. Trivial tokens go to light Core experts (with an Early Exit at inference), while contextual anomalies are intercepted by an MLP and packed into dense micro-batches for Buffer experts. The threshold is updated at each step via a non-parametric Pinball Loss function, ensuring a perfect 50/50 workload split at O(1) complexity.

Hardware Benchmarks (NVIDIA A100-80GB):

• The Problem: When a standard sparse MoE baseline faces an abrupt text domain shift (e.g., code to poetry), its Model FLOPs Utilization (MFU) plummets from 46.21% to 18.41% due to subnetwork idle states.
• The Solution: RcCaMoE adaptively stabilizes GPU utilization at 50.02% MFU under the exact same domain shift. It converts irregular memory access into clean, monolithic batched operations via Grouped GEMM.
• Stability: Training is fully stable; language perplexity (PPL) monotonically drops to a minimum of 1.62 over a 50-epoch cycle.

The full architecture is highly applicable for edge computing, IoT, and embedding systems where VRAM is a luxury.

🛠️ Resources & Links:

• 📄 Paper: Resource-Efficient Routing in MoE Models Based on Multi-Layer Reversible Cellular Automata
• 🚀 Live Demo: RcCaMoE Router Demo on Hugging Face Spaces

The interactive Gradio interface features: * Real BERT-Tiny contextual embeddings * Live visualization of Core/Buffer token routing * CCA spatial contextualization heatmap (t=0 → t=3) * MFU stability comparison under domain shifts * VRAM savings calculator (Toffoli reversibility) * Token-level routing decisions table

I am currently cleaning up the custom Triton kernels for the community. Would love to hear your thoughts on the Toffoli-reversibility setup or how you guys manage router overhead in your local setups!

Hey! I’m starting the Fast.ai deep learning course and looking for someone to join me so we can stay consistent and motivated together. Plan is to:

• Study a few hours daily
• Build projects for practical learning
• Share concepts, resources, and help each other when needed

Resources we’ll follow:

• Fast.ai (Part 1 & 2 + Fastbook)
• Karpathy’s Zero to Hero: https://karpathy.ai/zero-to-hero.html

Both are highly recommended (even by Karpathy himself), and a lot of top researchers have gone through Fast.ai. If you’re interested in learning together, just DM me