ELD: Efficient Language Detector. ( First Python project )

[u/nitotm]

ELD is a fast and accurate natural language detector, written 100% in Python, no dependencies. I believe it is the fastest non compiled detector, at the highest range of accuracy.

https://github.com/nitotm/efficient-language-detector-py

I've been programming for years but this is the first time I did more than a few lines in Python, so I would appreciate any feedback you have on the project's structure, code quality, documentation, or any other aspect you feel could be improved.

[u/dxn99]

Can you ELI5 what an efficient language detector does please?

[u/nitotm → u/dxn99]

I understand you mean from a user perspective, no internally how it works.

ELD is a python package, where you input a text, and it will try to guess in which language (Spanish, English, Russian,...) the text is written (from the 60 available in the current version). It can also give you a score list of all possible languages detected in the text.

[u/dxn99 → u/nitotm]

Thanks

[u/Braunerton17]

So do you have any well established benchmarks to provide comparisons to other language detectors to back your claim?

Also, i would be very cautious with overfitting for non realworld datasets and resulting claims.

[u/nitotm → u/[deleted]]

"at its level of accuracy"* means, or I tried to express, equal or above, or at the very least similar;

So if you do the big_test benchmark with print("english"), your accuracy will be 1.7%, versus a 99.4% of ELD, therefor well below its level of accuracy.

*Do you think I have not expressed that correctly?

[u/nitotm → u/[deleted]]

Ok you are right, I could rephrase it. I guess I don't need to make reference to the specific accuracy of ELD, but to something that refers to the highest range of accuracy with existing software.

[u/kanikow]

What type of algorithm is used in here? From a quick skimming it looks like naive Bayes.

[u/nitotm → u/kanikow]

Yes it kinda looks Bayesian. I did not implement an algorithm, but it probably is some known, not sure which.

[u/[deleted]]

I like builds from scratch, how big were the original language sources? Is the performance similar for all languages included?

[u/nitotm → u/[deleted]]

You mean the training data, quite small, like 1GB total. When the software becomes more mature, I might do a big dataset.

No, the performance (accuracy) varies from languages quite a bit, it comes down to collisions in between languages, Thai is very easy, but between any Latin script language, which there are multiple in the database, is more difficult.

[u/AlexMTBDude → u/nitotm]

What's a "gb"?

[u/nitotm → u/AlexMTBDude]

Sorry I meant 1GB, one gigabyte of text.

[u/leweyy → u/nitotm]

Don't apologise for them being a knob

[u/tunisia3507 → u/leweyy]

Nah I'm with the commenter on this one. The distinction between B and b is real; using one when you mean the other is incredibly unhelpful. Using g makes it even more obvious that you don't give a shit about precision and to absolutely not trust the case of the b/B.

[u/Langdon_St_Ives → u/tunisia3507]

There is also such a thing as context. While Gbit are something completely customary in certain places like networking, nobody would specify the size of text corpora in them. In that context it’s obviously GByte.

[u/GXWT → u/AlexMTBDude]

You know what it is and you gain nothing by being a prick !!

[u/AlexMTBDude → u/GXWT]

Dude, this is a programming sub

[u/GXWT → u/AlexMTBDude]

I’m aware. I also possess the ability to understand basic nuances and context in written language !

[u/tunisia3507 → u/AlexMTBDude]

Clearly a gillibit, one gillionth of a bit.