Restarting development of the Rust Digger: 20,000 new crates in 8 months

Date: 2024.03.27

I started to write the Rust Digger on Sat Jun 17 11:08:42 2023 +0300, I made the first post about it on 2023.07.22, that was more than 8 months ago. I made some progress with the site after the first report, but in the recent months I have not touched in and I never got to the second post. In that time I worked on a number of other Rust projects and I posted a lot on the Rust Maven site. A month ago I started to record a Rust course in Hebrew and recently even recorded a few videos abour Rust in English: Strings and memory reallocation in Rust and Extreme Clippy for a new crate.

Add Clippy lints

This last video brought me to add a lot more lints to the Rust Digger project and to start to clean up the code. I've recorded a video about that and I'll publish it soon. Follow my YouTube channel to get notified when I publish it.

Number of crates

There are now 140,209 crates almost 20,000 more than the 120,484 crates last July. So 20,000 new crates in 8 months.

Repository

The percentage of crates without repository field slightly went down from 20.06% to 19.36%, but there are now 27,148 crates without the repository field compared to 24,173 8 months ago. That means a lot of the new crates came without the repository entry. Some probably have it in the homepage field. That should be better incorporated in the report, but I think it would be better if these crates had the link to their repository in the repository field.

Real repository?

One huge issue we still don't know if the repository field leads to the repository of the relevant crate. We need to Verify that the referred repository indeed contains the code of the crate.

VCS platforms

75.57% of all the crates use GitHub as their repository, 2.55% use the hosted version of GitLab. There are many organizations and individual people who run their own git hosting service. The most popular ones are GitLab, Forgejo, Gitea, cgit. There are now almost a 100 Version control hosts mapped in the Rust Digger project. Most of them also include the platform they use. We don't yet have statistics showing how popular are these platforms, but that should be a simple matter of adding the numbers together. There are also 858 crates (0.61%) where we did not categorize the repository yet. Down from 1,821 crates due to the longer list of manually mapped hoss.

Continuous Integration

There are 58,175 (41.3%) crates that are on GitHub but have no CI. There are also 1,840 (1.3%) crates that are on GitLab but have no CI.

How can you help?

  • Many of the crates listed as having unrecognized VCS has some incorrect URL as repo. Track down the real repo and send a Pull-Request to fix it or notify the author in some other polite way. We still only show the 100 most recently released crates, but the most likely impact you'll have on crates that were released recently.
  • My personal annoyance is that there are a few crates (306) where the repository link uses http instead of https and there are a few (232) crates that link to www.github.com instead github.com. I know both redirect to the correct place, but if you are looking for an easy PR, fixing these would be that.
  • The biggest issue with the repositories is of course that there are 24,013 crates that have no repository and no homepage
    Finding the repo, adding the repository field will help the connectivity. A much easier thing to do is to visit the 3,292 crates that have homepage but no repository. In many cases it would be enough to copy the content of the homepage field to the repository field.
  • The lack of Continuous Integration for at least 42.6% of the crates show that there is still a lot to do on that front. I bet many of those don't have any tests either. If you have time, check out one of those crates. on GitHub or on GitLab. Add tests if they don't have yet and add CI configuration.
  • Finally, you can help with the development of Rust Digger as well.

Rust Digger Stats from the stats page

name value percentage
total 140,851 100%
Has homepage but no repository 3,292 2.33%
No homepage and no repository 24,013 17.04%
On GitHub but no CI 58,175 41.3%
On GitLab but no CI 1,840 1.3%

VCS hosts from the VCS page

name platform crates percentage
GitHub GitHub 106,415 75.55%
Has no repository Unknown 27,305 19.38%
GitLab GitLab 3,601 2.55%
Other repositories we don't recognize Unknown 864 0.61%
Codeberg Unknown 464 0.32%
srht Unknown 410 0.29%
Repo with http Unknown 306 0.21%
GitHub with www Unknown 233 0.16%
Gitee Unknown 166 0.11%
Free Desktop GitLab 130 0.09%
Bitbucket Unknown 119 0.08%
Gnome GitLab 75 0.05%
Asonix Forgejo 70 0.04%
Tor Project GitLab 62 0.04%
Redox OS GitLab 51 0.03%
pijul.com Unknown 48 0.03%
Qrnch tech Unknown 36 0.02%
sourcehut Unknown 35 0.02%
Debian GitLab 28 0.01%
framagit.org GitLab 23 0.01%
deuxfleurs Forgejo 19 0.01%
Cronce GitLab 19 0.01%
hatter Gitea 15 0.01%
moparisthebest Gitea 13 0%
e3t Unknown 13 0%
Wikimedia GitLab 12 0%
pipapo Gitea 12 0%
University Stuttgart Gitea 11 0%
Open Logistics Foundation GitLab 11 0%
Open CoDE GitLab 11 0%
cscherr Forgejo 11 0%
scd31 GitLab 10 0%
picodata GitLab 10 0%
kitware GitLab 10 0%
bauke Forgejo 10 0%
Philomathic Life Unknown 9 0%
gitgud GitLab 9 0%
mozilla Mercurial 8 0%
LCR Code GitLab 8 0%
fem.mint.lgbt Forgejo 7 0%
computer.surgery GitLab 7 0%
communiquons Gitea 7 0%
chiselapp Fossil 7 0%
alienscience Forgejo 7 0%
4neko Gitea 7 0%
themackabu GitLab 6 0%
Kernel Cgit 6 0%
colean Forgejo 6 0%
batsense.net Forgejo 6 0%
3lp.cx Gitea 6 0%
vdx.hu Forgejo 5 0%
Syndicate-lang Forgejo 5 0%
Open Privacy Gitea 5 0%
notabug Unknown 5 0%
ngni.us Forgejo 5 0%
Software Heritage GitLab 4 0%
mentality Gitea 4 0%
Lightsout Games Gitea 4 0%
jekel.ca Gitea 4 0%
CERN GitLab 4 0%
vwolfe Forgejo 3 0%
skreutz Cgit 3 0%
orbitsolutions Gitea 3 0%
Midefos Gitea 3 0%
in2p3.fr GitLab 3 0%
geomar GitLab 3 0%
flying-kestrel Unknown 3 0%
fizz.buzz Gitea 3 0%
vanten-s.com Forgejo 2 0%
sourceforge Unknown 2 0%
rtng Gitea 2 0%
pijul.org Unknown 2 0%
lemonsh Forgejo 2 0%
Jeffa Gitea 2 0%
envs Gitea 2 0%
duguying Gitea 2 0%
Dubzland GitLab 2 0%
cyplo.dev Forgejo 2 0%
based.computer Forgejo 2 0%
arielaw.ar Forgejo 2 0%
archlinux GitLab 2 0%
Apple Bear Gitea 2 0%
whoisthisjoker Gitea 1 0%
txmn Forgejo 1 0%
tozt Cgit 1 0%
simaek Gitea 1 0%
sergeych.net Gitea 1 0%
reclaimprotocol GitLab 1 0%
radial Forgejo 1 0%
panter GitLab 1 0%
onders.org Forgejo 1 0%
latte Gitea 1 0%
jesterpm Cgit 1 0%
hrzn Forgejo 1 0%
hedwing.dev Gitea 1 0%
git.gay Forgejo 1 0%
flueren.eu Gitea 1 0%
Disroot Forgejo 1 0%
buny.computer Forgejo 1 0%
Bitcoin Ninja Unknown 1 0%
ascee.nl Gitea 1 0%
aikuro.net Gitea 1 0%
meli-email.org Gitea 0 0%
dragonma Cgit 0 0%