6

There are so many libraries, for instance for C# networking or Java encryption.

I am looking for data about the "popularity" of published libraries.

Any measure of "popularity" is fine, for instance it could be defined as:

  • the number of downloads of that library
  • the number of open source projects using that library
  • the number of people talking about it
  • etc

Bonus if the data is easily searchable using a web browser.

2 Answers 2

4

Github is the de facto internet repository for open source code. They've released source code data from packages hosted on this site (press release, blog post with examples).

The data is available as part of Google BigQuery (details). Example queries are here. You can run queries in the browser (requires google account)

This 3TB+ dataset comprises the largest released source of GitHub activity to date. It contains a full snapshot of the content of more than 2.8 million open source GitHub repositories including more than 145 million unique commits, over 2 billion different file paths, and the contents of the latest revision for 163 million files, all of which are searchable with regular expressions.

To compare languages, you can count and group by extension name, for example

  AND sample_path LIKE '%.py' -- for python

enter image description here

Tip:

Don’t analyze the main [bigquery-public-data:github_repos.contents] table — at 1.5 TB, it will instantly consume your monthly free terabyte. Use instead the official [bigquery-public-data:github_repos.sample_contents] extract (~23 GB), or one of the full language tables I left at [fh-bigquery:github_extracts.contents_*].

1
  • Indeed a growing number of libraries are present on Github. Does this data set contain "popularity" information such as number of downloads, number of forks, number of stars? Well, source code activity is often proportional to popularity.
    – Nicolas Raoul
    Commented Nov 4, 2016 at 12:56
1

https://mvnrepository.com lets you know how many times a particular Java library has been downloaded, with a breakdown for each version, see the "Usages" column for Log4j:

enter image description here

... and for its competitor LogBack:

enter image description here

Obviously, you have to divide the number of downloads by the duration the library has been available (date column) and consider that people continue downloading old libraries, especially when they are more stable.

Caveats:

  • Not everyone uses that particular site to download libraries
  • Only open source libraries are on it

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.