For a very long time, I've been hunting for sources of data to process in my database. I've found that everything is either behind a pay wall and wants me to pay (lots of) money, which I don't have, or it's "allegedly" available but so well hidden behind an eternal maze of weird links and cryptic webpages (government websites) that it might as well not exist.
Only in an extremely small number of cases have I been able to actually find any kind of non-garbage data set for me to automate downloading and updating my database with, which can actually be used for anything.
I've found websites listing "free APIs". These are always complete trash. I've gone through them all and they are all "novelty" nonsense such as "random joke" or something useless like that. The kind of data I want is meaningful statistics and information which can actually be used for something.
Besides, "APIs" are fundamentally privacy-disrespecting even if free and even if they have some sort of useful data. I don't want to announce to the world or some external party what piece of information I want in real time; I want to have the whole dataset locally and make "secret" (local) queries on that, without anyone knowing what data I'm using/interested in.
Is there a real website which actually has collected real data sets of non-garbage data? I don't even mean that they need to have "processed" it in any way; just that they link to the sources where I can manually figure out how to grab and process it.
I'm not mentioning any specific kind of data here because my data needs are very general. In fact, I want to base whatever service I'm going to build on whatever data is actually available, as I can't just decide to do thing X, and then try to find data for it, since I can never find any quality/updated data for whatever really interests me. So instead, I hope to become inspired to build some sort of service by learning that some kind of data is actually available for me to process and present in some useful manner for people.