The U.S. gov creates and makes available a lot of open data on various topics, but the one that I am interested in is new legislation.

Let's say that there is an open data source for new laws (which I believe to be the case). I'm interested in creating a database for "scholarly" (legal journal) articles on these new laws. The government would (presumably) have apis, or other protocols for gathering, indexing, collating, etc. on these new laws. Would it be possible to license, or otherwise obtain, these protocols from the government to create a "parallel" private sector data set that would articulate or interface with the government data set instead of "recreating the wheel?"

Put another way, how would the government create these open data sets, and are the creation tools available by license or other otherwise from the government? I am interested in "back end," data collection procedures. I think I know people that can handle the front-end, presentation issues.

  • 1
    define protocols; like 'ftp', 'https', 'mailto', 'javascript', and 'dat'? also, if you can provide more details/specifics this would be much easier to answer. most of what you are asking about exists, has been done, is being done, is slated to be done; nothing is across the board in federal gov, each implementation has its own characteristics. to that end, almost all of gov data is open, and almost all of their tools are open licensed/open source. going to have to be much more specific. also, come to legislative data and transparency con next month cha.house.gov/LDTC17 – albert May 22 '17 at 17:54
  • @albert: I edited the question incorporating some of your suggestions, and am trying to reformulate it. My latest iteration is something like, "if the government data is open data, when If at all) is the code behind the data gathering open source?" – Tom Au May 22 '17 at 18:56

It depends

It depends on which particular agency/department. It depends on which dataset(s). It depends on how they go about collecting/curating/presenting/utilizing said datasets (FOSS vs. proprietary), etc. It depends on how many corporate entities are already at play in that particular space; how much of the space have they already captured, how does the space feel about said capture, etc.

The federal government has not mandated FOSS, nor has it defined FOSS and its relationship to open data, so there is no clear path* for government worker bees to follow. Last year the Obama administration laid out a path for Open Source to start being created by government, but that is about as mandated as it gets. Note the clear absence of a path to start adopting open source. You can see this in the following examples:

HUD offers many of their datasets in SAS and another proprietary format that I can't recall off the top of my head. They give you two options, but you can't do anything with either, if you don't have a license for the software they are distributing it with.

It seems like almost every single federal GIS portal has gone the way of ESRI/ArcGIS, and while most offer downloads of open source data formats, some rely entirely on ESRI. You either have to use ESRI's tools, or find a way around them (like pyesridump).

A lot of the older satellite imagery data NASA, NOAA, USDA, and USGS have are JPG2000/MrSID; the options for getting them converted into an open format are extremely limited, especially if your on one of the operating systems that they do not support (mostly OS X).

Speaking of operating systems, some data may also come with open source software, but that software sometimes is only built for one OS. USGS and NASA have done a fairly good job creating FOSS tools, but most are not cross-platform, so they still lock some potential users out.

Many corporate entities offer free-ish (freeware, basic account, etc.) software/technology that looks great to those not familiar. Here the lack of a clear definition hurts/holds us back; those not in the know, hear "free" and think they are taking the right path only to find out (typically) to late into the process that free != the entire product/service, etc. **Socrata entices groups because their data platform is free to adopt by government institutions. But the free runs out once you hit a certain number of datasets. It's the same with Google Maps: free to use until you hit a certain usage rate. Combine Google Maps with Google Fusion Tables and see how confusing that is for governments publishing on said platforms. They are both free-ish, but you can't just download data from Fusion Tables, and you can't use Maps free all the time.
I am going to keep picking on Google: Google Earth was made free relatively recently, so it seems ideal to publish GIS data in KML format if you were not in the know. Said ideal decision became less than ideal when Google Earth announced their latest version, which only works in Google Chrome. Google Chrome is a proprietary browser that you may not be allowed to install or simply may not want to install. Lastly, the below quote from Google Earth's Wikipedia entry sums up just how FOSS Google Earth really is (it is not):

"Every image created from Google Earth using satellite data provided by Google Earth is a copyrighted map. Any derivative from Google Earth is made from copyrighted data which, under United States Copyright Law, may not be used except under the licenses Google provides. Google allows non-commercial personal use of the images (e.g. on a personal website or blog) as long as copyrights and attributions are preserved. By contrast, images created with NASA's globe software World Wind use The Blue Marble, Landsat or USGS layer, each of which is a terrain layer in the public domain. Works created by an agency of the United States government are public domain at the moment of creation. This means that those images can be freely modified, redistributed and used for commercial purposes."

Even the most widely used/consumed processes, technology, and formats in U.S. federal gov (state and local too) that have longevity are not clear, nor precise when wading through this arena. (Almost) Every single document of (almost) every single gov site has some disclaimer at the bottom of it explaining you need software to open up PDFs, and points to Adobe's Reader site. What is wrong with this picture?
a) PDF is not an open format, even though Adobe claims and markets it as one. There are certain aspects of it Adobe specifically omitted from the open license to continue their mastery in this space. One particularly horrible aspect: you can't get accessible PDF software without paying for it. That's right, its an "open" format, but functionality required for some people to even be able to use it at all, costs money.
b) the corporate links these government sites point to almost all fall under the free-ish examples I've already touched on. Sure, you can open and view a PDF in Adobe's free reader, but you can't do anything else without a license. Microsoft's free products follow the same theme as well. They let you taste it, but if you want to eat, you're going to have to pay up.
c) Why do all of the links point to Adobe? or for office documents, why do they all point to Microsoft? There are open source/free software options that could be linked to, as well as open source/free software formats they could adopt, yet the only spaces I've seen this in are on extremely niche USGS/NASA sites, where the entire team has adopted FOSS/OS. The lack of options even being presented to users is mind-boggling to me. It is a mix of confusion, denial, ignorance, atrophy, and going-with-flow/don't-rock-the-boat that serves no one save the government employees and Adobe/MS/Corporate entity.

Lastly, and this muddies the waters even more, some departments/agencies/personnel have their own definitions of FOSS/open data. USDA APHIS runs a database to which they don't provide table headers for. I requested the definitions because it is impossible to understand the table/underlying data without them. USDA APHIS response to me was that they are proprietary. And that about sums about the wild west show that is U.S. federal government open data/FOSS.

The FCC had done a great job initially adopting and promoting open source, however the only link in the content of this page 404s. Following it up in the wayback machine, there are two entries a year and a half apart 2015-03 and 2016-08 that redirect to a dfferent URL. None of this means that the FCC doesn't support FOSS, or that they've stopped, but its extremely hard/confusing to follow along. I'm assuming its the same for the premise for asking this question in the first place. While FCC does go above and beyond in adopting/supporting/promoting open source here in federal us dotgov terms, did they really? I don't think so, these MS-Office tools for tracking tie you to software, and possibly an OS (at least at one time they did): Microsoft Access-based Tracking Tool (.exe), Microsoft Excel-based Tracking Tool (.exe).

NOAA's ESI (Environmental Sensitivity Index) maps and tools mostly all fall into this categorization too: you can view the data in ESRI ArcGIS maps online, but you can't do anything with the tools built for them without accessing ESRI's ArcGIS ecosystem, which is not open source.

I can provide a plethora of other resources for state/local that are doing the same things with different software/data.

And all of this doesn't even touch on what/how the rest of the facets of openness (Open Access, Open Science, Open Education, OER, etc.) are playing out currently in this arena.

If you have a particular dataset/department/agency in mind, it never hurts to engage them and feel them out. In many cases it feels like they want to hear from you/will work with you. Many simply are not aware of the entire ecosystem and/or have yet to see anyone they know on the federal level implementing something like it or similar.

In summation, there is no clear answer and every single question on this topic in this space has a unique answer.

*18f and USDS have blazed paths to open source; NASA, USGS, and NOAA have done a lot as well, just not as a whole agency. To be fair, the latter three have substantially bigger variants to deal with: number of employees, existing infrastructure/architecture not built as FOSS, niche/specific details unique to particular datasets, data collections, etc.

•• Aside from Google and all of its products, I'm not singling out any characters or actors on purpose. These are just the default examples I go to when discussing this. There is a place for corporate/for-profit/enterprise in government. That said, everyone has to be a good faith actor in this field, or it all falls apart.
Google (Alphabet) has become entirely too dominant, across too many (crucial) spheres of the web, while simultaneously ditching it's famous motto "Don't be Evil". Not all Google/Googlers are bad, and they are still doing a lot of great work on the web, but their actions have clearly shifted from their roots. An open web needs healthy, competitive, decentralized ecosystems. Google has done extremely little recently in regards to that, and in some instances, worked against it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.