4

I'm working on a project of predicting the bus arrival time for each stops based on the historical data. I checked the database for Boston and New York, but the result is not very perfect.

For Boston, I only found the GTFS schedule data and the real-time data API.

For New York, I did find the historical database. But the trip id in the corresponding data is not in the same format to the trip id from the GTFS schedule data of New York, which makes it hard to connect the historical data to the scheduled data.

Can someone provide some suggestions or some other database which I could use?

-----update

What I plan to use are two types of data: GTFS data and historical data.

The GTFS data is downloaded from here: http://web.mta.info/developers/developer-data-terms.html#data

The historical data is downloaded from here: http://data.mytransit.nyc/bus_time/

The first one is used to represent the scheduled arrival time of buses at different stops and the second one is the record of the location of each bus at different time points at a specific day.

The connection between these two types of data is the trip_id, which represent a specific trip for a bus. Theoretically, they should have the same trip_id, but they don't. In other words, there are many trips in the historical data with the trip_id which I cannot find from the GTFS scheduled data. As a result, I cannot confirm the specific route and the stop list for these trips.

Thus, I wondered whether you could provide some suggestions.

Improve this question
2
  • How many of the ids do match up? Some or just zero? If some do match, then others might not because the other bus routes might have changed, right? How long is the historical time horizon? And who provides the historical data? Is there someone you can contact or other documentation on how trip_ids are constructed?
    – eigenvector
    Feb 7, 2017 at 6:57
  • The historical data is recorded every day. I picked several days' historical data in January and compared it with the scheduled data (GTFS data) for the January as well. After I checked these several days' data and only less than 1/10 of trips had the same id within the schedule data and the historical data. Thus, I believe there must be some changing of the format for the trip_id in the historical data. Unfortunately, there is no explanation of the trip_id in the historical data in the readme file. But you're right, and I should contact the people who are maintaining the historical dataset.
    – JoshuaW1990
    Feb 7, 2017 at 15:24

2 Answers 2

Reset to default
4

I have fully figure out what happened to my problems. For the GTFS data of New York, it can be found here: http://transitfeeds.com/p/mta These data has been divided into several different parts, thus we need to download all of them. For the historical data of New York, it can be found here: http://data.mytransit.nyc/bus_time/

When I compared the full data of the GTFS schedule data and the historical data, every trip_id can be found in both of them. Thus, I think these two parts together should represent the complete data for bus system for data analysis.

Improve this answer
1
  • nicely done! sorry i wasn't much help!
    – albert
    Feb 21, 2017 at 21:38
1

The thing is, the quality of the historical data, if available at all, is going to vary widely from city to city. While it is certainly best practice to contain the trip_id field in all databases, cash-strapped and time-starved municipal governments can't always afford to maintain best practices.

Certainly, in cities with strong Open Data initiatives, such as NYC, Oakland or Chicago, the GTFD feed standardized trip_id field should also be found in the historical data that shows the actual arrival/departure times.

Improve this answer
2
  • Thank you for your advice. After googling, it looks like a lot of cities providing the GTFS scheduling dataset (I found the GTFS data from a lot of cities here: transitfeeds.com). Unfortunately, not many cities provide the historical data. Actually, I only found the historical data(data.mytransit.nyc) from MTA(NYC) and the quality of this dataset is still limited. I guess the reason is that this dataset is maintained personally, not MTA.
    – JoshuaW1990
    Feb 25, 2017 at 2:10
  • In this historical dataset, except for some unreasonable values (which are rare, but still existed), it provides the locations for the all the operated bus around every 1 minute and 30 seconds, but I cannot get the actual arrival/departure time from it. For the other cities, I probably can only get the real-time data from some API interface.
    – JoshuaW1990
    Feb 25, 2017 at 2:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.