Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [web-crawling]

Web crawling is done typically for building and maintaining a Web Index. The crawled data is compared to previous versions of that dataset for changes. Web scraping is similar to web crawling, except that the site being crawled is scraped by the bot (crawler).

2
votes
0answers
163 views

Facebook Mypersonality dataset

I want to have access to a database for Facebook Photos dictionary that contains 17,2mn records (photo_id,owner_id, created_time). At this time, the owner of the dataset has decided to stop sharing ...
0
votes
0answers
39 views

sharing data generated with copyrighted news

This question is related to How can I Share my Data Sets Without Worrying About Copyright Issues?, but asks for the limits of what can still be done. I have a news dataset (scraped from web) that I'd ...
0
votes
0answers
28 views

How to download data set from stock exchage site?

I want to download different type of data set from Dhaka Stock Exchange. For example I download this dataset that is data of DSE market summary from November 2015 to current. But, I have no idea how ...
1
vote
1answer
30 views

The list of restricted second-level domains such as .co.uk and .ac.jp

Some country-code top-level domains (ccTLD) contain general-purpose second-level domains, like .co and .ac, so the end user of the domain is identified with the third-level domain, such as oxford.ac....
0
votes
1answer
81 views

Extracting data in tabular form from camelcamelcamel

Can anyone give me pointers on how I can extract price and sales rank data from camelcamelcamel.com? They display only a chart showing the evolution of prices/sales rank over time, but I'm trying to ...
0
votes
0answers
25 views

Are there any automated techniques one can use to gather data online for a dataset?

I am a developer myself and would like to use latest technologies to build some open data datasets. I would like to know if you are aware of any techniques or algorithm one can use to automate gather ...
3
votes
2answers
105 views

Places to get large (volume wise) datasets reasonably well formatted and free to share?

I've been struggling with this problem for almost 2 hours now :( I need some datasets that are somewhere around 50 - 5000 GB large when uncompressed to showcase and test various data storage and ...
3
votes
1answer
34 views

What tool can I use to manage a researcher network bibliography?

I am making a website for a researcher network using Jekyll and hosting it on github pages. Each researcher (250+ people) should have his own document with an up to date bibliography. I could ...
0
votes
2answers
189 views

How do football API's get their data?

I am wondering how sport API's get their data. Do they use web scraping or web crawling? If they do: is it legal to create my web scraper to gather data? I checked various sites and they all lead ...
16
votes
1answer
341 views

The “right to mine” and scraping the web in EU

Sorry for the somewhat general question, to which the answers necessary depend on the country of operation and necessarily are also subject to become obsolete at some point. But I think it to be quite ...
1
vote
0answers
69 views

Scraping product image from eCommerce websites [closed]

What are the legal clauses associated with scraping images of products displayed on eCommerce websites (such as Home Depot, Walmart, Amazon, eBay etc)? The intention is to obtain meta-data from the ...
5
votes
2answers
685 views

how to search archive.org for PDF files on a captured website between some date range

i am trying to search health insurance companies' website content on http://archive.org/ during the period of 2005-2008 for legal documents related to pre-existing medical conditions. most of these ...
3
votes
0answers
81 views

What API can I use to retrieve HealthCare Data?

I am looking for an API which provides the healthcare data of a patient. What I am trying to do is: User logs in to our website. Then the user want to get the CDA data from his/her healthcare ...
3
votes
2answers
118 views

How to obtain airline data

I am looking to obtain data on daily (or even weekly) airfares but am not sure if this type of data publicly exists or not. The bureau of transportation seems to have some of this type of data, but ...
3
votes
0answers
14 views

Rules and regulations of all airlines in one site for consumers to compare

There are lot of rules and regulations that airlines have. At the same time, due to business dynamics and other changes in the environment, the rules get changed. It would be useful if there is a site ...
3
votes
1answer
60 views

Data about what kind of entities appear on public web pages

Schema.org is a vocabulary "for describing the kind of entities the most common web applications need" (source). But which entities would that be? Intuitively I’d say the most common entities are ...
5
votes
1answer
77 views

Lobby meetings of European Commission - Integrity Watch

The staff of the European Commission is obliged to keep a register of their meetings with lobbyists. The site Integrity Watch administered by Transparency International keeps track of these meetings ...
1
vote
0answers
15 views

Data on jcpenneys sales

I'm looking for a source for prices on jcpenneys items from the years 2010-2015. This is to investigate their fair and square marketing disaster. However, I cannot find any catalogs online or any sort ...
3
votes
2answers
107 views

Scraping pages to get prices can be considered as OpenData

I have a discussion in a Google Group about the next thing: One person wants to scrape multiple web sites to gather prices of different types of technology products (cell phones, TVs, computers, ...
6
votes
3answers
141 views

Data sets for predicting home value

I'm working on a project for predicting the cost of houses. I'm looking for data sets enumerating the number of schools, hospitals, parks and any other condition that may affect the price of houses (e....
7
votes
1answer
82 views

List of political blogs

I'm analyzing political texts and I need the writings of random political authors. So far I found a dataset of four political blogs normalized for text analysis. But that's not enough. I thought ...
3
votes
2answers
105 views

Employer's website is “web scraping” from another site, can I be liable even though I did not write that code?

So recently I came about an article about web scraping, and it said how most of the time scraping is illegal. I did not know this in past, and did not know the website I was working on was using a ...
2
votes
0answers
156 views

Is the FIFA 16 Ultimate Team database available anywhere?

I know there's already websites such as Futhead that can act as a database. However, I'm looking for something (preferrably a single file) that can actually be parsed, as I'm looking to make my own ...
7
votes
2answers
122 views

Creating data from web tables with import.io failed - other tools?

I found this site with solar system moon orbit data: Table of moons in solar system. I ran that through the http://import.io site and it only came up with Jupiter data. Is there a more comprehensive ...
4
votes
0answers
90 views

Memex open data (Dark web)

Has any data been created by Memex that is available for open use? Is Memex even available to be used in order to create open data of the "dark web"?
2
votes
2answers
6k views

Previous Craigslist Postings

Is there a way to access previous craigslist postings in bulk? Particularly interested in being able to select certain categories and download all listings at certain time periods for that category. ...
3
votes
3answers
909 views

Datasets: Is there any public data-set for FAQ (Frequently asked questions) in different domains?

My thesis is about Analysis and Auto generation of FAQ in different domains. For conducting experiments, I need high volume of FAQs. That's the reason I am looking for a publicly available data-set ...
4
votes
0answers
94 views

The graph of the web: the dataset of links between web pages

I was looking for the dataset of links between web pages and found these options: Common Crawl: Public web crawling data. It includes "the HTTP headers returned and the links (including the type of ...
7
votes
2answers
229 views

Why is my scraper returning inconsistent results and timing out?

I've been building a scraper in python (using mechanize) to get a list of banks off of: https://connectonline.asic.gov.au/RegistrySearch/faces/landing/ProfessionalRegisters.jspx The scraper is: ...
4
votes
1answer
134 views

Business performance data of web company like number of employees, revenue etc

Is there a dataset describing the #employees, #customers, revenue, investment etc for web company such as google, mozilla, yahoo, facebook? WIKI has few information like #employees, revenue. Querying ...
7
votes
2answers
530 views

Exclusion lists when crawling web directories

I have a case where I'm building a connector between a federated search engine, and an archive that's serving all of their data via HTTP. They've been kind enough to have their pipeline generate a ...
5
votes
1answer
58 views

Advertising Corpus for SVM

This is a bit specific. I want to train a support vector machine or naive bayes classifier to spot advertisements in HTML so I can clean before LDA analysis. Only a few percentage points of bad data ...
4
votes
2answers
2k views

Looking for web scraper that works on YouTube channel video catalogs

I'm trying to find a web scraper that will help me get detailed data from several YouTube channels I'm interested in researching. I've tried the trial version of Easy Web Extractor, but the processed ...
7
votes
1answer
93 views

What are the legal uses of the data from livingwage.mit.edu?

What are the legal uses of this data? For example, could I design a non-for-profit service using this data, a for-profit application that I sell subscriptions to, or a spreadsheet that I use myself ...
1
vote
0answers
268 views

Need dataset for second hand product sales of an e-commerce site like olx

I wish to do analysis of second hand products sold online please tell me from where i can get dataset. Dataset can contain fields like user-id, name of product, selling price, date of sell etc
1
vote
0answers
66 views

How do small companies access user transaction data?

I am not sure if this is relevant in this forum but I have always wonder how companies like Retention Science, Factual and several other smaller companies seem to be able to get transaction data or ...
3
votes
1answer
156 views

How to crawl data from a library website

I need to crawl research papers, abstracts, titles, and authors and store them for ontological usage. What are the steps to crawl such data?
7
votes
1answer
143 views

Web crawling to create a business

If I were to web crawl supermarkets' websites and retrieve info and create a comparison tool for sale would that be illegal?
6
votes
2answers
94 views

african newpapers for the past 20 years for machine learning

I am starting a project looking at textual analysis (topic analysis and sentiment analysis) for newspapers African countries, including Gambia, Nigeria, and Tanzania. However, I was wondering if ...
2
votes
1answer
86 views

Why and how are certain websites able to state in their terms of service that you can't crawl their data?

As an example, yelp and craigslist both have clauses in their terms of service that attempt to scare you away from trying to crawl/scrape their pages. Would they be able to legally prevent you from ...
14
votes
2answers
9k views

Data of vehicle traffic

Does anyone know/have good dataset or data feed of vehicle traffic on freeways/highways or urban area? I'm searching for data from vehicle counters that have small interval of measurement, like 1 or ...
5
votes
2answers
280 views

Do you know a technique for text mining judicial decisions?

I'm trying to find a method to mine sepecific informations in judicial decisions (judge's name, court, area of law and neutral citation number[canadian way to cite decisions uniformly across the ...
3
votes
3answers
868 views

Is it legal to crawl research papers from ACM/IEEE?

I am working on a project that need me to grab lot of research papers abstracts, titles, authors and display it on a web site of my own. I chose to get list of research papers from DBLP and then ...
10
votes
4answers
6k views

A web API user's guide for free and open data

There are many web-based Application Programming Interfaces (API) that allow data-hungry people to get their hands on all sorts of data. It's a great way to collect data as-needed, instead of looking ...
5
votes
2answers
1k views

Alexa top dataset

Is Alexa's Top available in a dataset format? Several datasets exist, but include only basic information, like domains and their positions in the top. What about other info, like category of the ...
3
votes
3answers
6k views

Twitter crawlers for tweets, retweets and social network

I am looking for some twitter crawlers for crawling the tweets/retweets and the social graph of some thousands of users. Suggestions are most welcome.
3
votes
1answer
213 views

Crawling user data from different applications

I would like to know if there is an existing tool/crawler/scraper/service to crawl the data of a particular user from different websites and applications. For example, can I crawl the data of a user A ...
10
votes
4answers
697 views

Examples of scraping from “real-world” data sources using OCR, etc?

Are there any examples of open data that have been scraped from "real-world" sources? To explain what I mean by "real-world", here are some (made up) examples and then some criteria: Train departure ...
3
votes
2answers
338 views

Open Seed for Crawl

I am currently running a crawl of the internet using Nutch. This requires a list of URLs to start as a seed. I currently have a 500k url seed. But I am looking for any open data sources to provide ...
7
votes
2answers
515 views

Open Web Crawling Dumps

I am currently using the Common Crawl dump of web crawl data. Are their any other good providers of free web crawling data that I can blend with my Common Crawl data?