Content removal bias in web scraped data: A solution applied to real estate ads
| dc.contributor.author | Marconi, Gabriele | |
| dc.date.accessioned | 2023-03-24T15:26:13Z | |
| dc.date.available | 2023-03-24T15:26:13Z | |
| dc.date.issued | 2022-12-31 | |
| dc.description.abstract | propose a solution to content removal bias in statistics from web scraped data. Content removal bias occurs when data is removed from the web before a scraper is able to collect it. The solution I propose is based on inverse probability weights, derived from the parameters of a survival function with complex forms of data censoring. I apply this solution to the calculation of the proportion of newly built dwellings with web scraped data on Luxembourg, and I run a counterfactual experiment and a Montecarlo simulation to confirm the findings. The results show that the extent of content removal bias is relatively small if the scraping occurs frequently compared with the online permanence of the data; and that it grows larger with less frequent scraping. | en |
| dc.identifier.issn | 2451-3458 | |
| dc.identifier.other | jd000112-0001 | |
| dc.identifier.uri | https://doi.org/10.1515/openec-2022-0119 | |
| dc.identifier.uri | https://tustorage.ulb.tu-darmstadt.de/handle/tustorage/44 | |
| dc.publisher | De Gruyter | en |
| dc.relation.ispartofseries | Open Economics; | |
| dc.rights.uri | https://creativecommons.org/licenses/by/4.0/ | |
| dc.subject | Scraping | en |
| dc.subject | Inverse probability weighting | en |
| dc.subject | Crawling | en |
| dc.subject | Bias | en |
| dc.subject | Survival analysis | en |
| dc.subject | Missing data | en |
| dc.subject | Real estate | en |
| dc.subject | Luxembourg | en |
| dc.subject | Big data | en |
| dc.subject | Housing | en |
| dc.title | Content removal bias in web scraped data: A solution applied to real estate ads | en |
| dc.type | Article | en |
| dcat.distribution.pdf | https://tustorage.ulb.tu-darmstadt.de/handle/tustorage/45 | |
| dspace.entity.type | Dataset | |
| relation.isDistributionOfDataset | 11bc9de1-92de-465d-862f-b229d92c6bdf | |
| relation.isDistributionOfDataset.latestForDiscovery | 11bc9de1-92de-465d-862f-b229d92c6bdf | |
| wdm.type.level | dataset |
Files
License bundle
1 - 1 of 1
No Thumbnail Available
- Name:
- license.txt
- Size:
- 1.71 KB
- Format:
- Item-specific license agreed upon to submission
- Description: