Content removal bias in web scraped data: A solution applied to real estate ads
Description
propose a solution to content removal bias in statistics from web scraped data. Content removal bias occurs
when data is removed from the web before a scraper is able to collect it. The solution I propose is based on inverse
probability weights, derived from the parameters of a survival function with complex forms of data censoring. I apply
this solution to the calculation of the proportion of newly built dwellings with web scraped data on Luxembourg, and I
run a counterfactual experiment and a Montecarlo simulation to confirm the findings. The results show that the extent
of content removal bias is relatively small if the scraping occurs frequently compared with the online permanence of
the data; and that it grows larger with less frequent scraping.
Toolbox-ID
jd000112-0001
Identifier(s)
https://doi.org/10.1515/openec-2022-0119
https://tustorage.ulb.tu-darmstadt.de/handle/tustorage/44
https://tustorage.ulb.tu-darmstadt.de/handle/tustorage/44
Publisher
De Gruyter
License
https://creativecommons.org/licenses/by/4.0/
Subject(s)
Scraping
Inverse probability weighting
Crawling
Bias
Survival analysis
Missing data
Real estate
Luxembourg
Big data
Housing
Inverse probability weighting
Crawling
Bias
Survival analysis
Missing data
Real estate
Luxembourg
Big data
Housing