summaryrefslogtreecommitdiff
path: root/crawler/scrape.go
AgeCommit message (Expand)AuthorFilesLines
2021-01-12Finalized the crawler for the new shop.Maximilian Möhring1-0/+2
2019-05-15Improves structured logging. (crawler)Maximilian Möhring1-4/+5
2019-05-15Renames functions. (crawler)Maximilian Möhring1-5/+5
2019-05-14Changes imports to lower case. (crawler)Maximilian Möhring1-1/+1
2018-06-16Adds scraper for Drankdozijn. (crawler)horus1-0/+2
2018-06-15Removes unnecessary code. (crawler)horus1-4/+0
2018-06-15Introduces config for user agent, robots.txt and crawler delay. (crawler)horus1-0/+28
2018-05-14Various fix, e.g. it repairs wrong image urls. (crawler)horus1-1/+1
2018-02-20Removes Tx, because I get 'busy buffer' error. (crawler)horus1-21/+3
2018-02-20Bugfix. (crawler)horus1-7/+7
2018-02-19Saving scraped offers runs in Tx. (crawler)horus1-2/+22
2018-02-19Bugfix. (crawler)horus1-8/+8
2018-02-19Implements retry on error. (crawler)horus1-9/+16
2018-02-17Bugfix in crawler for whiskysite.nl. It's now active (crawler)horus1-2/+2
2018-02-17Introduces context on errors. (crawler)horus_arch1-12/+12
2018-02-16Structured logging part two. (crawler)horus1-1/+1
2018-02-16Adds structured logging. (crawler)horus1-4/+4
2018-02-15Bugfix. (crawler)horus1-2/+2
2018-02-12Commits missing function parameter. (crawler)horus1-1/+1
2018-02-08Better parallelization.Max1-22/+22
2018-02-07Adds sanitizer.horus_arch1-0/+2
2018-02-07Parallelized the crawler.horus_arch1-0/+69