summaryrefslogtreecommitdiff
path: root/crawler
AgeCommit message (Collapse)AuthorFilesLines
2018-06-16Adds support for cl. (crawler)horus1-3/+25
2018-06-15Removes unnecessary code. (crawler)horus2-9/+0
2018-06-15Introduces config for user agent, robots.txt and crawler delay. (crawler)horus12-26/+109
2018-06-15Introduces central crawler config. (crawler)Max9-32/+14
2018-06-15Improves debugging output. (crawler)Max1-0/+1
2018-06-15Tries to validate image url by making head request. (crawler)horus1-0/+26
2018-06-15Fix because changed html. (crawler)horus2-1/+5
2018-05-23Reflects changes in TWS url structure. (crawler)horus1-1/+1
2018-05-23Repairs --list-shops flag. (crawler)horus1-1/+1
2018-05-19Bugfix. (crawler)horus1-1/+1
2018-05-19Bugfix. (crawler)horus2-2/+3
2018-05-14Renames command line flags. Suppress unhelpful error message. (crawler)Max1-4/+8
2018-05-14Adds command line support to crawl only specific shops. (crawler)horus1-0/+5
2018-05-14Bugfix. (crawler)horus1-1/+1
2018-05-14Add feature to list all crawlable shops. (crawler)Max2-0/+20
2018-05-14Various fix, e.g. it repairs wrong image urls. (crawler)horus7-20/+49
2018-02-21Changes shop name from Whiskysite.nl to Whiskysite. (crawler)horus1-1/+1
2018-02-21Adds some docu. (crawler)horus1-4/+7
2018-02-21Prevents panic when passing a non-existent path as config file. (crawler)horus1-2/+7
2018-02-20Reintroduces debug as a config setting. (crawler)horus2-9/+19
2018-02-20Refactoring + adds a more granular log level setting. (crawler)horus5-54/+78
2018-02-20Removes Tx, because I get 'busy buffer' error. (crawler)horus3-32/+12
2018-02-20Better error reporting. (crawler)horus1-2/+3
2018-02-20Bugfix. (crawler)horus3-14/+15
2018-02-20Bugfix. (crawler)horus2-5/+10
2018-02-19Better error reporting. (crawler)horus1-4/+9
2018-02-19Less Fatal(), more Warn(). (crawler)horus3-4/+8
2018-02-19Saving scraped offers runs in Tx. (crawler)horus3-6/+27
2018-02-19Adds config option to crawl only specific shops. (crawler)horus2-1/+14
2018-02-19Bugfix. (crawler)horus3-3/+6
2018-02-19Removes dead code.horus1-14/+0
2018-02-19Adds repair function. (crawler)horus2-11/+73
2018-02-19Improves sanitizing. (crawler)horus3-5/+39
2018-02-19Bugfix. (crawler)horus2-12/+12
2018-02-19Implements retry on error. (crawler)horus1-9/+16
2018-02-19Fix nasty bug. (crawler)horus2-16/+16
2018-02-19Bugfix + better error reporting. (crawler)horus1-5/+18
2018-02-19Bugfix + detects age. (crawler)horus_arch3-4/+34
2018-02-19Changes log level while crawling from warning to info. (crawler)horus_arch8-52/+52
2018-02-17Merge branch 'master' of git.iamfabulous.de:alkobote.dehorus1-5/+4
2018-02-17Minor improvement in logging. (crawler)horus7-16/+22
2018-02-17Bugfix. (crawler)horus_arch1-2/+0
2018-02-17Fix detect duplicates query, prevents error message. (crawler)horus_arch1-3/+4
2018-02-17Messing with database constraints. (crawler)horus1-33/+29
2018-02-17Bugfix. (crawler)horus1-1/+1
2018-02-17Add error context for MC Whisky. (crawler)horus_arch1-3/+27
2018-02-17Checks for correct abv per spirit type. (crawler)horus1-2/+27
2018-02-17Bugfix in crawler for whiskysite.nl. It's now active (crawler)horus2-9/+19
2018-02-17Sets config option to disable url shorter. (crawler)horus_arch2-11/+20
2018-02-17Adds crawler for whiskysite.nl. (crawler)horus_arch2-5/+93