summaryrefslogtreecommitdiff
path: root/crawler
AgeCommit message (Expand)AuthorFilesLines
2018-05-14Adds command line support to crawl only specific shops. (crawler)horus1-0/+5
2018-05-14Bugfix. (crawler)horus1-1/+1
2018-05-14Add feature to list all crawlable shops. (crawler)Max2-0/+20
2018-05-14Various fix, e.g. it repairs wrong image urls. (crawler)horus7-20/+49
2018-02-21Changes shop name from Whiskysite.nl to Whiskysite. (crawler)horus1-1/+1
2018-02-21Adds some docu. (crawler)horus1-4/+7
2018-02-21Prevents panic when passing a non-existent path as config file. (crawler)horus1-2/+7
2018-02-20Reintroduces debug as a config setting. (crawler)horus2-9/+19
2018-02-20Refactoring + adds a more granular log level setting. (crawler)horus5-54/+78
2018-02-20Removes Tx, because I get 'busy buffer' error. (crawler)horus3-32/+12
2018-02-20Better error reporting. (crawler)horus1-2/+3
2018-02-20Bugfix. (crawler)horus3-14/+15
2018-02-20Bugfix. (crawler)horus2-5/+10
2018-02-19Better error reporting. (crawler)horus1-4/+9
2018-02-19Less Fatal(), more Warn(). (crawler)horus3-4/+8
2018-02-19Saving scraped offers runs in Tx. (crawler)horus3-6/+27
2018-02-19Adds config option to crawl only specific shops. (crawler)horus2-1/+14
2018-02-19Bugfix. (crawler)horus3-3/+6
2018-02-19Removes dead code.horus1-14/+0
2018-02-19Adds repair function. (crawler)horus2-11/+73
2018-02-19Improves sanitizing. (crawler)horus3-5/+39
2018-02-19Bugfix. (crawler)horus2-12/+12
2018-02-19Implements retry on error. (crawler)horus1-9/+16
2018-02-19Fix nasty bug. (crawler)horus2-16/+16
2018-02-19Bugfix + better error reporting. (crawler)horus1-5/+18
2018-02-19Bugfix + detects age. (crawler)horus_arch3-4/+34
2018-02-19Changes log level while crawling from warning to info. (crawler)horus_arch8-52/+52
2018-02-17Merge branch 'master' of git.iamfabulous.de:alkobote.dehorus1-5/+4
2018-02-17Minor improvement in logging. (crawler)horus7-16/+22
2018-02-17Bugfix. (crawler)horus_arch1-2/+0
2018-02-17Fix detect duplicates query, prevents error message. (crawler)horus_arch1-3/+4
2018-02-17Messing with database constraints. (crawler)horus1-33/+29
2018-02-17Bugfix. (crawler)horus1-1/+1
2018-02-17Add error context for MC Whisky. (crawler)horus_arch1-3/+27
2018-02-17Checks for correct abv per spirit type. (crawler)horus1-2/+27
2018-02-17Bugfix in crawler for whiskysite.nl. It's now active (crawler)horus2-9/+19
2018-02-17Sets config option to disable url shorter. (crawler)horus_arch2-11/+20
2018-02-17Adds crawler for whiskysite.nl. (crawler)horus_arch2-5/+93
2018-02-17Introduces context on errors. (crawler)horus_arch15-111/+252
2018-02-17Minor refactoring. (crawler)horus_arch2-37/+30
2018-02-16Log level is now configurable. (crawler)horus3-13/+22
2018-02-16Pass the config file via command line flag. (crawler)horus1-2/+6
2018-02-16Structured logging part two. (crawler)horus10-57/+102
2018-02-16Adds structured logging. (crawler)horus17-64/+196
2018-02-15Bugfix. (crawler)horus3-3/+7
2018-02-14Bugfix. (crawler)horus2-1/+2
2018-02-14Add shop short url. (crawler)horus3-10/+19
2018-02-14Adds URL shorting with polr. (crawler)horus4-2/+82
2018-02-14Changes config params.horus1-16/+18
2018-02-14Bugfix. (crawler+website)horus1-1/+1