summaryrefslogtreecommitdiff
path: root/crawler
AgeCommit message (Expand)AuthorFilesLines
2019-05-14Refactores logging functions. (crawler)Maximilian Möhring13-102/+101
2019-05-14Changes imports to lower case. (crawler)Maximilian Möhring10-10/+10
2019-05-12Minor fix. (crawler)Max1-0/+7
2019-05-10Improves edge case in spirit type detecting. (crawler)horus1-0/+8
2019-05-07Improves detection of different spirit types and removes a lot of dead code.horus2-126/+89
2019-05-07Better spirit type detection. (crawler)horus3-8/+117
2019-04-16Rewrites crawler for drankdozijn. (crawler)horus1-280/+131
2019-01-12Better handling for nil interfaces. (crawler)horus1-26/+36
2019-01-11Crawler for Drankdozijn now uses the official API. (crawler)Max1-160/+365
2018-09-16Enhances crawler for Rum & Co. (crawler)horus1-7/+46
2018-09-16Fix crawler for Rum & Co. (crawler)horus1-3/+7
2018-09-16Fix champagne offerings for Drankdozijn. (crawler)horus1-2/+2
2018-09-16Fix crawler for Drankdozijn. (crawler)horus1-2/+14
2018-09-16Rename flags. Adds new flag to exclude shops. (crawler)horus3-12/+28
2018-09-16Bug fix. (crawler)horus1-1/+2
2018-07-09Improves name sanitizing. (crawler)Max1-8/+16
2018-06-16Corrects short url. (crawler)horus1-1/+1
2018-06-16Bugfix. (crawler)horus1-3/+14
2018-06-16Adds champagner / Drankdozijn. (crawler)horus1-2/+8
2018-06-16Adds scraper for Drankdozijn. (crawler)horus3-0/+202
2018-06-16Removes validating abv based of spirit type. (crawler)horus1-12/+14
2018-06-16Detects cl in sanitize_name(). (crawler)horus1-1/+1
2018-06-16Improves sanitizing function. (crawler)horus1-2/+8
2018-06-16Adds support for cl. (crawler)horus1-3/+25
2018-06-15Removes unnecessary code. (crawler)horus2-9/+0
2018-06-15Introduces config for user agent, robots.txt and crawler delay. (crawler)horus12-26/+109
2018-06-15Introduces central crawler config. (crawler)Max9-32/+14
2018-06-15Improves debugging output. (crawler)Max1-0/+1
2018-06-15Tries to validate image url by making head request. (crawler)horus1-0/+26
2018-06-15Fix because changed html. (crawler)horus2-1/+5
2018-05-23Reflects changes in TWS url structure. (crawler)horus1-1/+1
2018-05-23Repairs --list-shops flag. (crawler)horus1-1/+1
2018-05-19Bugfix. (crawler)horus1-1/+1
2018-05-19Bugfix. (crawler)horus2-2/+3
2018-05-14Renames command line flags. Suppress unhelpful error message. (crawler)Max1-4/+8
2018-05-14Adds command line support to crawl only specific shops. (crawler)horus1-0/+5
2018-05-14Bugfix. (crawler)horus1-1/+1
2018-05-14Add feature to list all crawlable shops. (crawler)Max2-0/+20
2018-05-14Various fix, e.g. it repairs wrong image urls. (crawler)horus7-20/+49
2018-02-21Changes shop name from Whiskysite.nl to Whiskysite. (crawler)horus1-1/+1
2018-02-21Adds some docu. (crawler)horus1-4/+7
2018-02-21Prevents panic when passing a non-existent path as config file. (crawler)horus1-2/+7
2018-02-20Reintroduces debug as a config setting. (crawler)horus2-9/+19
2018-02-20Refactoring + adds a more granular log level setting. (crawler)horus5-54/+78
2018-02-20Removes Tx, because I get 'busy buffer' error. (crawler)horus3-32/+12
2018-02-20Better error reporting. (crawler)horus1-2/+3
2018-02-20Bugfix. (crawler)horus3-14/+15
2018-02-20Bugfix. (crawler)horus2-5/+10
2018-02-19Better error reporting. (crawler)horus1-4/+9
2018-02-19Less Fatal(), more Warn(). (crawler)horus3-4/+8