diff options
| author | dev | 2026-06-26 03:26:07 +0200 |
|---|---|---|
| committer | dev | 2026-06-26 03:26:07 +0200 |
| commit | 8e2d742e59b3923852e1ef6e7a5e2ee1de14ce45 (patch) | |
| tree | 9cbcab194f21431255222c7785e15795e494a728 /src/imdbdata.go | |
| parent | f3ed9396113de4d81ae0fadabc10ae59f7076862 (diff) | |
| download | hnimdbbot-8e2d742e59b3923852e1ef6e7a5e2ee1de14ce45.tar.gz | |
refactor: pipeline SPARQL and wiki data in parallel
- Merge fetchWikiArticles + fetchWikiArticlesData into one pipeline
- SPARQL producer fetches batches, commits each to DB, forwards resolved articles
- Wiki data consumer runs concurrently, fetching at 2s/request
- Each SPARQL batch commits independently (no global transaction)
- Rate limits respected for both Wikidata SPARQL and wiki server
- No parallel requests to either endpoint
Diffstat (limited to 'src/imdbdata.go')
0 files changed, 0 insertions, 0 deletions