hnimdbbot/src/imdbdata.go, branch main

feat: add three-level logging with per-request debug output

2026-06-26T12:14:52+00:00

- New --log-level flag: debug (default info), info, silent
  debug: every API request logged (method, URL, status, duration)
  info:  normal events (batch progress, entry counts, summaries)
  silent: only warnings and fatal errors
- Replaced all log.Printf/Fatalf calls with level-gated helpers
- API request timing added to queryWikiArticle, queryWikidataBatch, downloadFile
- Retries and backoff logged in debug mode

fix: use INSERT IGNORE for imdb_genre to handle re-runs

2026-06-24T02:53:05+00:00

The previous run left partial data after a mid-transaction rollback.
INSERT IGNORE makes the junction table insert idempotent.

feat: adapt genre code for n:m relation via imdb_genre

2026-06-24T02:40:22+00:00

- genre table: (id, name) with unique name constraint
- imdb_genre table: (id, imdb_id, genre_id) junction table
- Upsert genres via INSERT ... ON DUPLICATE KEY UPDATE
- Link via imdb_genre using LAST_INSERT_ID
- Check missing genres via LEFT JOIN imdb_genre

feat: populate genre table from title.basics.tsv

2026-06-24T02:26:46+00:00

- Parse genres field (rec[8]) from title.basics.tsv, split by comma
- Insert into genre table via SELECT to resolve imdb.id from imdb_id
- Update fetchAndUpdateImdbData to check for missing genres too
- Skip download if TSV already exists (supports stubbed downloadFile)

fix: correct TSV parsing — use line-by-line reader and proper column indices

2026-06-24T02:21:20+00:00

- Replace csv.Reader with bufio.Scanner to avoid quote-parsing issues
  that skipped ~355 entries (e.g. tt1853728 was on line 4.8M and got
  lost when csv.Reader encountered malformed quoted fields earlier)
- Fix column indices: startYear=rec[5], runtimeMinutes=rec[7]
  (was rec[4]/rec[5] which mapped to isAdult/startYear)
- Update basics for ALL imdb entries, not just those missing ratings

chore: delete .gz files after extracting in downloadImdbDatasets

2026-06-24T01:52:01+00:00

move download path

2026-06-24T01:48:42+00:00

feat: fetchAndUpdateImdbData — download IMDB datasets and populate imdb table

2026-06-24T01:46:14+00:00

- Check for imdb entries with NULL average_rating
- Download title.basics.tsv.gz and title.ratings.tsv.gz to imdbdata/
- Decompress alongside gzip originals
- Parse only rows matching our imdb_ids (memory-efficient)
- Update: average_rating, num_votes, title_type, primary_title,
  original_title, start_year, runtime_minutes
- Results: 3394 ratings, 3093 basics updated out of 3448 entries