Skip to content

← All questions

12k rows of reference data: commit the CSV or import it into a db dataset?

Asked 2,870 views0 answers

Our pricing rules are about twelve thousand rows and they change roughly weekly, on a schedule that has nothing to do with our release schedule. Today they are a committed CSV, resolved per environment, and it works — scenarios load a row by rule id and assert the computed price.

What I am unsure about is whether it should stay that way. Arguments I can make on both sides:

  • The CSV is diffable and a review of a pricing change is a review of a diff, which is genuinely useful. It also needs no database in CI, and our API-layer runs currently need nothing but the runner.
  • A db dataset can be reloaded with sdods data import --to table --truncate without a commit, and the weekly churn currently produces a weekly pull request that nobody reads and everybody approves.

Has anyone moved a fixture of roughly this size and regretted it in either direction? Specifically interested in whether the parse cost per worker turns out to matter at this scale, or whether that is the wrong thing to be thinking about.

asked by Paulo Mendes97 ·

No answers yet

Nobody has answered this one. If you have hit the same thing and worked it out, that is worth writing down.

Ask a related question

Have the same question?

Ask it with the command you ran and the output you got, and it will be answered here.

Ask a question

Related