Skip to content

← All questions

12k rows of reference data: commit the CSV or import it into a db dataset?

Asked 2,870 views0 answers

Our pricing rules are about twelve thousand rows and they change roughly weekly, on a schedule that has nothing to do with our release schedule. Today they are a committed CSV, resolved per environment, and it works — scenarios load a row by rule id and assert the computed price.

What I am unsure about is whether it should stay that way. Arguments I can make on both sides:

  • The CSV is diffable and a review of a pricing change is a review of a diff, which is genuinely useful. It also needs no database in CI, and our API-layer runs currently need nothing but the runner.
  • A db dataset can be reloaded with sdods data import --to table --truncate without a commit, and the weekly churn currently produces a weekly pull request that nobody reads and everybody approves.

Has anyone moved a fixture of roughly this size and regretted it in either direction? Specifically interested in whether the parse cost per worker turns out to matter at this scale, or whether that is the wrong thing to be thinking about.

asked by Paulo Mendes97 ·

No answers yet

Know the answer?

Reviewed before it appears. Wrap commands and output in triple backticks and they will render as a code block.

Have a different question? Ask it.

Related