Skip to content

← All questions

12k rows of reference data: commit the CSV or import it into a db dataset?

Example threads. These questions and answers were written by the SDODS team to document real problems and how to fix them. The people named are illustrative, not real users. Questions asked on this site are marked new.

Our pricing rules are about twelve thousand rows and they change roughly weekly, on a schedule that has nothing to do with our release schedule. Today they are a committed CSV, resolved per environment, and it works — scenarios load a row by rule id and assert the computed price.

What I am unsure about is whether it should stay that way. Arguments I can make on both sides:

  • The CSV is diffable and a review of a pricing change is a review of a diff, which is genuinely useful. It also needs no database in CI, and our API-layer runs currently need nothing but the runner.
  • A db dataset can be reloaded with sdods data import --to table --truncate without a commit, and the weekly churn currently produces a weekly pull request that nobody reads and everybody approves.

Has anyone moved a fixture of roughly this size and regretted it in either direction? Specifically interested in whether the parse cost per worker turns out to matter at this scale, or whether that is the wrong thing to be thinking about.

No answers yet

Know the answer?

Wrap commands and output in triple backticks and they will render as a code block.

Checking whether you’re signed in…

Have a different question? Ask it.

Related