Skip to main content
Software Engineering

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

7
  • "No old data should be deleted, only new data should be added..." - an absurd principle on which to design any computer system. Commented Feb 1, 2024 at 14:16
  • People write many things; they won’t always be applicable to your situation. So write a PoC already and bench it, measure it. It’s not that hard. Configure N fake sensors to produce M observations per second. Crank up the rate till Postgres starts to fall behind. Now you have a design constraint. // Hint: to append more rows per second to the log, send more than one INSERT per COMMIT. It can also be helpful to append such records to local text file, to avoid losing observations due to power fail or reboot. See also “fsync”. Commented Feb 1, 2024 at 19:06
  • @flo, not sure whether that's sarcasm haha, but on a serious note, you should never design any system on the assumption of infinite accumulation of data. A system that is otherwise quite functional and performant over a very reasonable period of time, may well eventually grind to a halt if you don't think up-front about how old data will be archived or purged. I've worked in a few businesses which have suffered pain like this. Commented Feb 1, 2024 at 19:19
  • 1
    @J_H thank you very much! That actually helps me a lot. As I said, I'm a student and still have a lot to learn. Getting experience, help and approaches from an experienced person is worth its weight in gold. Thank you! I'll get right on it and give it a try. Thanks for the tip with the local file! Commented Feb 1, 2024 at 23:31
  • Thank you too @Steve for the answer. Please don't take offense at my sarcasm ;) But I have to agree with you and I have to think about this point. I don't know how off the top of my head, because theoretically I would also like to access "very" old sensor data so that I can display it as a graph. But if there are too many, you will probably have to cut back and limit this to 1 year or so and delete old ones. Thank you for taking the time to share your experience with me. Commented Feb 1, 2024 at 23:42