0

I have computer 1 logging voltage data to a file volts.json every second.

My second computer connects via ssh and grabs that file every 5 minutes. Splunk indexes that file for a dashboard.

Is scp efficient in this manner, if so then ok. Next is how to manage the file and keep it small without growing to 2mb lets say? is there a command to roll off the earlier logs and keep the newest?

the json looks like this right now:

{ "measuredatetime": "2022-06-27T18:00:10.915668", "voltage": 207.5, "current_A": 0.0, "power_W": 0.0, "energy_Wh": 2, "frequency_Hz": 60.0, "power_factor": 0.0, "alarm": 0 } { "measuredatetime": "2022-06-27T18:00:11.991936", "voltage": 207.5, "current_A": 0.0, "power_W": 0.0, "energy_Wh": 2, "frequency_Hz": 59.9, "power_factor": 0.0, "alarm": 0 }
Improve this question
4
  • Yes, scp is ok, you might want to add -C if you're not doing it already, since that kind of data will be compressed a lot. The other questions depend on the program doing the logging and the program doing the rendering. You could also mount via sshfs and follow the file directly, for example. Commented Jun 28, 2022 at 0:49
  • How is the data logged? Is the program that logs the data keeping the file open, or does it re-open the file for writing every time. Is it seeking to the end and then writing, or is it appending? How crucial is it that you get every data point, or is it ok if one or two entries are dropped at the time of rotating the log? Commented Jun 28, 2022 at 7:13
  • You can use rsync, as stated here rsync support lots of compressions Commented Jun 28, 2022 at 11:57
  • -EduardoTrápani thanks, didnt know about sshfs that is actually cool. -Kusalananda Its a python script file write append mode Thanks, I thought about rsync but wasnt sure -k.Cyborg Commented Jun 28, 2022 at 12:21

1 Answer 1

Reset to default
1
  • To keep directories synchronized through ssh,the typical tool is rsync.
  • To roll log files and save space, logrotate is well dedicated.
  • To secure an unattended simple task through ssh, .ssh/authorized_keys with forced command is an excellent practice.

Example:

  • set /etc/logrotate.d/volts file (imitate classical syslog settings)

  • create a task-dedicated key pair with ssh-keygen; in this particular case, you do not want a passphrase; security is ensured by autorized_keys restrictions

  • in .ssh/authorized_keys, set:

    command="rsync --server --sender -logDtpre.iLsf . /path/to/volts/" ssh-rsa AAAAB3NzaC1yc2E[...pubkey...] blabla

  • on the other side, in crontab, set

    rsync -e "ssh -i /path/to/privatekey" -a otherhost:/path/to/volts/ /path/to/volts

On computer 1, you could also replace the log file by a named pipe, make a daemon script that consumes the stream and writes safely to a file (e.g using a semaphore to manage concurrent I/O), so that you have a good control over the data integrity.

Improve this answer
3
  • Thanks for the details. I was thinking about logrotate but never used it and didnt know all the options, coming back this morning it makes sense reading your suggestion. Also rsync I thought about but didnt want to complicate syncing but looks elegant. I do have ssh working with keys so that makes scp working good. I am logging every 10s but want to do 1s. For the past 12 hours have 1.4M uncompressed file it is syncing. But as splunk indexes every 5 min (now) I dont need previous data so I can probably set the logrotate every hour and deal with KBs. Commented Jun 28, 2022 at 12:29
  • @Zippy if disk space is critical on computer 1, you should make a ssh servlet (using authorized_keys forced command) that read-and-reset the logfile, provided that the concurrent access between write and consume is managed. So that computer 1 has no logrotate need and disk space is kept as small as possible. It's a simple FIFO design. Depending on what you know about the writer process. Commented Jun 28, 2022 at 13:38
  • If writer process uses shell >> redirection or similar, consumer may just rename-read-unlink the logfile. Commented Jun 28, 2022 at 13:56

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.