2

Summary: I want to transfer a file which changes over time, probably faster than the available connection allows a successful transfer.

Setup:

  • A remote device with bad network reliability and speed most of the time, but with "windows" of decent connection (able to scp ~>100 mb overnight).
  • The device could spend anywhere from minutes to days (but usually not weeks) without reception.
  • The device logs to a sqlite database with current date-time, several times a day.
  • The db is around 5-10 mb, and I've found it can be compressed to about 10% of its original size experimentally.
  • Database could be reset/deleted (low frequency, but happens, eg once per month or less) by a third party.
  • I need to have a local copy of the database locally.

Current solution:

I have a script which does the following:

  1. Connect via ssh to the device.
  2. "snapshot" the db to a compressed file, unless a snapshot already exists.
  3. rsync -ruvhP the snapshot to a local drive (this starts a new transfer, or continues an interrupted one)
  4. untar transferred file to a sqlite file.
  5. Read first date-time entry of db. Use it as a unique name to rename the local database.
  6. Delete remote snapshot (but not the local one). This allows to transfer new rows from the db (or the whole db if it was reset), but waste bandwidth "re-downloading" data we already have (the old rows if it wasn't reset). If the newly created snapshot is exactly like the old one, rsync does not waste bandwith. The script did create+destroy "uselessly" the snapshot though...

Problem: Depending on the number in the implementation above where the connection fails, there are lots of edge cases, producing unusable corrupted/incomplete db, etc. This is starting to look a lot like reinventing the wheel...

Questions:

  1. Are there libraries/solutions to this problem? I imagine services like dropbox and similar have situations like these solved, except for:
  • Ideally, I would like to avoid installing services on the remote device.
  • In the case the remote device "resets" the db, the new remote version would overwrite the local, without keeping the local(s) - At least in the dropbox case.
  1. Any alternatives patterns/better implementations to deal with edge cases?
  2. What about simpler storage alternatives, like CSV (to allow incremental transfers), or rsync --compressed?
Improve this question
1

1 Answer 1

Reset to default
2

Delta

What you want to do is transfer the delta. That is the difference between your local copy and the remote copy.

One way to do this is via an Event Log. The log contains each operation/data change applied in order. The client maintains the id of the last event received, when syncing the events since that event are retrieved.

Another way to do this is to create a snapshot and determine the difference between it and the previous snapshot. Transmit those differences.

Of course the amount of change might be so high that your current solution of compressing a full db might be smaller. So it does make sense to hae some smarts to discard stupid events, or choose between the delta stream and a compressed copy.

Checksums

The first issue is detecting that the copy is bad. Checksums work a treat here. How you apply them will depend on what you are transferring.

If you are transferring the full db, then a single checksum is enough. Trying to checksum each block or smaller unit isn't helpful because there is no meaningful recovery method. You have to recopy the file regardless.

If you chunk the db into segments (split into separate files) then it makes sense to checksum each segment. If one segment is corrupted you need only recopy that segment.

If you are doing this on an event stream then you can checksum each event (or block of events). This is even better because you can at least restore the first X uncorrupted events while you are attempting to retrieve that xth corrupted event. Its also helpful in that you know at what time your db matched the remote (due to timestamps).

Self-Healing

These take checksums to their next level. They add a lot more data but if the connection regularly flips bits, it has a chance of restoring the bits back to their original state, thus avoiding the need to retransmit.

Some of the simpler schemes add about 3/4th more data, so it can be more efficient than retransmitting the same copy twice, with the hope that one copy gets through error free. But it will depend on the kinds of errors you are getting over the connection.

Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.