From memory buffer to disk as fast as possible

I would like to present a scenario and discuss suitable design patterns to address it.

Consider a simple situation: a camera records to a memory buffer for ten seconds before stopping. Once recording ends, a binary file descriptor opens, and data is transferred to disk.

A major limitation of this approach is that recordings are restricted by the available RAM size. But, frame loss may not be a problem.

To mitigate this, one potential solution is to use a dedicated thread or process for writing to disk. In this setup, the producer memory buffer is shared between the main and writer threads/processes. However, this introduces a new issue: when the writer thread locks the buffer, the camera may be unable to place new frames, leading to potential frame loss.

Question Is there a design pattern that addresses the problem highlighted in the second scenario?

Below some code examples for the two scenarios in Python.

First scenario in Python:

import io from picamera2 import Picamera2 from picamera2.encoders import Encoder as NullEncoder from picamera2.outputs import FileOutput # Init camera cam = Picamera2() # Init memory buffer mem_buff = io.BytesIO() mem_out = FileOutput(mem_buff) # Open camera cam.start() # Just writes frames without encoding i.e.: BGR888 encoder = NullEncoder() # Recording time to_record = 10 print(f"Start recording for {to_record} seconds") cam.start_recording(encoder, mem_out) time.sleep(to_record) cam.stop_recording() print("Finish recording") cam.close() # Begin data transfer to disk out_fpath = "video.bin" disk_transfer_start = time.perf_counter() with open(out_fpath, "wb") as fd: fd.write(mem_buff.getvalue()) disk_transfer_el = time.perf_counter() - disk_transfer_start print(f"Data transfer took {disk_transfer_el} sec") # Get a sense of how many frames are missing totbytes = os.path.getsize(out_fpath) byteel = 2304*1296*3 # (frame_width * frame_height * num_channels) num_frames = totbytes / byteel print(f"Video has {num_frames} frames") 

A possible implementation of the second scenario in Python:

import io from threading import Thread, Event, Lock from picamera2 import Picamera2 from picamera2.encoders import Encoder as NullEncoder from picamera2.outputs import FileOutput def disk_writer(mem_buff: io.BytesIO, bin_fd, write_interval: int, stop_event: Event, lock: Lock): while not stop_event.is_set(): start_loop = time.perf_counter() lock.acquire() curr_buff_pos = mem_buff.tell() lock.release() if curr_buff_pos > 0: lock.acquire() bin_fd.write(mem_buff.getvalue()) mem_buff.seek(0) mem_buff.truncate(0) lock.release() elapsed = time.perf_counter() - start_loop if elapsed < write_interval: time.sleep(write_interval - elapsed) if mem_buff.tell() > 0: bin_fd.write(mem_buff.getvalue()) mem_buff.seek(0) mem_buff.truncate(0) # Init camera cam = Picamera2() # Init memory buffer mem_buff = io.BytesIO() mem_out = FileOutput(mem_buff) # Get output file descriptor bin_fd = open("video.bin", "wb") # Open camera cam.start() # Create writing thread and start stop_event = Event() lock = Lock() write_interval = 5 writer_thread = Thread(target=disk_writer, args=(mem_buff, bin_fd, write_interval, stop_event, lock)) writer_thread.start() # Just writes frames without encoding i.e.: BGR888 encoder = NullEncoder() # Recording time to_record = 10 print(f"Start recording for {to_record} seconds") cam.start_recording(encoder, mem_out) time.sleep(to_record) cam.stop_recording() print("Finish recording") stop_event.set() writer_thread.join() cam.close() bin_fd.close() # Get a sense of how many frames are missing totbytes = os.path.getsize(out_fpath) byteel = 2304*1296*3 # (frame_width * frame_height * num_channels) num_frames = totbytes / byteel print(f"Video has {num_frames} frames")