2

Suppose I have some utility functions defined as:

(defun write-bytes-to-file (bytes file-path bits) (with-open-file (stream file-path :direction :output :if-does-not-exist :create :if-exists :append :element-type (list 'unsigned-byte bits)) (dolist (b bytes)) (write-byte b stream)))) (defun read-file-bytes-to-list (file-path bits) (with-open-file (stream file-path :direction :input :element-type (list 'unsigned-byte bits)) (read-bytes-to-list stream nil)))

Furthermore, let's say I run:

(write-bytes-to-file '(65 65 65) "foo.txt" 8) (write-bytes-to-file '(512) "foo.txt" 9)

Now this leaves me with a variable byte-width file with three 8-bit bytes, and one with 9-bit width. Normally I'd use my read-file-bytes-to-list function with a bit-width input to read the file with the correct alignment, but in this case I can't really do that. Is there a built-in way I can change the byte alignment mid-stream in Common LISP? Essentially I'd like to read the integers back as intended (65, 65, 65, and 512). Thanks for your help. I'm using the SBCL implementation.

EDIT: If there is no convenient way to handle this using the standard LISP libraries/utilities, I realize that I most likely will have to handle this with bit-wise operations; that's fine. I was just wondering if there is a better way to do it, before doing so.

Improve this question

1 Answer 1

Reset to default
3

First of all, please consider using lisp-binary when encoding/decoding binary formats.

I modified your writing function to fix the parentheses and print the actual stream element type:

(defun write-bytes-to-file (bytes file-path bits) (with-open-file (stream file-path :direction :output :if-does-not-exist :create :if-exists :append :element-type (list 'unsigned-byte bits)) (print (stream-element-type stream)) (dolist (b bytes) (write-byte b stream)))))

With this function, I built a binary file:

USER> (when-let (file (probe-file "/tmp/test.data")) (delete-file file)) T USER> (write-bytes-to-file '(1 1 1 1) "/tmp/test.data" 9) (UNSIGNED-BYTE 16) NIL USER> (write-bytes-to-file '(1 1 1 1) "/tmp/test.data" 8) (UNSIGNED-BYTE 8) NIL USER>

As you can see, encoding (unsigned-byte 9) elements is done by opening a stream that uses bytes of size 16. If you look at the generated file with hexdump:

$ hexdump /tmp/test.data 0000000 0001 0001 0001 0001 0101 0101 000000c

You can see that the first 4 ones are encoded on 16 bits words, followed by 4 ones encoded on 8 bits. In order to decode the data, you need to open it several times (as for writing) and seek the appropriate position in the file. The following was tested on SBCL, and there is not guarantee it will work portably:

(defun read-file-bytes-to-list (file-path bits count &optional (bits-offset 0)) (with-open-file (stream file-path :direction :input :element-type (list 'unsigned-byte bits)) (destructuring-bind (_ bits) (stream-element-type stream) (declare (ignore _)) (loop initially (file-position stream (/ bits-offset bits)) repeat count collect (read-byte stream) into bytes finally (return (values bytes (* bits (file-position stream))))))))

At the end of the function, we return both the decoded bytes and the file position, expressed in bits, after reading is done. Returning the amount of bits is necessary, since file-position returns a multiple of the stream element's size.

When we reopen the file, we give the last offset as bits, and the new stream element type is used to compute the offset to give to file-position.

For example:

USER> (read-file-bytes-to-list "/tmp/test.data" 9 4) (1 1 1 1) 64 USER> (read-file-bytes-to-list "/tmp/test.data" 8 4 64) (1 1 1 1) 96
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Thank you for the very lucidly explained answer. This approach seems very sensible, and idiomatic too.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.