Python - What is the most efficient way to generate padding?

Question

Here's the problem: I'm reading binary files in fairly large blocks (512 KiB) and wish to pad the last block with zeros whenever it is shorter than the block size.

Currently, I'm doing something like this:

bytes = f.read(self.chunksize) if len(bytes) > 0: len_diff = self.chunksize - len(bytes) if len_diff > 0: bytes += reduce(lambda x,y: x+y, ["\0" for i in range(0, len_diff)])

Obviously this is terribly inefficient since that reduce will make a lot of string concatenations. I'm wondering though, how can I achieve this with Python? In C, I'd simply calloc and be done with it.

If it isn't achievable with Python, I'm willing to convert this code into a C module and/or abandon Python entirely for this project, since it's still on the early stages.

Cheers!

EDIT: I'm feeling terrible for not remembering to use the * operator. :-)

This solution worked perfectly for me:

bytes += "\0" * len_diff

EDIT #2: Using ljust() instead simplified my code a bit, so the correct answer goes to Jeff.

docs.python.org/library/mmap.html

Ignacio Vazquez-Abrams
– Ignacio Vazquez-Abrams

2011-04-24 22:24:13 +00:00
Commented Apr 24, 2011 at 22:24 — Ignacio Vazquez-Abrams
– Ignacio Vazquez-Abrams, Commented Apr 24, 2011 at 22:24

Jeff Mercado · Accepted Answer · 2023-03-25 08:53:34Z

32

Since you're working with strings, you can use ljust() to do the padding.

readBytes = f.read(self.chunksize) if readBytes: readBytes = readBytes.ljust(self.chunksize, b'\0')

edited Mar 25, 2023 at 8:53

answered Apr 24, 2011 at 22:31

Jeff Mercado

136k34 gold badges268 silver badges282 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

João Neves Over a year ago

Awesome! Just what I was looking for. I don't do much string manipulation these days. :P

Danilo Bargen Over a year ago

In Python 3, the second parameter to ljust must be b'\0'.

John La Rooy · Accepted Answer · 2011-04-24 22:23:41Z

7

bytes += "\0"*len_diff

should help

answered Apr 24, 2011 at 22:23

John La Rooy

306k54 gold badges378 silver badges514 bronze badges

2 Comments

João Neves Over a year ago

I even feel bad for not thinking about this first. :) Will accept as soon as SO lets me.

João Neves Over a year ago

Sorry, the solution given for @Jeff is actually what I was looking for, much less hackish. This one works too, though. :)

Pavan Yalamanchili · Accepted Answer · 2011-04-24 22:24:04Z

try this.

bytes = "\0" * self.chunksize rbytes = f.read(self.chunksize) bytes[:len(rbytes)] = rbytes

or

bytes = f.read(self.chunksize) bytes += "\0" * (self.chunksize - len(bytes))

tom10 · Accepted Answer · 2011-04-24 22:25:33Z

2

How about:

bytes += "\0"*len_diff

answered Apr 24, 2011 at 22:25

tom10

69.5k11 gold badges133 silver badges143 bronze badges

Comments

Elijah · Accepted Answer · 2020-07-10 18:11:58Z

I needed to pad something encrypted. Here's how to do it.

from Crypto.Util.Padding import pad ... _bytes = pad(_bytes, block_size)

Collectives™ on Stack Overflow

Python - What is the most efficient way to generate padding?

5 Answers 5

2 Comments

2 Comments

Comments

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

2 Comments

2 Comments

Comments

Comments

Comments

Related