0

I have an embedded Linux device (not a Raspberry Pi) which has two partitions on an eMMC drive. I have used a vendor-provided tool to copy these two partitions onto a USB drive whose size roughly equals the size of the eMMC drive.

What I did next was mount the USB drive onto a Linux computer and use tools such as e2fsck and resize2fs -M to resize the second filesystem to a minimum viable size, and finally used gdisk to resize the partition itself to match.

In all calculations I've been careful to ensure that the partition does not get smaller than the filesystem, even to a point where I've made the partition slightly larger, and then used resize2fs to enlarge the filesystem to match it.

After these operations I have a 16 GB USB drive with a 118 MiB boot partition and a 3.1 GiB root filesystem partition. I then used dd to read data directly from the USB drive, utilizing bs= and count= parameters to limit the amount of data read to only extend up to where the second partition ends plus an extra sector to accommodate the "zero sector".

The result is that I now have an image file in which the two partitions have exactly the size of their corresponding sources but when I use the "File" -> "Properties" option of the 7-Zip tool it claims that the "Physical size" (circled in red) is larger than these partitions, and larger than what I instructed dd to copy from the source.

Where does this "Physical size" value come from? Is it part of the partition table? Or does dd read it from the disk somehow?

Is there any way for me to modify it so as to get rid of the "Unexpected end of data" error?

Picture from 7-zip

Improve this question

2 Answers 2

Reset to default
3

Where does this "Physical size" value come from? Is it part of the partition table?

Yes.

See GUID Partition Table - Wikipedia

Is there any way for me to modify it so as to get rid of the "Unexpected end of data" error?

Yes.

It seems the standard Linux tools know how to fix up a truncated GPT disk.

Creating test-short.img with the same problem

$ dd if=/dev/zero bs=1M count=10 of=test.img 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0161485 s, 649 MB/s $ gdisk test.img GPT fdisk (gdisk) version 1.0.8 Partition table scan: MBR: not present BSD: not present APM: not present GPT: not present Creating new GPT entries in memory. Command (? for help): n Partition number (1-128, default 1): First sector (34-20446, default = 2048) or {+-}size{KMGTP}: Last sector (2048-20446, default = 20446) or {+-}size{KMGTP}: 18430 Current type is 8300 (Linux filesystem) Hex code or GUID (L to show codes, Enter = 8300): Changed type of partition to 'Linux filesystem' Command (? for help): w Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING PARTITIONS!! Do you want to proceed? (Y/N): y OK; writing new GUID partition table (GPT) to test.img. Warning: The kernel is still using the old partition table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) The operation has completed successfully. $ dd if=test.img bs=1M count=9 of=test-short.img 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.00650826 s, 1.5 GB/s

How to fix the problem

IIUC, you left one empty sector at the end of the image file, which we can use for the "secondary GPT header" (trailer?). Assuming the table has the usual 128 partition entries, you need to add another 32 sectors for those, and then you can fix the problem.

$ dd if=/dev/zero bs=512 count=32 >> test-short.img $ gdisk test-short.img GPT fdisk (gdisk) version 1.0.8 Warning! Disk size is smaller than the main header indicates! Loading secondary header from the last sector of the disk! You should use 'v' to verify disk integrity, and perhaps options on the experts' menu to repair the disk. Caution: invalid backup GPT header, but valid main header; regenerating backup header from main header. Warning! Error 25 reading partition table for CRC check! Warning! One or more CRCs don't match. You should repair the disk! Main header: OK Backup header: ERROR Main partition table: OK Backup partition table: ERROR Partition table scan: MBR: protective BSD: not present APM: not present GPT: damaged **************************************************************************** Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk verification and recovery are STRONGLY recommended. **************************************************************************** Command (? for help): p Disk test-short.img: 18464 sectors, 9.0 MiB Sector size (logical): 512 bytes Disk identifier (GUID): 01BD61C8-5DC3-42CA-824D-B5C1B3B09A77 Partition table holds up to 128 entries Main partition table begins at sector 2 and ends at sector 33 First usable sector is 34, last usable sector is 20446 Partitions will be aligned on 2048-sector boundaries Total free space is 4030 sectors (2.0 MiB) Number Start (sector) End (sector) Size Code Name 1 2048 18430 8.0 MiB 8300 Linux filesystem Command (? for help): x Expert command (? for help): e Relocating backup data structures to the end of the disk Expert command (? for help): w Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING PARTITIONS!! Do you want to proceed? (Y/N): y OK; writing new GUID partition table (GPT) to test-short.img. Warning: The kernel is still using the old partition table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) The operation has completed successfully.

To confirm that test-short.img was fixed

$ gdisk test-short.img GPT fdisk (gdisk) version 1.0.8 Partition table scan: MBR: protective BSD: not present APM: not present GPT: present Found valid GPT with protective MBR; using GPT. Command (? for help): p Disk test-short.img: 18464 sectors, 9.0 MiB Sector size (logical): 512 bytes Disk identifier (GUID): 01BD61C8-5DC3-42CA-824D-B5C1B3B09A77 Partition table holds up to 128 entries Main partition table begins at sector 2 and ends at sector 33 First usable sector is 34, last usable sector is 18430 Partitions will be aligned on 2048-sector boundaries Total free space is 2014 sectors (1007.0 KiB) Number Start (sector) End (sector) Size Code Name 1 2048 18430 8.0 MiB 8300 Linux filesystem Command (? for help): q
Improve this answer
5
  • This does not resolve the issue; after performing the operation and creating a new image file the "Physical size" is still showing a larger value. Commented Sep 23, 2024 at 15:49
  • Weird. I do see a second issue. "limit the amount of data read to only extend up to where the second partition ends plus an extra sector to accommodate the "zero sector"". I don't know exactly what you mean by "zero sector", but GPT requires an absolute minimum of two sectors at the end. I think the "w" command should have failed for you with something like "Warning! Secondary partition table overlaps the last partition by 32 blocks! Try reducing the partition table size by 128 entries. (Use the 's' item on the experts' menu.) Aborting write of new partition table." Commented Sep 23, 2024 at 19:41
  • That "zero sector" was just a term I picked up elsewhere when I was trying to research the issue. If the GPT primary header is at the start of the disk, and the secondary header is at the end then that poses a problem for me. I am not reading the whole disk but just a very limited part of it. So technically the primary header says that the secondary header is in a location which will not be part of the final image. This happens because the gdisk commands are executed before I create the image. Commented Sep 24, 2024 at 8:13
  • and hence my answer is to run gdisk to fix the out-of-spec image file, after you created the image file by reading part of the disk Commented Sep 24, 2024 at 9:46
  • 1
    Alright so, looks like your answer was correct after all, I just did not understand what I was supposed to do first. The solution was to use dd to create the image first, making sure there was ~40 sectors extra in the "count" parameter to extend beyond to last partition, and then use gdisk and its expert menu to fix the partition table on the image file. Commented Sep 24, 2024 at 19:08
-1

When a file system is created, 5% is reserved for writes done by root. This was originally intended to allow recovery by the system even after runway user processes.

So the "minimum viable size" is non-viable for non-root processes.

Alternatively, create a 4GB partition, do your copy, then reduce the size to a suitable minimum.

Improve this answer
1
  • The partitions which appear to the USB drive are created by the vendor-provided tool during the initial copy process. I have no control on how large they will be; I can only reduce them "after the fact". Commented Sep 23, 2024 at 15:51

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.