6

I'm trying to allocate a 64 KiB buffer in Watcom C 16-bit DOS. I'm using the "compact" memory model which defines the code segment to be limited to 64 KiB and addressed by near pointers (just offset), but the data segment to be addressed through far pointers (segment:offset).

From the documentation I also get the impression that using malloc in the compact memory model is simply an alias for _fmalloc.

However, when I try to allocate 64 KiB, or as close as I can get (1024 × 64 − 1, the maximum representable with a 16-bit size_t) the malloc fails. But when I allocate 64 KiB minus 32 bytes (i.e. 1024 × 64 − 32), the malloc succeeds.

How can I allocate 64 KiB (1024 × 64 − 1, i.e. 65535/FFFFh bytes)? And why does my malloc succeed when I subtract 32 bytes?

Improve this question
5
  • good point, so malloc(0x10000) == malloc(0x0) on this system. Commented Nov 5, 2018 at 9:28
  • Are you sure it's 32 and not 26? I'm looking at the OpenWatcom code, getting 26 by arithmetic, and failing to find my error. Commented Nov 6, 2018 at 22:42
  • @Tommy Yes, definitely 32 bytes. Are you in "compact" memory model? I'm not on OpenWatcom, but the original Watcom v11.0 (I will try to upgrade to 11.0c just in case). Commented Nov 7, 2018 at 8:02
  • @Tommy Just tested with Watcom 11.0c, definitely have to subtract 32 bytes. Commented Nov 7, 2018 at 8:09
  • Well I got my answer working forwards from github.com/open-watcom/open-watcom-v2/blob/… but obviously miscalculated sizeof(heapblk). If I can find my error I'll turn this into an answer. Commented Nov 7, 2018 at 14:29

3 Answers 3

Reset to default
14

Typically memory allocators need some extra data in addition to the requested allocation size. This is for maintaining a list of allocated blocks on the heap, it's usually a linked-list node along with a couple of other bits of info like if the block is currently allocated or free. This tiny structure usually goes behind the pointer returned to you by malloc and is typically rounded up to some power-of-2 size to ensure pointers returned by malloc are aligned.

Because of this, you cannot allocate exactly the size of your heap because we need additional space for that block's link structure.

For example, eight malloc(8192) calls would also fail on the 8th call because each allocation would occupy, in your case, 8 KiB + 32 bytes.

Improve this answer
5
  • 7
    I'm not entirely convinced that eight 8K allocations would fail. There's no real reason why these would have to come from a single 64K segment. The arena data could be used not just to link together blocks in this segment but also link together whole other segments (32 bytes is pretty chunky for a block detail section when all you really need is a size, forward and back pointers and possibly a sentinel). Commented Nov 5, 2018 at 10:17
  • It depends on the implementation, I was deducing it was a 32 byte overhead from the question above, which seems high to me, but it could be the OP didn't exhaustively test every size until it starts to fail. I suppose you could test it by measuring the address gap for 2 consecutive malloc calls on a pristine heap, or better yet dump the memory around your allocate to detect the presence of a embedded allocation link structure. I have no information on the actual malloc() strategy used here other than what the original question put forward. Commented Nov 5, 2018 at 10:20
  • 1
    Thanks Paul, for your answer. I'm sure you are correct but I'm looking for actual documentation from/by Watcom that would confirm this. Commented Nov 6, 2018 at 8:01
  • You could inspect the memory around the pointer that malloc returns: e.g. void *p1 = malloc(256); memset(p1, 0xcc, 256); void *p2 = malloc(256); memset(p2, 0xdd, 256); --> Stop your debugger and dump memory around 'p2' -- You should be able to see if there is a gap between the 0xcc's and 0xdd's that the malloc subroutine inserted for housekeeping data. Commented Nov 6, 2018 at 8:33
  • I posted this comment under another answer but here you can see where the 32 bytes comes from. Commented Dec 2, 2022 at 13:27
7

I have not tried it but the following trick might work:

You can use halloc(0x1000, 0x10) to allocate a "huge" memory block of 0x1000 × 0x10 bytes (= 64 KiB) size. According to the manual this block will be 0x10-byte aligned if the second argument is 0x10.

halloc() can allocate more than 64K but returns a huge memory pointer, not a far memory pointer (see below).

In "Real Mode" ("normal" MS-DOS) the linear address is calculated as 0x10*segment+offset so if the offset is a multiple of 0x10 (if the address is aligned to 0x10 bytes) you can calculate a far pointer that points to the same memory using the following method:

You take the offset, divide it by 0x10 and add the result to the segment. Then you set the segment to 0.

Example:

union { struct { unsigned short off; unsigned short seg; } s; void far *ptr; } u; u.ptr = (void far *)huge_pointer; // Example: 0x1234:FED0 u.s.seg += u.s.off >> 4; u.s.off = 0; far_ptr = u.ptr; // should be 0x2221:0000

Both 0x1234:FED0 and 0x2221:0000 point to the same memory.

However there are two differences between the pointers:

  • hfree() and similar memory management functions must be called with the "original" pointer returned by halloc(), not with the "calculated" one

  • The difference between huge and far memory pointers is that far pointers do not allow crossing segment boundaries: A far pointer to X:Y can only access (0x10000 − Y) bytes of memory. So a far pointer to 0x1234:FED0 can only access the first 304 bytes of the memory block (0x10000 − 0xFED0 = 304) while a far pointer to 0x2221:0000 can access the first 64 KiB (note that halloc() can allocate more than 64K).

Improve this answer
5
  • 1
    This looks like a good solution to me. The code for halloc is here and it looks as though it always allocates aligned to a paragraph boundary, whatever the size is, anyway. This is definitely a better solution than the one I'd have suggested (making sure not all DOS memory is allocated to the heap -- I'm not familiar with Watcom's tools, but there should be an option for how much heap to allocate somewhere -- and then using INT 21h/AH=48h to perform the allocation). Commented Nov 6, 2018 at 21:13
  • Another thing to note from the halloc code is that the returned pointer always has its offset set to zero, which should simplify the translation to a far pointer somewhat... Commented Nov 6, 2018 at 21:15
  • Another option is to just call int 21h function ah = 48h Allocate Memory directly yourself. This appears to be (from inspecting the code and stepping through it) what Watcom's library ultimately calls anyway to allocate more segments to the heap. Commented Dec 2, 2022 at 13:32
  • Since when does halloc return a pointer with offset != 0? Since you know your size is 64k you just cast it to a far pointer. Commented Feb 11, 2024 at 4:09
  • @Joshua The Watcom documentation does not say that the offset in the pointer returned by halloc() is guaranteed to be 0. For this reason, you have to assume that at least some versions of the Watcom library may return non-0 offsets. Commented Feb 11, 2024 at 8:14
3

A community wiki answer because it includes a lot of useful evidence, but I seem to flub the numbers somewhere:

Per OpenWatcom, the definition of _fmalloc is this. Notably:

if( amt == 0 || amt > - ( sizeof( heapblk ) + TAG_SIZE * 2 ) ) { return( NULL ); }

Allowing for integer rounding and given that size_t is 16-bit, that asserts that if the block you asked for is larger than 65536 minus sizeof( heapblk ) + TAG_SIZE * 2 then the allocation will fail.

heapblk is a struct:

struct heapblk { tag len; /* size of heap (0 = 64K) */ heapptr prev; /* segment selector/offset for previous heap */ heapptr next; /* segment selector/offset for next heap */ freeptr rover; /* roving pointer into free list */ unsigned int b4rover; /* largest block before rover */ unsigned int largest_blk; /* largest block in the heap */ unsigned int numalloc; /* number of allocated blocks in heap */ unsigned int numfree; /* number of free blocks in the heap */ freelist freehead; /* listhead of free blocks in heap */ #if defined( __WARP__ ) unsigned int used_obj_any :1; /* allocated with OBJ_ANY - block may be in high memory */ #endif };

And from line 50 of the same file, TAG_SIZE is just sizeof(tag).

On line 106 tag turns out to be a typedef for unsigned int so it's also 16 bit.

heapptr is a union defined on line 115, of a __segment and a heapblk_nptr (a.k.a. a _WCNEAR). So it's two bytes as nptr means near pointer, and a __segment means 16 bit.

freeptr is a union defined on line 120, of a freelist_nptr and an unsigned int. So also two bytes.

freelist is a struct defined on line 125 consisting of a tag, and two freeptrs. So six bytes.

So a heapblk is 22 bytes of meaningful storage:

  • 8 bytes for the four unsigned ints;
  • 2 bytes for the tag (because it's another unsigned int);
  • 6 more for the freelist;
  • 2 for the freeptr; and
  • 4 in total for the heapptrs.

That plus another 4 for the 2*TAG_SIZE is 26.

Therefore as posited elsewhere, you cannot allocate a full 64kb because some storage is reserved for heap management; however I seem to total that to 26 bytes, not 32.

Improve this answer
4
  • I can’t see how in this case, but alignment issues sometimes explain surprising struct sizes... Commented Nov 7, 2018 at 18:30
  • Yeah, I kind of assumed that since everything is two bytes in size and it's a 16-bit machine, everything was naturally aligned. But maybe I'm wrong that everything is two bytes in size, or there's an additional alignment factor that I'm unaware of? Commented Nov 7, 2018 at 18:42
  • I can’t think of anything. But wouldn’t sizeof( heapblk ) refer to the heapblk typedef rather than struct heapblk? (Yeah that doesn’t help solve the mystery at all...) Commented Nov 7, 2018 at 18:50
  • 1
    The reason seems to be that segments (where the memory is ultimately allocated out of) have a heapblk, a freelist, and 2 * TAG_SIZE in them. They are allocated with __AllocSeg in allocseg.c and you can see that one of the first things that function does is check against: ( sizeof( heapblk ) + sizeof( freelist ) + TAG_SIZE * 2 ) ) which is 32 bytes. Commented Dec 2, 2022 at 13:00

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.