How do I use the Rust memory allocator for a C library that can be provided an allocator?

Question

I'm writing Rust bindings to a C library which has the option to use a third-party memory allocator. Its interface looks like this:

struct allocator { void*(*alloc)(void *old, uint); void(*free)(void*); };

The corresponding Rust struct is, I guess, the following:

#[repr(C)] #[derive(Copy, Clone, Debug, PartialEq)] pub struct Allocator { alloc: Option<extern "C" fn(*mut c_void, c_uint) -> *mut c_void>, free: Option<extern "C" fn(*mut c_void)>, }

How can I implement these two extern functions that should mimic the allocator? I did not find anything really looking like the allocator API in Rust (I understand why however), so I'm curious if it is possible.

reddit.com/r/rust/comments/2eqdg2/allocate_a_vec_on_cs_heap There are some thoughts on a similar topic, although the allocations lifetime management may be tricky. — snuk182
– snuk182, Commented Aug 31, 2016 at 13:21
As you point out, that topic is about making Rust use the same allocator as C. That's possible with custom allocators. — Shepmaster
– Shepmaster, Commented Aug 31, 2016 at 13:53

Shepmaster · Accepted Answer · 2016-08-31 16:58:03Z

It's not as easy as you might like.

The allocation methods are exposed in the heap module of the alloc crate.

Creating some wrapper methods and populating the struct is straight-forward, but we quickly run into an issue:

#![feature(heap_api)] extern crate libc; extern crate alloc; use libc::{c_void, c_uint}; use alloc::heap; #[repr(C)] #[derive(Copy, Clone, Debug, PartialEq)] pub struct Allocator { alloc: Option<extern "C" fn(*mut c_void, c_uint) -> *mut c_void>, free: Option<extern "C" fn(*mut c_void)>, } extern "C" fn alloc_ext(old: *mut c_void, size: c_uint) -> *mut c_void { if old.is_null() { heap::allocate(size as usize, align) as *mut c_void } else { heap::reallocate(old as *mut u8, old_size, size as usize, align) as *mut c_void } } extern "C" fn free_ext(old: *mut c_void) { heap::deallocate(old as *mut u8, old_size, align); } fn main() { Allocator { alloc: Some(alloc_ext), free: Some(free_ext), }; }

The Rust allocator expects to be told the size of any previous allocation as well as the desired alignment. The API you are matching doesn't have any way of passing that along.

Alignment should (I'm not an expert) be OK to hardcode at some value, say 16 bytes. The size is trickier. You will likely need to steal some old C tricks and allocate a little bit extra space to store the size in. You can then store the size and return a pointer just past that.

A completely untested example:

#![feature(alloc, heap_api)] extern crate libc; extern crate alloc; use libc::{c_void, c_uint}; use alloc::heap; use std::{mem, ptr}; #[repr(C)] #[derive(Copy, Clone, Debug, PartialEq)] pub struct Allocator { alloc: Option<extern "C" fn(*mut c_void, c_uint) -> *mut c_void>, free: Option<extern "C" fn(*mut c_void)>, } const ALIGNMENT: usize = 16; extern "C" fn alloc_ext(old: *mut c_void, size: c_uint) -> *mut c_void { unsafe { // Should check for integer overflow let size_size = mem::size_of::<usize>(); let size = size as usize + size_size; let memory = if old.is_null() { heap::allocate(size, ALIGNMENT) } else { let old = old as *mut u8; let old = old.offset(-(size_size as isize)); let old_size = *(old as *const usize); heap::reallocate(old, old_size, size, ALIGNMENT) }; *(memory as *mut usize) = size; memory.offset(size_size as isize) as *mut c_void } } extern "C" fn free_ext(old: *mut c_void) { if old.is_null() { return } unsafe { let size_size = mem::size_of::<usize>(); let old = old as *mut u8; let old = old.offset(-(size_size as isize)); let old_size = *(old as *const usize); heap::deallocate(old as *mut u8, old_size, ALIGNMENT); } } fn main() { Allocator { alloc: Some(alloc_ext), free: Some(free_ext), }; let pointer = alloc_ext(ptr::null_mut(), 54); let pointer = alloc_ext(pointer, 105); free_ext(pointer); }

Isn't [... using Vec as an allocator ...] the more high-level solution?

That's certainly possible, but I'm not completely sure how it would work with reallocation. You'd also have to keep track of the size and capacity of the Vec in order to reconstitute it to reallocate / drop it.

You could in principle also store the allocation sizes in a HashMap or similar. I'm not sure you'd gain anything over overallocating and storing in the allocation, though (except for a better chance of detecting a bad free from the C side).
@ChrisEmerson good point! However, when I've tried similar things in the past, I've experienced abnormally poor performance. Without knowing exactly why, I think it has to do with poor cache locality. Multiple threads is also more painful with a shared collection.
UPD: Here is where I faced the C's unsafety. The called library code was using my free_ext implementation with a null pointer, so I had to add its check explicitly: if old.is_null() { println!("no dealloc for empty pointer"); return; }
Btw, Vec as allocator approach also works. May be useful to the ones who don't want or cannot use unstable API.
Also, as new allocator API has been introduced, the answer is also modified a bit.

Collectives™ on Stack Overflow

How do I use the Rust memory allocator for a C library that can be provided an allocator?

1 Answer 1

6 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Linked

Related