When it comes to boost's pointers specifically, I think that they should be avoided at all costs. They come at a cost that is much larger than anyone would initially expect. They provide an interface that allows you to skip vital and important parts of your memory and resourcement management.
When it comes to any software development I think that it is important to think about your data. It is very important how your data is represented in memory. The reason for this is that CPU-speed has been increasing at a much greater rate than memory-access time. This often makes the memory-caches the main bottleneck of most modern computer games. By having your data aligned linearly in memory according to access order is much friendlier to the cache. This kind of solutions often lead to cleaner designs, simpler code and definetly code that is more easy to debug. Smart pointers easily lead to frequent dynamic memory allocations of resources, this causes them to be scattered all over the memory.
This is not a premature optimization, it's a healthy decision that can and should be taken as early as possible. It's a question of architectural understanding of the hardware that your software will run on and it is important.