Assuming you are talking about shared_ptr here...
Is this simply because of using the reference of the smart pointer after its been destroyed?
This is a good answer. You may not know absolutely the lifetime of the pointer your reference refers too.
To get around this, you'd want to look into boost::weak_ptr. It doesn't participate in reference counting. When you need to use it, it gives you a shared_ptr which goes away once your done with it. It will also let you know when the refered to pointer has been collected.
From the weak_ptr documentation
The weak_ptr class template stores a "weak reference" to an object that's already managed by a shared_ptr. To access the object, a weak_ptr can be converted to a shared_ptr using the shared_ptr constructor or the member function lock. When the last shared_ptr to the object goes away and the object is deleted, the attempt to obtain a shared_ptr from the weak_ptr instances that refer to the deleted object will fail: the constructor will throw an exception of type boost::bad_weak_ptr, and weak_ptr::lock will return an empty shared_ptr.
Note the method expired() will also tell you if your ptr is still around.