I have some experiences on particle system simulation (namely DEM - Discrete element method), in which an individual particle with realistic shape (convex and non-convex) is approximated by gluing 3D spheres together (non-overlapping or overlapping), and act like rigid body. The collision detection is performed by the simplest sphere-sphere contact.

Nevertheless, for particle shape representing by polyhedrons, almost all the collision detection algorithms for broad and narrow phases are based on convex polyhedron (e.g. GJK). For non-convex/concave shapes, convex decomposition is required to divide the concave object into many convex components (e.g. V-HACD).

At this point, I really want to know the details on:

1. how to store the concave objects into memory (like BSP tree?);

2. collision detection and contact forces calculation between convex-concave, concave-concave objects.

I have searched quite a few relevant books like "Real-time collision detection" and "3D game engine design", etc, unfortunately I haven't find any details on this topic yet. Could some veteran game developers give some advice or insight on how to implement collision between concave objects?

Cheers, David

Update 1:

Seems the physics engines likes Bullet3 and reactPhysics3D are able to simulate collision of non-convex objects. Can someone give some rough idea on how to do this? before I dig into the source codes, since i have only some academic experiences on molecular dynamics / DEM simulations mainly based on 3D spheres.