Personally, I would pick Allegro 5. It's reasonably stable and has a pretty decent API. Now, like SFML or SDL, it is very low-level, so you'll still need other systems to handle things like physics and so forth.for the following reasons:
It has a CMake-based build system. This makes it much easier than most build systems for working on multiple platforms. This seems like a minor thing, but it's pretty important overall. It makes building the library much simpler.
SFML is still in significant flux. 2.0 is a pretty substantial change. Whereas Allegro has already been through a significant change with version 5, and it's now all about stability.
It covers modularity in a different way from SFML. With SFML, you ask for specific components, but you get all of them. If you ask for the graphics component, you get all of the graphics component. With Allegro 5, you always get the graphics component. But if you don't want to use Allegro's bitmap loaders, you don't have to; you can use your own.
It has very good documentation. SFML uses Doxygen-style documentation, but even that is pretty Spartan. Whereas Allegro 5 is more comprehensive, with functions grouped together based on specific systems that they cover. It's not perfect, but it is better overall.
They're smart enough to not try to do threading. With C++11/Boost.Threads either widely available or just around the corner, using another thread wrapper can be more than just problematic: it can be dangerous to getting your code working properly.