For checking environment variables I use if ("$ENV{QT_LIBS_DIR}" STREQUAL "") to see if the environment variable is defined. So your example becomes:
if ("$ENV{QT_LIBS_DIR}" STREQUAL "") message(FATAL "You need to have system-wide variable with path to Qt libs") set(QT_CMAKE_MODULES_FOUND FALSE) elseif() set(QT_CMAKE_MODULES "$ENV{QT_LIBS_DIR}/lib/cmake" CACHE PATH "CMake's modules which provided by Qt itself for Qt bindings") set(QT_CMAKE_MODULES_FOUND TRUE) endif()
But for what it is worth, I believe that find_package for qt gets the path for qt according to the qmake executable that it finds first in your path. This way you do not really need to use an environment variable. This applies to qt4; see How specify Qt version/location? for details.
For QT5, the Qt5/CMake manual says "In order for find_package to be successful, Qt 5 must be found below the CMAKE_PREFIX_PATH, or the Qt5_DIR must be set in the CMake cache to the location of the Qt5WidgetsConfig.cmake file. The easiest way to use CMake is to set the CMAKE_PREFIX_PATH environment variable to the install prefix of Qt 5."