Got bad news and good news. The bad news is it's simply not implemented.
The good news is: as Qt is open source, you can crack it open and take a look to know that. And if there's a bug you can submit a fix. Here's the deal, in the generic code for QWidget::setWindowFlags in qwidget.cpp:9144 you have this:
void QWidget::setWindowFlags(Qt::WindowFlags flags) { if (data->window_flags == flags) return; Q_D(QWidget); if ((data->window_flags | flags) & Qt::Window) { // the old type was a window and/or the new type is a window QPoint oldPos = pos(); bool visible = isVisible(); setParent(parentWidget(), flags); // if both types are windows or neither of them are, we restore // the old position if (!((data->window_flags ^ flags) & Qt::Window) && (visible || testAttribute(Qt::WA_Moved))) { move(oldPos); } // for backward-compatibility we change Qt::WA_QuitOnClose attribute value only when the window was recreated. d->adjustQuitOnCloseAttribute(); } else { data->window_flags = flags; } }
So essentially it just sets window_flags. The mac behavior of QWidget is in qwidget_mac.mm.
And you will find no reference to Qt::WindowStaysOnBottomHint in that file. (You'll find Qt::WindowStaysOnTopHint though...)
I'll stop at saying "not possible unless you either patch Qt, or otherwise go beneath Qt".
Patching qwidget_mac.mm is left as an exercise to the reader. :-)