Problem overriding WM_NCHITTEST

[Uwe Kotyczka]

In a pure Win32 application I have a dialog with style WS_THICKFRAME.
I want that the user is only able to resize the right border.

Paul DiLascia suggests WM_NCHITTEST in an article which is
even the MSDN which shipped with VC++ 6.0. He writes,
WM_NCHITTEST ensures that the right mouse cursors are shown
when you drag the window border. However his article deals
with MFC while my application is pure Win32.

So my idea was to override WM_NCHITTEST:

BOOL CALLBACK MyDlgProc(HWND hDlg, UINT uMessage, wParam wParam,
lParam lParam)
{
switch (uMessage)
{
...
case WM_NCHITTEST:
LRESULT ret = DefWindowProc(hDlg, uMessage, wParam,
lParam);
switch (ret)
{
case HTTOPRIGHT:
case HTRIGHT:
case HTBOTTOMRIGHT:
return HTRIGHT;

case HTTOP:
case HTTOPLEFT:
case HTLEFT:
case HTBOTTOMLEFT:
case HTBOTTOM:
return HTBORDER;

default:
//return FALSE;
return ret;
}
}
return FALSE;
}

HWND hDlg = CreateDialog(hInstance, MAKEINTRESOURCE(IDD_DIALOG_TEST),
0, (DLGPROC) MyDlgProc);
ShowWindow(hDlg, SW_SHOW);

To track down the reason of the strange behaviour I wrote
two minimal applications showing a modeless dialog, one
in MFC and one in pure Win32. IN both apps I overrode
WM_NCHITTEST. You can download the projects at
http://home.arcor.de/kotyczka/download/Test.zip.
(It's only 9 kB.)

The MFC app works as expected, the Win32 app does not.
Using Spy++ I can see that in the MFC app the dialog
windows handles WM_NCHITTEST as expected. In the Win32
app I see in the debugger, that WM_NCHITTEST is caught
with the expected return values from DefWindowsProc.
Watching WM_NCHITTEST in Spy++ however shows that
whenever I return anything _else_ than HTNOWHERE (zero),
then Spy++ sees HTNOWHERE. And I return HTNOWHERE,
then Spy++ sees the expected values.

So I guess that I have a major problem understanding
how to override WM_NCHITTEST properly in pure Win32.
Could you take a look at my sample to find out what
I am doing wrong?

BTW, using modal instead of modeless dialogs does
not make a difference for the WM_NCHITTEST behaviour.

What am I missing?

[David Lowndes]

Post by Uwe Kotyczka
...
...
return HTRIGHT;

The issue is probably because the "return" result for a dialog has to
be conveyed via the DWL_MSGRESULT value rather than the return value
of the dialog proc.

See the Return value discussion here:
http://msdn.microsoft.com/en-gb/library/windows/desktop/ms645469(v=vs.85).aspx

Dave

[Uwe Kotyczka]

Post by David Lowndes

Post by Uwe Kotyczka
...
...
return HTRIGHT;

The issue is probably because the "return" result for a dialog has to
be conveyed via the DWL_MSGRESULT value rather than the return value
of the dialog proc.
See the Return value discussion here:http://msdn.microsoft.com/en-gb/library/windows/desktop/ms645469(v=vs...

Thanks David, that was exactly what I was missing.

case HTTOPRIGHT:
case HTRIGHT:
case HTBOTTOMRIGHT:
SetWindowLong(hDlg, DWL_MSGRESULT, HTRIGHT);
return TRUE;

works perfectly as expected.

It's really great getting helpful answers so quickly
in a group that offically does not exist any more
for more than two years.