How to prevent MFC dialog closing on Enter and Escape keys

In this article I explain how to prevent an MFC dialog from handling the Enter and Escape keys and not pass it on. With all required details for you to completely understand why does it work the way it does. There are several possible solutions, so you can choose the most suitable one.

The problem is the following: you have a dialog, no matter - top level window or just a child container; you press Escape or Enter keys - the dialog disappears, even if there are no OK and Cancel buttons on the dialog. It's very funny seeing a child dialog drops out it's parent.

Let's see why this happens:

OrderXMLReaderShell.exe!CFilterEditorDlgBase::OnCancel()  Line 61 C++
  mfc100ud.dll!_AfxDispatchCmdMsg(CCmdTarget * pTarget, unsigned int nID, int nCode, void (void)* pfn, void * pExtra, unsigned int nSig, AFX_CMDHANDLERINFO * pHandlerInfo)  Line 82 C++
  mfc100ud.dll!CCmdTarget::OnCmdMsg(unsigned int nID, int nCode, void * pExtra, AFX_CMDHANDLERINFO * pHandlerInfo)  Line 381 + 0x27 bytes C++
  mfc100ud.dll!CDialog::OnCmdMsg(unsigned int nID, int nCode, void * pExtra, AFX_CMDHANDLERINFO * pHandlerInfo)  Line 87 + 0x18 bytes C++
  mfc100ud.dll!CWnd::OnCommand(unsigned int wParam, long lParam)  Line 2729 C++
  mfc100ud.dll!CWnd::OnWndMsg(unsigned int message, unsigned int wParam, long lParam, long * pResult)  Line 2101 + 0x1e bytes C++
  mfc100ud.dll!CWnd::WindowProc(unsigned int message, unsigned int wParam, long lParam)  Line 2087 + 0x20 bytes C++
  mfc100ud.dll!AfxCallWndProc(CWnd * pWnd, HWND__ * hWnd, unsigned int nMsg, unsigned int wParam, long lParam)  Line 257 + 0x1c bytes C++
  mfc100ud.dll!AfxWndProc(HWND__ * hWnd, unsigned int nMsg, unsigned int wParam, long lParam)  Line 420 C++
  mfc100ud.dll!AfxWndProcBase(HWND__ * hWnd, unsigned int nMsg, unsigned int wParam, long lParam)  Line 420 + 0x15 bytes C++
  user32.dll!_InternalCallWinProc@20()  + 0x23 bytes 
  user32.dll!_UserCallWinProcCheckWow@32()  + 0xb3 bytes 
  user32.dll!_SendMessageWorker@20()  + 0xee bytes 
  user32.dll!_SendMessageW@16()  + 0x49 bytes 
  user32.dll!_IsDialogMessageW@8()  + 0xef46 bytes 
> mfc100ud.dll!CWnd::IsDialogMessageW(tagMSG * lpMsg)  Line 198 C++
  mfc100ud.dll!CWnd::PreTranslateInput(tagMSG * lpMsg)  Line 4713 C++
  mfc100ud.dll!CDialog::PreTranslateMessage(tagMSG * pMsg)  Line 82 C++
  OrderXMLReaderShell.exe!CFindDlg::PreTranslateMessage(tagMSG * pMsg)  Line 56 C++
  mfc100ud.dll!CWnd::WalkPreTranslateTree(HWND__ * hWndStop, tagMSG * pMsg)  Line 3311 + 0x14 bytes C++
  mfc100ud.dll!AfxInternalPreTranslateMessage(tagMSG * pMsg)  Line 233 + 0x12 bytes C++
  mfc100ud.dll!CWinThread::PreTranslateMessage(tagMSG * pMsg)  Line 777 + 0x9 bytes C++
  mfc100ud.dll!AfxPreTranslateMessage(tagMSG * pMsg)  Line 252 + 0x11 bytes C++
  mfc100ud.dll!AfxInternalPumpMessage()  Line 178 + 0x18 bytes C++
  mfc100ud.dll!CWinThread::PumpMessage()  Line 900 C++
  mfc100ud.dll!CWinThread::Run()  Line 629 + 0xd bytes C++
  mfc100ud.dll!CWinApp::Run()  Line 832 C++
  mfc100ud.dll!AfxWinMain(HINSTANCE__ * hInstance, HINSTANCE__ * hPrevInstance, wchar_t * lpCmdLine, int nCmdShow)  Line 47 + 0xd bytes C++
  OrderXMLReaderShell.exe!wWinMain(HINSTANCE__ * hInstance, HINSTANCE__ * hPrevInstance, wchar_t * lpCmdLine, int nCmdShow)  Line 26 C++
  OrderXMLReaderShell.exe!__tmainCRTStartup()  Line 547 + 0x2c bytes C
  OrderXMLReaderShell.exe!wWinMainCRTStartup()  Line 371 C
  kernel32.dll!@BaseThreadInitThunk@12()  + 0x12 bytes 
  ntdll.dll!___RtlUserThreadStart@8()  + 0x27 bytes 
  ntdll.dll!__RtlUserThreadStart@8()  + 0x1b bytes 

The execution flow got into CDialog::OnCancel virtual method. Here is the default MFC's implementation:

void CDialog::OnCancel()
{
 EndDialog(IDCANCEL);
}

It simply ends the dialog, thus destroying the window.

Lets find out how do we get there. The control flow is the following:

1. Escape key is down
2. WM_KEYDOWN message is queued into message queue
3. AfxInternalPumpMessage gets the message from the queue and invokes AfxPreTranslateMessage
4. CWnd::IsDialogMessage is reached
5. IsDialogMessage WinAPI function is called by CWnd::IsDialogMessage; this function is the core of dialog management - it does things like keyboard navigation.
6. There is a user32.dll!_SendMessageW@16() call from IsDialogMessageW
7. Which brings us to CWnd::OnCommand with wParam equals to IDCANCEL
8. End finally to CDialog::OnCancel

IsDialogMessageW examines the message and invokes SendMessage(hwndDlg, WM_COMMAND, MAKELONG(IDCANCEL, BN_CLICKED), ...);.

Short summary - MFC calls IsDialogMessage API function, which in turns emulates IDCANCEL button click, which is being handled by MFC eventually destroying the dialog.

It is not obvious why IsDialogMessageW emulates IDOK and IDCANCEL button clicks. I believe this is done to make developer's life easier. Imagine you have no control over the message loop, can't do any message pre translation. You'll have to handle Enter and Escape keys for every control on the dialog. That's pain.

What can we do to solve the problem? You can't change OS behavior for sure, but can do the following:

1. Override CDialog::OnOK/OnCancel and do nothing

   The following applies:

   • Dialog is not destroyed on Enter or Escape
   • Focused inline controls don't receive WM_KEYDOWN for VK_ENTER and VK_ESCAPE codes
   • OK/Cancel button clicks have no effect
2. Override CDialog::PreTranslateMessage and don't call the base if pMsg->message == WM_KEYDOWN && (pMsg->wParam == VK_ESCAPE || pMsg->wParam == VK_RETURN)

   If CDialog::PreTranslateMessage returns TRUE the following applies:

   • Dialog is not destroyed on Enter or Escape
   • Focused inline controls don't receive WM_KEYDOWN for VK_RETURN and VK_ESCAPE codes, unless you do dispatch them before CDialog::PreTranslateMessage returns
   • OK/Cancel button clicks have desired effect

   If CDialog::PreTranslateMessage returns FALSE the following applies:

   • MFC calls parent's PreTranslateMessage (which brings us back if the parent is a CDialog instance)
3. Override CWinThread::PreTranslateMessage, examine the message and don't call the base

   The following applies:

   • CWnd::WalkPreTranslateTree is not invoked, thus disabling message pre translation along with IsDialogMessage call (specific messages only, it's not as scary as it seems, but hits the performance and hard to implement properly)

     If CWinThread::PreTranslateMessage returns TRUE the following applies:

     • Message is not dispatched
     • Focused inline controls don't receive WM_KEYDOWN for VK_RETURN and VK_ESCAPE codes

     If CWinThread::PreTranslateMessage returns FALSE the following applies:

     • Message is dispatched
     • Focused inline controls receive WM_KEYDOWN for VK_RETURN and VK_ESCAPE codes
   • The behavior is application wide

The very best option would be to override CWnd::IsDialogMessage and return in case of VK_RETURN and VK_ESCAPE, but this is impossible since CWnd::IsDialogMessage is not virtual. CWnd::IsDialogMessage is an API function wrapper, the closest item to modify the behavior.

So which method is the best? Each does the trick, but... Option 3 has serious drawback. Option 1 and 2 are class specific. I vote for option 2. You could create some base CDialog class with the trick and use it for all your dialogs.