Hungry Mind , Blog about everything in IT - C#, Java, C++, .NET, Windows, WinAPI, ...

Object Oriented WinAPI hooks

Some time ago I was playing with a WH_GETMESSAGE message hook to intercept any mouse and keyboard input. The hook was installed in several UI threads controlled by different application modules. In order to install a message hook, you issue the following code:

const HHOOK hHook = ::SetWindowsHookEx(WH_GETMESSAGE, &GetMsgProc, NULL, ::GetCurrentThreadId());

and your GetMsgProc looks like this one:

LRESULT CALLBACK GetMsgProc(int code, WPARAM wParam, LPARAM lParam)
{
   // Extra work here...
   return ::CallNextHookEx(NULL, code, wParam, lParam);
}

What if you need to pass control over to some object, invoke a member function? Keep in mind that GetMsgProc is a free function (that doesn't receive hook handle in its parameters). Therefore you can't use anything to map invocation context to OOP world citizens. You could use a global object to overcome this issue, but still you can't install more than one message hook with the same callback function.

So the only thing that can map to an object ... is the callback function itself. Hopefully, we are able to create that function on the fly, in memory:

struct IGetMessageHookTarget
{
   virtual LRESULT GetMsgProc(int code, WPARAM wParam, LPARAM lParam)
   {
      return ::CallNextHookEx(NULL, code, wParam, lParam);
   }
};

class GetMessageHook : IGetMessageHookTarget
{
   static const size_t ProcSize = 18;
public:
   GetMessageHook() {
      BYTE getMsgProcBytes[ProcSize] =
      {
         0x55,                           // push        ebp
         0x8B, 0xEC,                     // mov         ebp,esp
         0xA1, 0x00, 0x00, 0x00, 0x00,   // mov         eax,dword ptr ds:[this]
         0xB9, 0x00, 0x00, 0x00, 0x00,   // mov         ecx,this
         0x8B, 0x10,                     // mov         edx,dword ptr [eax]
         0x5D,                           // pop         ebp
         0xFF, 0xE2,                     // jmp         edx
      };

      *reinterpret_cast<size_t *>(&getMsgProcBytes[4]) = reinterpret_cast<size_t>(this);
      *reinterpret_cast<size_t *>(&getMsgProcBytes[4 + 1 + sizeof(size_t)]) = reinterpret_cast<size_t>(this);

      const HANDLE hCurrentProcess = ::GetCurrentProcess();
      _getMsgProcBytes = ::VirtualAllocEx(hCurrentProcess, NULL, ProcSize, MEM_COMMIT | MEM_RESERVE, PAGE_EXECUTE_READWRITE);
      CopyMemory(_getMsgProcBytes, getMsgProcBytes, sizeof(getMsgProcBytes));
      ::VirtualProtectEx(hCurrentProcess, _getMsgProcBytes, ProcSize, PAGE_EXECUTE, NULL);
      
      _hHook = ::SetWindowsHookEx(WH_GETMESSAGE, reinterpret_cast<HOOKPROC>(_getMsgProcBytes), NULL, ::GetCurrentThreadId());
   }

   ~GetMessageHook() {
      ::UnhookWindowsHookEx(_hHook);
      ::VirtualFreeEx(::GetCurrentProcess(), _getMsgProcBytes, ProcSize, MEM_RELEASE);
   }

protected:
   virtual LRESULT GetMsgProc(int code, WPARAM wParam, LPARAM lParam)
   {
      return ::CallNextHookEx(NULL, code, wParam, lParam);
   }

private:
   HHOOK _hHook;
   LPVOID _getMsgProcBytes;
};

GetMessageHook constructor creates an in-memory code that passes control over to GetMsgProc member function. Believe me - it works. At least on x86.

Herman Miller Embody and Aeron

В семье пополнение...

Cavalcade of Cartoon Comedy

Наткнулся на серию мультфильмов-пародий, называется Кавалькада мультипликационных комедий / Cavalcade of Cartoon Comedy (Seth MacFarlane / Сет Макфарлейн) 2008 г.

How to prevent MFC dialog closing on Enter and Escape keys

In this article I explain how to prevent an MFC dialog from handling the Enter and Escape keys and not pass it on. With all required details for you to completely understand why does it work the way it does. There are several possible solutions, so you can choose the most suitable one.

The problem is the following: you have a dialog, no matter - top level window or just a child container; you press Escape or Enter keys - the dialog disappears, even if there are no OK and Cancel buttons on the dialog. It's very funny seeing a child dialog drops out it's parent.

Let's see why this happens:

OrderXMLReaderShell.exe!CFilterEditorDlgBase::OnCancel()  Line 61 C++
  mfc100ud.dll!_AfxDispatchCmdMsg(CCmdTarget * pTarget, unsigned int nID, int nCode, void (void)* pfn, void * pExtra, unsigned int nSig, AFX_CMDHANDLERINFO * pHandlerInfo)  Line 82 C++
  mfc100ud.dll!CCmdTarget::OnCmdMsg(unsigned int nID, int nCode, void * pExtra, AFX_CMDHANDLERINFO * pHandlerInfo)  Line 381 + 0x27 bytes C++
  mfc100ud.dll!CDialog::OnCmdMsg(unsigned int nID, int nCode, void * pExtra, AFX_CMDHANDLERINFO * pHandlerInfo)  Line 87 + 0x18 bytes C++
  mfc100ud.dll!CWnd::OnCommand(unsigned int wParam, long lParam)  Line 2729 C++
  mfc100ud.dll!CWnd::OnWndMsg(unsigned int message, unsigned int wParam, long lParam, long * pResult)  Line 2101 + 0x1e bytes C++
  mfc100ud.dll!CWnd::WindowProc(unsigned int message, unsigned int wParam, long lParam)  Line 2087 + 0x20 bytes C++
  mfc100ud.dll!AfxCallWndProc(CWnd * pWnd, HWND__ * hWnd, unsigned int nMsg, unsigned int wParam, long lParam)  Line 257 + 0x1c bytes C++
  mfc100ud.dll!AfxWndProc(HWND__ * hWnd, unsigned int nMsg, unsigned int wParam, long lParam)  Line 420 C++
  mfc100ud.dll!AfxWndProcBase(HWND__ * hWnd, unsigned int nMsg, unsigned int wParam, long lParam)  Line 420 + 0x15 bytes C++
  user32.dll!_InternalCallWinProc@20()  + 0x23 bytes 
  user32.dll!_UserCallWinProcCheckWow@32()  + 0xb3 bytes 
  user32.dll!_SendMessageWorker@20()  + 0xee bytes 
  user32.dll!_SendMessageW@16()  + 0x49 bytes 
  user32.dll!_IsDialogMessageW@8()  + 0xef46 bytes 
> mfc100ud.dll!CWnd::IsDialogMessageW(tagMSG * lpMsg)  Line 198 C++
  mfc100ud.dll!CWnd::PreTranslateInput(tagMSG * lpMsg)  Line 4713 C++
  mfc100ud.dll!CDialog::PreTranslateMessage(tagMSG * pMsg)  Line 82 C++
  OrderXMLReaderShell.exe!CFindDlg::PreTranslateMessage(tagMSG * pMsg)  Line 56 C++
  mfc100ud.dll!CWnd::WalkPreTranslateTree(HWND__ * hWndStop, tagMSG * pMsg)  Line 3311 + 0x14 bytes C++
  mfc100ud.dll!AfxInternalPreTranslateMessage(tagMSG * pMsg)  Line 233 + 0x12 bytes C++
  mfc100ud.dll!CWinThread::PreTranslateMessage(tagMSG * pMsg)  Line 777 + 0x9 bytes C++
  mfc100ud.dll!AfxPreTranslateMessage(tagMSG * pMsg)  Line 252 + 0x11 bytes C++
  mfc100ud.dll!AfxInternalPumpMessage()  Line 178 + 0x18 bytes C++
  mfc100ud.dll!CWinThread::PumpMessage()  Line 900 C++
  mfc100ud.dll!CWinThread::Run()  Line 629 + 0xd bytes C++
  mfc100ud.dll!CWinApp::Run()  Line 832 C++
  mfc100ud.dll!AfxWinMain(HINSTANCE__ * hInstance, HINSTANCE__ * hPrevInstance, wchar_t * lpCmdLine, int nCmdShow)  Line 47 + 0xd bytes C++
  OrderXMLReaderShell.exe!wWinMain(HINSTANCE__ * hInstance, HINSTANCE__ * hPrevInstance, wchar_t * lpCmdLine, int nCmdShow)  Line 26 C++
  OrderXMLReaderShell.exe!__tmainCRTStartup()  Line 547 + 0x2c bytes C
  OrderXMLReaderShell.exe!wWinMainCRTStartup()  Line 371 C
  kernel32.dll!@BaseThreadInitThunk@12()  + 0x12 bytes 
  ntdll.dll!___RtlUserThreadStart@8()  + 0x27 bytes 
  ntdll.dll!__RtlUserThreadStart@8()  + 0x1b bytes 

The execution flow got into CDialog::OnCancel virtual method. Here is the default MFC's implementation:

void CDialog::OnCancel()
{
 EndDialog(IDCANCEL);
}

It simply ends the dialog, thus destroying the window.

Lets find out how do we get there. The control flow is the following:

  1. Escape key is down
  2. WM_KEYDOWN message is queued into message queue
  3. AfxInternalPumpMessage gets the message from the queue and invokes AfxPreTranslateMessage
  4. CWnd::IsDialogMessage is reached
  5. IsDialogMessage WinAPI function is called by CWnd::IsDialogMessage; this function is the core of dialog management - it does things like keyboard navigation.
  6. There is a user32.dll!_SendMessageW@16() call from IsDialogMessageW
  7. Which brings us to CWnd::OnCommand with wParam equals to IDCANCEL
  8. End finally to CDialog::OnCancel

IsDialogMessageW examines the message and invokes SendMessage(hwndDlg, WM_COMMAND, MAKELONG(IDCANCEL, BN_CLICKED), ...);.

Short summary - MFC calls IsDialogMessage API function, which in turns emulates IDCANCEL button click, which is being handled by MFC eventually destroying the dialog.

It is not obvious why IsDialogMessageW emulates IDOK and IDCANCEL button clicks. I believe this is done to make developer's life easier. Imagine you have no control over the message loop, can't do any message pre translation. You'll have to handle Enter and Escape keys for every control on the dialog. That's pain.

What can we do to solve the problem? You can't change OS behavior for sure, but can do the following:

  1. Override CDialog::OnOK/OnCancel and do nothing

    The following applies:

    • Dialog is not destroyed on Enter or Escape
    • Focused inline controls don't receive WM_KEYDOWN for VK_ENTER and VK_ESCAPE codes
    • OK/Cancel button clicks have no effect
  2. Override CDialog::PreTranslateMessage and don't call the base if pMsg->message == WM_KEYDOWN && (pMsg->wParam == VK_ESCAPE || pMsg->wParam == VK_RETURN)

    If CDialog::PreTranslateMessage returns TRUE the following applies:

    • Dialog is not destroyed on Enter or Escape
    • Focused inline controls don't receive WM_KEYDOWN for VK_RETURN and VK_ESCAPE codes, unless you do dispatch them before CDialog::PreTranslateMessage returns
    • OK/Cancel button clicks have desired effect

    If CDialog::PreTranslateMessage returns FALSE the following applies:

    • MFC calls parent's PreTranslateMessage (which brings us back if the parent is a CDialog instance)
  3. Override CWinThread::PreTranslateMessage, examine the message and don't call the base

    The following applies:

    • CWnd::WalkPreTranslateTree is not invoked, thus disabling message pre translation along with IsDialogMessage call (specific messages only, it's not as scary as it seems, but hits the performance and hard to implement properly)

      If CWinThread::PreTranslateMessage returns TRUE the following applies:

      • Message is not dispatched
      • Focused inline controls don't receive WM_KEYDOWN for VK_RETURN and VK_ESCAPE codes

      If CWinThread::PreTranslateMessage returns FALSE the following applies:

      • Message is dispatched
      • Focused inline controls receive WM_KEYDOWN for VK_RETURN and VK_ESCAPE codes
    • The behavior is application wide

The very best option would be to override CWnd::IsDialogMessage and return in case of VK_RETURN and VK_ESCAPE, but this is impossible since CWnd::IsDialogMessage is not virtual. CWnd::IsDialogMessage is an API function wrapper, the closest item to modify the behavior.

So which method is the best? Each does the trick, but... Option 3 has serious drawback. Option 1 and 2 are class specific. I vote for option 2. You could create some base CDialog class with the trick and use it for all your dialogs.

Funny site error

Copyright 2007-2011 Chabster