From 2f1eb7876ce087844fe59c21ca41f1eb139de473 Mon Sep 17 00:00:00 2001 From: clover caruso Date: Tue, 6 Oct 2026 22:08:37 -0700 Subject: [PATCH] Fix Windows surface capture and accelerate desktop streaming MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Use cursor-free monitor capture on modern Windows, drain to the latest frames, and retain the GDI compatibility fallback. Align UWP window content extents, keep shell popup stacking correct, and close rejected capture sessions. Initialize shell surfaces on bounded workers, preserve covered background pixels, and verify caption hit regions before optimistic dragging. Validated fresh Calculator and Store launches, Start/taskbar stacking, resizing, dragging, native cursor reconnects, and idle capture on Windows 10/11. Animated browser video reached 37–44 FPS with no dropped frames; inference sustained 42.9 tokens/sec. x86/x64/ARM64 imports and subsystem checks plus cursor ASan/UBSan tests pass. Assisted-by: gpt-6.1-sol --- guest/DESIGN.md | 8 +- guest/capture.c | 326 ++++++++++++++++++++++++++++++++++++++++-------- guest/capture.h | 6 +- guest/windows.c | 28 +++-- 4 files changed, 301 insertions(+), 67 deletions(-) diff --git a/guest/DESIGN.md b/guest/DESIGN.md index f1e4dbc8954fe9039c75e3a06a1efd7a0301e413..ba8edf00a88b8b7b36e7244f235d6e7927c223ad 100644 --- a/guest/DESIGN.md +++ b/guest/DESIGN.md @@ -1,10 +1,10 @@ # Guest display stream -The executable is freestanding C99. `cursor.c` contains pixel conversion and dirty-rectangle detection; `windows.c` owns desktop capture and transport; `capture.c` implements optional window surfaces. Zig supplies the cross compiler and Windows headers, with no Zig or C runtime in the executable. x86 and x64 target Windows 7; ARM64 targets Windows 10. PE subsystem versions and imported DLLs are checked by `python3 guest/test.py`. Runtime validation uses Windows 10 and 11; Windows 7 runtime behavior is unverified. +The executable is freestanding C99. `cursor.c` contains pixel conversion and dirty-rectangle detection; `windows.c` owns desktop capture and transport; `capture.c` implements modern desktop capture and optional window surfaces. Zig supplies the cross compiler and Windows headers, with no Zig or C runtime in the executable. x86 and x64 target Windows 7; ARM64 targets Windows 10. PE subsystem versions and imported DLLs are checked by `python3 guest/test.py`. Runtime validation uses Windows 10 and 11; Windows 7 runtime behavior is unverified. Build with `python3 guest/build.py --arch x86_64` (also `x86` and `aarch64`). In an elevated PowerShell session inside the logged-in administrator's desktop, run `./install.ps1 -Binary ./build/snowglobe-guest-x86_64.exe`. The installer uses Task Scheduler COM APIs available to PowerShell 2, installs into protected Program Files, and starts an elevated task at that user's logon. `-Remove` removes that task and executable. Elevation is required by the VirtIO serial driver's device ACL; capture must run in the interactive session, not session 0. The guest needs a compatible VirtIO serial driver and the channel named in `protocol.json`. A standard-user launcher requires a privileged handle broker; it is not part of this first backend. -`BitBlt` copies the primary desktop into a CPU DIB without drawing the cursor. RGB comparisons ignore the unused fourth byte. Unchanged pixels produce no screen packets; changed pixels produce one bounding rectangle. Cursor polling runs every millisecond; desktop capture targets 60 Hz. The host's existing encoder supplies WebRTC video. Secure desktops, RDP sessions that differ from the physical console, and capture failures return the viewer to QEMU capture. +Windows 10 build 19041 and later use Windows Graphics Capture for the primary monitor with cursor capture disabled. Older or unsupported systems use `BitBlt` into a CPU DIB without drawing the cursor. Only the latest available capture frames are read back; an unchanged desktop needs no readback. RGB comparisons ignore the unused fourth byte. Unchanged pixels produce no screen packets; changed pixels produce one bounding rectangle. Cursor polling runs every millisecond; desktop capture targets 60 Hz. The host's existing encoder supplies WebRTC video. Secure desktops, RDP sessions that differ from the physical console, and capture failures return the viewer to QEMU capture. `GetCursorInfo` supplies the shape, visibility, and observed position. `GetIconInfo` and black/white `DrawIconEx` renders recover RGBA and binary XOR planes, including colored XOR. Nonbinary color XOR and unsupported cursor sizes explicitly fall back to the browser's default cursor, preserving video. Animation timing uses the optional `GetCursorFrameInfo` user32 export, resolved by name. Missing or unusable animation metadata falls back to a static frame. Animation starts a browser-local clock; Windows does not expose the current animation phase through this interface. Ordinary shapes use the browser cursor; XOR shapes use local layers with difference blending. Neither follows streamed positions. @@ -29,9 +29,9 @@ The host broker validates a running VM's libvirt-owned channel path. One guest r ## Separate windows experiment -Windows 10 build 19041 and later can enable separate windows from the viewer’s View menu. Windows Graphics Capture supplies full unoccluded surfaces with cursor capture disabled. Its dependencies are loaded dynamically, preserving the Windows 7 import boundary. Desktop and taskbar surfaces use `PrintWindow` because Windows Graphics Capture rejects those shell HWNDs. Changed rectangles update a single host atlas and one existing video encoder. Metadata describes the atlas and stacking; the browser composes it and moves cached surfaces immediately during native caption drags. Atlas geometry is matched to the decoded RTP frame. +Windows 10 build 19041 and later can enable separate windows from the viewer’s View menu. Windows Graphics Capture supplies full unoccluded surfaces with cursor capture disabled. Its dependencies are loaded dynamically, preserving the Windows 7 import boundary. Desktop and taskbar surfaces initialize with a `PrintWindow` worker because Windows Graphics Capture rejects those shell HWNDs. Subsequent updates copy visible desktop spans, retaining covered pixels for local window dragging. Shell surfaces initialize before window capture sessions start, keeping capture indicators out of their initial pixels. Changed rectangles update a single host atlas and one existing video encoder. Metadata describes the atlas and stacking; the browser composes it and moves cached surfaces immediately during native caption drags. Atlas geometry is matched to the decoded RTP frame. -The experiment is off by default. Unsupported, layered, or oversized window sets fall back to desktop video. Secure desktops and session changes suspend guest capture. `PrintWindow` is synchronous; an unresponsive shell can interrupt the guest stream and trigger the host timeout. Protected content and custom caption behavior remain subject to the application’s capture support. +The experiment is off by default. Unsupported, layered, translucent, shell-popup, or oversized window sets close all window sessions and fall back to desktop video. Frame-pool resizing follows captured content dimensions; a window whose first or resized surface never arrives also falls back. Secure desktops and session changes suspend guest capture. Shell painting runs off the capture thread and falls back after 500 ms; its bounded drawing resources remain owned until the worker exits. Windows capture indicators remain subject to the OS; fallback disposes the sessions instead of leaving indicators attached to inactive windows. Protected content and custom caption behavior remain subject to the application’s capture support. ## Later backends diff --git a/guest/capture.c b/guest/capture.c index 4cf10d863a3c2b26032b5f5372d04e3b9eb72022..bc2f2e94d2b44c00c0b3b2941d1807cd3e02f9ab 100644 --- a/guest/capture.c +++ b/guest/capture.c @@ -37,22 +37,35 @@ static const GUID dxgi_iid = {0x54ec77fa,0x1377,0x44e6,{0x8c,0x32,0x88,0xfd,0x5f static const GUID access_iid = {0xa9b3d012,0x3df2,0x4ee3,{0xb8,0xd1,0x86,0x95,0xf4,0x57,0xd3,0xc1}}; static const GUID texture_iid = {0x6f15aaf2,0xd208,0x4e89,{0x9a,0xb4,0x48,0x95,0x35,0xd3,0x4f,0x9c}}; +struct shell_paint { + HWND hwnd; + HANDLE thread; + struct shell_paint *next; + HDC dc; + HBITMAP dib, selected; + uint8_t *bits; + size_t pixels; + DWORD started; +}; + struct window { HWND hwnd; uint32_t id, width, height; struct size pool_size; void *pool, *session; ID3D11Texture2D *staging; - uint8_t *before, *bits; - HDC dc; - HBITMAP dib, selected; + uint8_t *before; + struct shell_paint *paint; int initial, seen, shell; + DWORD waiting; }; -static struct window windows[SG_MAX_WINDOWS]; +static struct window windows[SG_MAX_WINDOWS], desktop; +static RECT bounds[SG_MAX_WINDOWS], capture_bounds[SG_MAX_WINDOWS]; static uint32_t layout[SG_WINDOWS_WORDS + SG_MAX_WINDOWS * SG_WINDOW_WORDS]; static uint32_t prior[SG_WINDOWS_WORDS + SG_MAX_WINDOWS * SG_WINDOW_WORDS]; static uint32_t count, next_id; -static size_t allocated; +static size_t allocated, painted; +static struct shell_paint *retired_paints; static int invalid; static HANDLE heap; /* Capture workers outlive session disposal; retain their DLLs until process exit. */ @@ -65,7 +78,7 @@ static delete_string_fn delete_string; static factory_fn get_factory; static uninitialize_fn uninitialize; static attribute_fn attribute; -static DWORD retry; +static DWORD retry, desktop_retry; static void release(void *object) { if (object) METHOD(object, 2, release_fn)(object); @@ -78,13 +91,41 @@ static void close_object(void *object) { } release(object); } +static void free_paint(struct shell_paint *paint) { + if (paint->thread) CloseHandle(paint->thread); + if (paint->selected) SelectObject(paint->dc, paint->selected); + if (paint->dib) DeleteObject(paint->dib); + if (paint->dc) DeleteDC(paint->dc); + painted -= paint->pixels; + HeapFree(heap, 0, paint); +} +static DWORD WINAPI paint_shell(void *data) { + struct shell_paint *paint = data; + BOOL success = PrintWindow(paint->hwnd, paint->dc, 2); + GdiFlush(); + return success; +} +static void reap_paints(void) { + struct shell_paint **link = &retired_paints; + while (*link) { + struct shell_paint *paint = *link; + if (WaitForSingleObject(paint->thread, 0) == WAIT_OBJECT_0) { + *link = paint->next; free_paint(paint); + } else link = &paint->next; + } +} +static void detach_paint(struct window *window) { + struct shell_paint *paint = window->paint; + if (!paint) return; + window->paint = NULL; + if (!paint->thread || WaitForSingleObject(paint->thread, 0) == WAIT_OBJECT_0) free_paint(paint); + else { paint->next = retired_paints; retired_paints = paint; } +} + static void drop(struct window *window) { close_object(window->session); close_object(window->pool); if (window->staging) ID3D11Texture2D_Release(window->staging); - if (window->selected) SelectObject(window->dc, window->selected); - if (window->dib) DeleteObject(window->dib); - if (window->dc) DeleteDC(window->dc); - window->dc = NULL; window->dib = window->selected = NULL; window->bits = NULL; + detach_paint(window); if (window->before) { allocated -= (size_t)window->width * window->height; HeapFree(heap, 0, window->before); } window->hwnd = NULL; window->session = window->pool = NULL; window->staging = NULL; window->before = NULL; @@ -103,26 +144,33 @@ static int factory(PCWSTR name, const GUID *iid, void **result) { } void sg_windows_stop(void) { + reap_paints(); uint32_t i; for (i = 0; i < SG_MAX_WINDOWS; ++i) drop(windows + i); + count = 0; + retry = 0; + for (i = 0; i < sizeof(prior) / sizeof(prior[0]); ++i) prior[i] = 0; +} + +void sg_capture_stop(void) { + sg_windows_stop(); + drop(&desktop); + desktop_retry = 0; release(pool_factory); release(interop); release(capture_device); if (context) ID3D11DeviceContext_Release(context); if (device) ID3D11Device_Release(device); pool_factory = interop = capture_device = NULL; context = NULL; device = NULL; if (uninitialize) uninitialize(); uninitialize = NULL; - count = 0; - retry = 0; - for (i = 0; i < sizeof(prior) / sizeof(prior[0]); ++i) prior[i] = 0; } -int sg_windows_start(void) { +int sg_capture_start(void) { initialize_fn initialize; typedef HRESULT (WINAPI *create_fn)(IDXGIAdapter *, D3D_DRIVER_TYPE, HMODULE, UINT, const D3D_FEATURE_LEVEL *, UINT, UINT, ID3D11Device **, D3D_FEATURE_LEVEL *, ID3D11DeviceContext **); create_fn create; device_fn wrap; IDXGIDevice *dxgi = NULL; - sg_windows_stop(); + if (device) return 1; heap = GetProcessHeap(); if (!runtime) runtime = LoadLibraryW(L"combase.dll"); if (!d3d) d3d = LoadLibraryW(L"d3d11.dll"); @@ -148,7 +196,7 @@ int sg_windows_start(void) { if (FAILED(status)) goto failed; return 1; failed: - sg_windows_stop(); + sg_capture_stop(); return 0; } @@ -173,11 +221,20 @@ static BOOL CALLBACK enumerate(HWND hwnd, LPARAM unused) { GetClassNameW(hwnd, class_name, 32); windows[i].shell = hwnd == GetShellWindow() || !lstrcmpW(class_name, L"Shell_TrayWnd") || !lstrcmpW(class_name, L"Shell_SecondaryTrayWnd") || !lstrcmpW(class_name, L"WorkerW"); - if (FAILED(attribute(hwnd, 9, &rect, sizeof(rect)))) GetWindowRect(hwnd, &rect); + if ((!lstrcmpW(class_name, L"ApplicationFrameWindow") && !GetWindowTextLengthW(hwnd)) || !lstrcmpW(class_name, L"Windows.UI.Core.CoreWindow") || !lstrcmpW(class_name, L"Xaml_WindowedPopupClass") || !lstrcmpW(class_name, L"XamlExplorerHostIslandWindow")) { invalid = 1; return FALSE; } + if (!windows[i].shell && (GetWindowLongW(hwnd, GWL_STYLE) & WS_CAPTION) != WS_CAPTION) { invalid = 1; return FALSE; } + capture_bounds[count] = rect; + if (!windows[i].shell) attribute(hwnd, 9, &rect, sizeof(rect)); + if (windows[i].before && !windows[i].initial && + (windows[i].width != (uint32_t)(rect.right - rect.left) || windows[i].height != (uint32_t)(rect.bottom - rect.top))) { + windows[i].initial = 1; windows[i].waiting = GetTickCount(); + } + bounds[count] = rect; uint32_t *record = layout + SG_WINDOWS_WORDS + count++ * SG_WINDOW_WORDS; record[SG_WINDOW_ID] = windows[i].id; record[SG_WINDOW_X] = (uint32_t)rect.left; record[SG_WINDOW_Y] = (uint32_t)rect.top; - record[SG_WINDOW_WIDTH] = (uint32_t)(rect.right - rect.left); record[SG_WINDOW_HEIGHT] = (uint32_t)(rect.bottom - rect.top); + record[SG_WINDOW_WIDTH] = windows[i].before && !windows[i].shell ? windows[i].width : (uint32_t)(rect.right - rect.left); + record[SG_WINDOW_HEIGHT] = windows[i].before && !windows[i].shell ? windows[i].height : (uint32_t)(rect.bottom - rect.top); for (i = SG_WINDOW_CAPTION_LEFT; i <= SG_WINDOW_CAPTION_BOTTOM; ++i) record[i] = 0; if ((GetWindowLongW(hwnd, GWL_STYLE) & WS_CAPTION) == WS_CAPTION) { TITLEBARINFO title = {0}; @@ -187,6 +244,16 @@ static BOOL CALLBACK enumerate(HWND hwnd, LPARAM unused) { record[SG_WINDOW_CAPTION_TOP] = (uint32_t)title.rcTitleBar.top; record[SG_WINDOW_CAPTION_RIGHT] = (uint32_t)(title.rcTitleBar.right - GetSystemMetrics(SM_CXSIZE) * 3); record[SG_WINDOW_CAPTION_BOTTOM] = (uint32_t)title.rcTitleBar.bottom; + LONG left = (LONG)record[SG_WINDOW_CAPTION_LEFT], right = (LONG)record[SG_WINDOW_CAPTION_RIGHT]; + LONG y = (title.rcTitleBar.top + title.rcTitleBar.bottom) / 2; + for (int sample = 0; sample < 3; ++sample) { + LONG x = left + 1 + sample * (right - left - 2) / 2; + DWORD_PTR hit; + if (right <= left || !SendMessageTimeoutW(hwnd, WM_NCHITTEST, 0, MAKELPARAM(x, y), SMTO_ABORTIFHUNG | SMTO_BLOCK, 1, &hit) || hit != HTCAPTION) { + for (i = SG_WINDOW_CAPTION_LEFT; i <= SG_WINDOW_CAPTION_BOTTOM; ++i) record[i] = 0; + break; + } + } } } return TRUE; @@ -196,46 +263,147 @@ static int start_window(struct window *window) { void *item = NULL, *cursor = NULL; struct size size; int success = 0; - if (FAILED(METHOD(interop, 3, window_fn)(interop, window->hwnd, &item_iid, &item))) goto done; + HWND target = window->hwnd ? window->hwnd : (HWND)MonitorFromPoint((POINT){0, 0}, MONITOR_DEFAULTTOPRIMARY); + if (FAILED(METHOD(interop, window->hwnd ? 3 : 4, window_fn)(interop, target, &item_iid, &item))) goto done; if (FAILED(METHOD(item, 7, size_fn)(item, &size)) || size.width <= 0 || size.height <= 0 || size.width > (INT32)SG_MAX_DIMENSION || size.height > (INT32)SG_MAX_DIMENSION) goto done; if (FAILED(METHOD(pool_factory, 6, pool_fn)(pool_factory, capture_device, DXGI_FORMAT_B8G8R8A8_UNORM, 2, size, &window->pool))) goto done; if (FAILED(METHOD(window->pool, 10, session_fn)(window->pool, item, &window->session))) goto done; if (FAILED(METHOD(window->session, 0, query_fn)(window->session, &cursor_iid, &cursor))) goto done; if (FAILED(METHOD(cursor, 7, bool_fn)(cursor, FALSE)) || FAILED(METHOD(window->session, 6, close_fn)(window->session))) goto done; window->pool_size = size; + window->waiting = GetTickCount(); success = 1; done: release(cursor); release(item); return success; } -int sg_windows_poll(sg_emit emit, int refresh) { +static int latest_frame(struct window *window, void **frame) { + void *next = NULL; + for (int i = 0; i < 2; ++i) { + if (FAILED(METHOD(window->pool, 7, object_fn)(window->pool, &next))) return 0; + if (!next) return 1; + close_object(*frame); + *frame = next; + } + return 1; +} + +int sg_desktop_capture(uint8_t *pixels, uint32_t width, uint32_t height, int refresh) { + reap_paints(); + void *frame = NULL, *surface = NULL, *access = NULL; + ID3D11Texture2D *texture = NULL; + D3D11_TEXTURE2D_DESC desc; + D3D11_MAPPED_SUBRESOURCE mapped; + struct size size; + int result = -1; + if (desktop_retry && (DWORD)(GetTickCount() - desktop_retry) < 1000) return -1; + if (!device && !sg_capture_start()) { desktop_retry = GetTickCount(); return -1; } + if (!desktop.pool && !start_window(&desktop)) goto done; + if (!latest_frame(&desktop, &frame)) goto done; + result = 0; + if (!frame) { + if (refresh && desktop.staging && desktop.width == width && desktop.height == height) goto readback; + if ((DWORD)(GetTickCount() - desktop.waiting) > 500 && (!desktop.staging || desktop.width != width || desktop.height != height)) result = -1; + goto done; + } + if (FAILED(METHOD(frame, 8, size_fn)(frame, &size))) { result = -1; goto done; } + if (size.width != (INT32)width || size.height != (INT32)height) { result = -1; goto done; } + if (desktop.pool_size.width != size.width || desktop.pool_size.height != size.height) { + if (FAILED(METHOD(desktop.pool, 6, recreate_fn)(desktop.pool, capture_device, DXGI_FORMAT_B8G8R8A8_UNORM, 2, size))) { result = -1; goto done; } + desktop.pool_size = size; + goto done; + } + result = -1; + if (FAILED(METHOD(frame, 6, object_fn)(frame, &surface)) || + FAILED(METHOD(surface, 0, query_fn)(surface, &access_iid, &access)) || + FAILED(METHOD(access, 3, query_fn)(access, &texture_iid, (void **)&texture))) goto done; + ID3D11Texture2D_GetDesc(texture, &desc); + if (desc.Width < width || desc.Height < height) goto done; + if (desktop.width != width || desktop.height != height) { + if (desktop.staging) ID3D11Texture2D_Release(desktop.staging); + desktop.staging = NULL; + desc.Width = width; desc.Height = height; + desc.BindFlags = 0; desc.MiscFlags = 0; desc.Usage = D3D11_USAGE_STAGING; desc.CPUAccessFlags = D3D11_CPU_ACCESS_READ; + if (FAILED(ID3D11Device_CreateTexture2D(device, &desc, NULL, &desktop.staging))) goto done; + desktop.width = width; desktop.height = height; + } + D3D11_BOX box = {0, 0, 0, width, height, 1}; + ID3D11DeviceContext_CopySubresourceRegion(context, (ID3D11Resource *)desktop.staging, 0, 0, 0, 0, (ID3D11Resource *)texture, 0, &box); +readback: + result = -1; + if (FAILED(ID3D11DeviceContext_Map(context, (ID3D11Resource *)desktop.staging, 0, D3D11_MAP_READ, 0, &mapped))) goto done; + for (uint32_t y = 0; y < height; ++y) { + uint8_t *src = (uint8_t *)mapped.pData + (size_t)y * mapped.RowPitch; + uint8_t *dst = pixels + (size_t)y * width * 4; + for (uint32_t x = 0; x < width * 4; ++x) dst[x] = src[x]; + } + ID3D11DeviceContext_Unmap(context, (ID3D11Resource *)desktop.staging, 0); + result = 1; +done: + if (texture) ID3D11Texture2D_Release(texture); + release(access); release(surface); close_object(frame); + if (result < 0) { drop(&desktop); desktop_retry = GetTickCount(); } + else desktop_retry = 0; + return result; +} + +static int atlas_fits(void) { + uint32_t width = layout[SG_WINDOWS_WIDTH], height = 0, x = 0, row_height = 0, last_id = 0; + for (uint32_t i = 0; i < count; ++i) { + uint32_t w = layout[SG_WINDOWS_WORDS + i * SG_WINDOW_WORDS + SG_WINDOW_WIDTH]; + if (w > width) width = w; + } + for (uint32_t i = 0; i < count; ++i) { + uint32_t *next = NULL; + for (uint32_t n = 0; n < count; ++n) { + uint32_t *record = layout + SG_WINDOWS_WORDS + n * SG_WINDOW_WORDS; + if (record[SG_WINDOW_ID] > last_id && (!next || record[SG_WINDOW_ID] < next[SG_WINDOW_ID])) next = record; + } + uint32_t w = next[SG_WINDOW_WIDTH], h = next[SG_WINDOW_HEIGHT]; + if (!w || !h || w > SG_MAX_DIMENSION || h > SG_MAX_DIMENSION) return 0; + if (x + w > width) { x = 0; height += row_height; row_height = 0; } + x += w; + if (h > row_height) row_height = h; + last_id = next[SG_WINDOW_ID]; + } + height += row_height; + return width <= SG_MAX_DIMENSION && height <= SG_MAX_DIMENSION && (size_t)width * height <= SG_MAX_PIXELS; +} + +int sg_windows_poll(sg_emit emit, int refresh, const uint8_t *pixels, uint32_t width, uint32_t height) { uint32_t i, n, words[SG_SURFACE_WORDS] = {0}; if (retry && (DWORD)(GetTickCount() - retry) < 1000 && !refresh) return 1; retry = 0; count = 0; invalid = 0; - layout[SG_WINDOWS_WIDTH] = (uint32_t)GetSystemMetrics(SM_CXSCREEN); - layout[SG_WINDOWS_HEIGHT] = (uint32_t)GetSystemMetrics(SM_CYSCREEN); + layout[SG_WINDOWS_WIDTH] = width; + layout[SG_WINDOWS_HEIGHT] = height; for (i = 0; i < SG_MAX_WINDOWS; ++i) windows[i].seen = 0; - EnumWindows(enumerate, 0); + HWND foreground = GetForegroundWindow(); + if (foreground && foreground != GetShellWindow() && foreground != FindWindowW(L"Shell_TrayWnd", NULL) && + (GetWindowLongW(foreground, GWL_STYLE) & WS_CAPTION) != WS_CAPTION) invalid = 1; + if (!invalid) EnumWindows(enumerate, 0); for (i = 0; i < SG_MAX_WINDOWS; ++i) if (windows[i].hwnd && !windows[i].seen) drop(windows + i); + if (!invalid && !atlas_fits()) invalid = 1; layout[SG_WINDOWS_COUNT] = invalid ? 0 : count; n = SG_WINDOWS_WORDS + layout[SG_WINDOWS_COUNT] * SG_WINDOW_WORDS; int changed = refresh; for (i = 0; i < n; ++i) if (layout[i] != prior[i]) changed = 1; if (changed && !emit(SG_WINDOWS, layout, n, NULL, 0)) return 0; for (i = 0; i < n; ++i) prior[i] = layout[i]; - if (invalid) return 1; - for (i = 0; i < SG_MAX_WINDOWS; ++i) { - struct window *window = windows + i; + if (invalid) { sg_windows_stop(); retry = GetTickCount(); return 1; } + for (i = 0; i < SG_MAX_WINDOWS * 2; ++i) { + struct window *window = windows + i % SG_MAX_WINDOWS; + if (window->shell != (i < SG_MAX_WINDOWS)) continue; void *frame = NULL, *surface = NULL, *access = NULL; ID3D11Texture2D *texture = NULL; D3D11_TEXTURE2D_DESC desc; + D3D11_BOX box; D3D11_MAPPED_SUBRESOURCE mapped; struct size size; struct sg_rect rect; uint32_t y; - int success = 1; + int success = 1, mapped_texture = 0; if (!window->hwnd) continue; uint32_t *record = NULL; for (n = 0; n < count; ++n) { @@ -244,22 +412,24 @@ int sg_windows_poll(sg_emit emit, int refresh) { } if (!record) continue; if (window->shell) { - RECT bounds; - if (!GetWindowRect(window->hwnd, &bounds)) goto frame_failed; - size.width = bounds.right - bounds.left; size.height = bounds.bottom - bounds.top; + size.width = capture_bounds[n].right - capture_bounds[n].left; + size.height = capture_bounds[n].bottom - capture_bounds[n].top; mapped.pData = NULL; } else { if (!window->pool && !start_window(window)) goto frame_failed; - struct size target = {(INT32)record[SG_WINDOW_WIDTH], (INT32)record[SG_WINDOW_HEIGHT]}; - if (window->pool_size.width != target.width || window->pool_size.height != target.height) { - if (FAILED(METHOD(window->pool, 6, recreate_fn)(window->pool, capture_device, DXGI_FORMAT_B8G8R8A8_UNORM, 2, target))) goto frame_failed; - window->pool_size = target; - window->initial = 1; + if (!latest_frame(window, &frame)) goto frame_failed; + if (!frame) { + if (window->initial && (DWORD)(GetTickCount() - window->waiting) > 500) goto frame_failed; + continue; } - if (FAILED(METHOD(window->pool, 7, object_fn)(window->pool, &frame))) goto frame_failed; - if (!frame) continue; if (FAILED(METHOD(frame, 8, size_fn)(frame, &size))) goto frame_failed; - if (size.width != target.width || size.height != target.height) goto frame_done; + if (window->pool_size.width != size.width || window->pool_size.height != size.height) { + if (size.width <= 0 || size.height <= 0 || size.width > (INT32)SG_MAX_DIMENSION || size.height > (INT32)SG_MAX_DIMENSION) goto frame_failed; + if (FAILED(METHOD(window->pool, 6, recreate_fn)(window->pool, capture_device, DXGI_FORMAT_B8G8R8A8_UNORM, 2, size))) goto frame_failed; + window->pool_size = size; + window->initial = 1; window->waiting = GetTickCount(); + goto frame_done; + } if (FAILED(METHOD(frame, 6, object_fn)(frame, &surface)) || FAILED(METHOD(surface, 0, query_fn)(surface, &access_iid, &access)) || FAILED(METHOD(access, 3, query_fn)(access, &texture_iid, (void **)&texture))) goto frame_failed; @@ -268,13 +438,26 @@ int sg_windows_poll(sg_emit emit, int refresh) { if (texture) { ID3D11Texture2D_GetDesc(texture, &desc); if (desc.Width < (UINT)size.width || desc.Height < (UINT)size.height) goto frame_failed; + LONG left = 0, top = 0; + LONG right = bounds[n].right - bounds[n].left, bottom = bounds[n].bottom - bounds[n].top; + if (size.width == capture_bounds[n].right - capture_bounds[n].left && size.height == capture_bounds[n].bottom - capture_bounds[n].top) { + left = bounds[n].left - capture_bounds[n].left; top = bounds[n].top - capture_bounds[n].top; + right += left; bottom += top; + } else if (size.width != right || size.height != bottom) goto frame_failed; + if (left < 0 || top < 0 || right > size.width || bottom > size.height || right <= left || bottom <= top) goto frame_failed; + box = (D3D11_BOX){(UINT)left, (UINT)top, 0, (UINT)right, (UINT)bottom, 1}; + size.width = right - left; size.height = bottom - top; + } + if (record[SG_WINDOW_WIDTH] != (uint32_t)size.width || record[SG_WINDOW_HEIGHT] != (uint32_t)size.height) { + record[SG_WINDOW_WIDTH] = (uint32_t)size.width; record[SG_WINDOW_HEIGHT] = (uint32_t)size.height; + if (!atlas_fits()) goto frame_failed; + if (!emit(SG_WINDOWS, layout, SG_WINDOWS_WORDS + count * SG_WINDOW_WORDS, NULL, 0)) { success = 0; goto frame_done; } + for (n = 0; n < SG_WINDOWS_WORDS + count * SG_WINDOW_WORDS; ++n) prior[n] = layout[n]; } if (window->width != (uint32_t)size.width || window->height != (uint32_t)size.height) { if (window->staging) ID3D11Texture2D_Release(window->staging); window->staging = NULL; - if (window->selected) SelectObject(window->dc, window->selected); - if (window->dib) DeleteObject(window->dib); - window->dib = window->selected = NULL; window->bits = NULL; + detach_paint(window); if (window->before) { allocated -= (size_t)window->width * window->height; HeapFree(heap, 0, window->before); window->before = NULL; } if (allocated + (size_t)size.width * size.height > SG_MAX_PIXELS) goto frame_failed; window->width = (uint32_t)size.width; window->height = (uint32_t)size.height; @@ -283,12 +466,21 @@ int sg_windows_poll(sg_emit emit, int refresh) { allocated += (size_t)window->width * window->height; if (window->shell) { BITMAPINFO info = {0}; - if (!window->dc) window->dc = CreateCompatibleDC(NULL); + size_t pixels = (size_t)window->width * window->height; + if (painted + pixels > SG_MAX_PIXELS) goto frame_failed; + window->paint = HeapAlloc(heap, HEAP_ZERO_MEMORY, sizeof(struct shell_paint)); + if (!window->paint) goto frame_failed; + struct shell_paint *paint = window->paint; + paint->pixels = pixels; painted += pixels; paint->hwnd = window->hwnd; + paint->dc = CreateCompatibleDC(NULL); info.bmiHeader.biSize = sizeof(BITMAPINFOHEADER); info.bmiHeader.biWidth = size.width; info.bmiHeader.biHeight = -size.height; info.bmiHeader.biPlanes = 1; info.bmiHeader.biBitCount = 32; - window->dib = CreateDIBSection(window->dc, &info, DIB_RGB_COLORS, (void **)&window->bits, NULL, 0); - if (!window->dc || !window->dib) goto frame_failed; - window->selected = SelectObject(window->dc, window->dib); + paint->dib = CreateDIBSection(paint->dc, &info, DIB_RGB_COLORS, (void **)&paint->bits, NULL, 0); + if (!paint->dc || !paint->dib) goto frame_failed; + paint->selected = SelectObject(paint->dc, paint->dib); + paint->started = GetTickCount(); + paint->thread = CreateThread(NULL, 0, paint_shell, paint, 0, NULL); + if (!paint->thread) goto frame_failed; } else { desc.Width = window->width; desc.Height = window->height; desc.BindFlags = 0; desc.MiscFlags = 0; desc.Usage = D3D11_USAGE_STAGING; desc.CPUAccessFlags = D3D11_CPU_ACCESS_READ; @@ -297,14 +489,45 @@ int sg_windows_poll(sg_emit emit, int refresh) { window->initial = 1; } if (window->shell) { - DWORD_PTR result; - if (!SendMessageTimeoutW(window->hwnd, WM_NULL, 0, 0, SMTO_ABORTIFHUNG | SMTO_BLOCK, 20, &result) || !PrintWindow(window->hwnd, window->dc, 2)) goto frame_failed; - GdiFlush(); - mapped.pData = window->bits; mapped.RowPitch = window->width * 4; + if (WaitForSingleObject(window->paint->thread, 0) != WAIT_OBJECT_0) { + if ((DWORD)(GetTickCount() - window->paint->started) >= 500) goto frame_failed; + return 1; + } + DWORD painted_ok; + if (!GetExitCodeThread(window->paint->thread, &painted_ok) || !painted_ok) goto frame_failed; + for (y = 0; y < window->height; ++y) { + LONG screen_y = (LONG)record[SG_WINDOW_Y] + (LONG)y; + if (screen_y < 0 || screen_y >= (LONG)layout[SG_WINDOWS_HEIGHT]) continue; + LONG x = (LONG)record[SG_WINDOW_X], right = x + (LONG)window->width; + if (x < 0) x = 0; + if (right > (LONG)layout[SG_WINDOWS_WIDTH]) right = (LONG)layout[SG_WINDOWS_WIDTH]; + while (x < right) { + LONG stop = right; + for (n = 0; n < count; ++n) { + uint32_t *other = layout + SG_WINDOWS_WORDS + n * SG_WINDOW_WORDS; + if (other[SG_WINDOW_ID] == window->id) break; + LONG left = capture_bounds[n].left - 2, top = capture_bounds[n].top - 2; + LONG edge = capture_bounds[n].right + 2; + if (screen_y < top || screen_y >= capture_bounds[n].bottom + 2) continue; + if (left <= x && edge > x) { x = edge; stop = x; break; } + if (left > x && left < stop) stop = left; + } + if (x >= stop) continue; + uint8_t *dst = window->paint->bits + ((size_t)y * window->width + (x - (LONG)record[SG_WINDOW_X])) * 4; + const uint8_t *src = pixels + ((size_t)screen_y * layout[SG_WINDOWS_WIDTH] + x) * 4; + for (LONG byte = 0; byte < (stop - x) * 4; ++byte) dst[byte] = src[byte]; + x = stop; + } + } + mapped.pData = window->paint->bits; mapped.RowPitch = window->width * 4; } else { - D3D11_BOX box = {0, 0, 0, window->width, window->height, 1}; ID3D11DeviceContext_CopySubresourceRegion(context, (ID3D11Resource *)window->staging, 0, 0, 0, 0, (ID3D11Resource *)texture, 0, &box); if (FAILED(ID3D11DeviceContext_Map(context, (ID3D11Resource *)window->staging, 0, D3D11_MAP_READ, 0, &mapped))) goto frame_failed; + mapped_texture = 1; + for (y = 8; y + 8 < window->height; ++y) { + const uint8_t *row = (const uint8_t *)mapped.pData + (size_t)y * mapped.RowPitch; + for (uint32_t x = 8; x + 8 < window->width; ++x) if (row[x * 4 + 3] != 255) goto frame_failed; + } } rect.x = window->width; rect.y = window->height; rect.width = rect.height = 0; if (refresh || window->initial) { rect.x = rect.y = 0; rect.width = window->width; rect.height = window->height; } @@ -330,17 +553,18 @@ int sg_windows_poll(sg_emit emit, int refresh) { } window->initial = 0; } - if (texture) ID3D11DeviceContext_Unmap(context, (ID3D11Resource *)window->staging, 0); goto frame_done; frame_failed: invalid = 1; frame_done: + if (mapped_texture) ID3D11DeviceContext_Unmap(context, (ID3D11Resource *)window->staging, 0); if (texture) ID3D11Texture2D_Release(texture); release(access); release(surface); close_object(frame); if (!success) return 0; if (invalid) { drop(window); break; } } if (invalid) { + sg_windows_stop(); retry = GetTickCount(); uint32_t disabled[SG_WINDOWS_WORDS] = {layout[SG_WINDOWS_WIDTH], layout[SG_WINDOWS_HEIGHT], 0}; if (!emit(SG_WINDOWS, disabled, SG_WINDOWS_WORDS, NULL, 0)) return 0; diff --git a/guest/capture.h b/guest/capture.h index 206b1d4783f0f94e5cd0773050c902c3ec95ef4f..b39099c0a6e0791ca717da96d97209e2763e763b 100644 --- a/guest/capture.h +++ b/guest/capture.h @@ -3,7 +3,9 @@ #include "cursor.h" typedef int (*sg_emit)(uint32_t type, const uint32_t *words, uint32_t count, const uint8_t *pixels, uint32_t stride); -int sg_windows_start(void); -int sg_windows_poll(sg_emit emit, int refresh); +int sg_capture_start(void); +int sg_windows_poll(sg_emit emit, int refresh, const uint8_t *pixels, uint32_t width, uint32_t height); +int sg_desktop_capture(uint8_t *pixels, uint32_t width, uint32_t height, int refresh); +void sg_capture_stop(void); void sg_windows_stop(void); #endif diff --git a/guest/windows.c b/guest/windows.c index a35a5b7d516f232fda1563777502409e032512e3..bc691b67a8e502da81c97c8093a16a13ab7f3bf5 100644 --- a/guest/windows.c +++ b/guest/windows.c @@ -194,7 +194,7 @@ static void stream(DWORD session) { if (capturing != request[15]) { capturing = request[15]; initial = 1; last_cursor = NULL; last_flags = ~0u; - if (!capturing) { sg_windows_stop(); windows_active = 0; } + if (!capturing) { sg_capture_stop(); windows_active = 0; } } } else { int enabled = request[15] && modern_windows; @@ -222,7 +222,7 @@ static void stream(DWORD session) { if (!active || !GetCursorInfo(&cursor)) { if (!suspended && !packet(SG_SUSPEND, 0, NULL, 0)) goto done; suspended = 1; initial = 1; last_cursor = NULL; last_flags = ~0u; - if (windows_active) sg_windows_stop(); + sg_capture_stop(); windows_active = 0; Sleep(100); continue; } @@ -270,14 +270,20 @@ static void stream(DWORD session) { if (!before || !dib) goto done; previous = SelectObject(dc, dib); width = w; height = h; initial = 1; } - if (!BitBlt(dc, 0, 0, (int)width, (int)height, screen, 0, 0, SRCCOPY | CAPTUREBLT)) { + int captured = modern_windows ? sg_desktop_capture(pixels, width, height, initial) : -1; + if (!captured) { + if (initial) continue; + goto window_capture; + } + if (captured < 0 && !BitBlt(dc, 0, 0, (int)width, (int)height, screen, 0, 0, SRCCOPY | CAPTUREBLT)) { if (!suspended && !packet(SG_SUSPEND, 0, NULL, 0)) goto done; suspended = 1; initial = 1; Sleep(100); continue; } - GdiFlush(); - QueryPerformanceCounter(&scanned); + if (captured < 0) GdiFlush(); + QueryPerformanceCounter(&now); ++stats[SG_CAPTURE_STATS_FRAMES]; - stats[SG_CAPTURE_STATS_CAPTURE_US] += microseconds(scanned.QuadPart - started.QuadPart, frequency.QuadPart); + stats[SG_CAPTURE_STATS_CAPTURE_US] += microseconds(now.QuadPart - started.QuadPart, frequency.QuadPart); + scanned = now; if (initial) { rect.x = 0; rect.y = 0; rect.width = width; rect.height = height; } else if (!sg_difference(before, pixels, width, height, &rect)) rect.width = rect.height = 0; QueryPerformanceCounter(&written); @@ -290,25 +296,26 @@ static void stream(DWORD session) { size_t offset = ((size_t)y * width + rect.x) * 4, i; for (i = 0; i < rect.width * 4; ++i) before[offset + i] = pixels[offset + i]; } +window_capture: if (windows_enabled) { if (!windows_active) { - windows_active = sg_windows_start(); + windows_active = sg_capture_start(); if (!windows_active) { uint32_t disabled[SG_WINDOWS_WORDS] = {0}; if (!emit(SG_WINDOWS, disabled, SG_WINDOWS_WORDS, NULL, 0)) goto done; windows_enabled = 0; } } - if (windows_active && !sg_windows_poll(emit, initial)) goto done; + if (windows_active && !sg_windows_poll(emit, initial, pixels, width, height)) goto done; } QueryPerformanceCounter(&now); - stats[SG_CAPTURE_STATS_WRITE_US] += microseconds(now.QuadPart - written.QuadPart, frequency.QuadPart); + if (captured) stats[SG_CAPTURE_STATS_WRITE_US] += microseconds(now.QuadPart - written.QuadPart, frequency.QuadPart); initial = 0; suspended = 0; } Sleep(1); } done: - sg_windows_stop(); + sg_capture_stop(); if (begin_period && end_period) end_period(1); if (winmm) FreeLibrary(winmm); CancelIo(output); @@ -337,6 +344,7 @@ void mainCRTStartup(void) { /* This optional user32 export supplies ANI step timing; static capture uses only documented APIs. */ cursor_frame = (cursor_frame_fn)symbol; for (;;) { + sg_capture_stop(); if (session != WTSGetActiveConsoleSessionId()) { Sleep(1000); continue; } output = CreateFileW(L"\\\\.\\Global\\" SG_CHANNEL, GENERIC_READ | GENERIC_WRITE, 0, NULL, OPEN_EXISTING, FILE_FLAG_OVERLAPPED, NULL); if (output != INVALID_HANDLE_VALUE) { stream(session); CloseHandle(output); } -- 2.54.0