Commit 2f1eb7876ce087844fe59c21ca41f1eb139de473 | home-infra

author	clover caruso <git@paperclover.net> 2026-10-06 22:08:37-07:00
committer	clover caruso <git@paperclover.net> 2026-10-06 23:11:29-07:00
log	2f1eb7876ce087844fe59c21ca41f1eb139de473
tree	8fcadaab87180f3a3c8b9a5bbd47d0c6a7f0f1be
parent	9e9eefa9e34d0067f716831ef397d23ecae5c895
signature	Signed by SSH key SHA256:52mNGHRsVFBDED9IAX5pe+LRWUefqTbxEReunq21QvU (~clover)

Fix Windows surface capture and accelerate desktop streaming

Use cursor-free monitor capture on modern Windows, drain to the latest frames, and retain the GDI compatibility fallback. Align UWP window content extents, keep shell popup stacking correct, and close rejected capture sessions. Initialize shell surfaces on bounded workers, preserve covered background pixels, and verify caption hit regions before optimistic dragging.

Validated fresh Calculator and Store launches, Start/taskbar stacking, resizing, dragging, native cursor reconnects, and idle capture on Windows 10/11. Animated browser video reached 37–44 FPS with no dropped frames; inference sustained 42.9 tokens/sec. x86/x64/ARM64 imports and subsystem checks plus cursor ASan/UBSan tests pass.

Assisted-by: gpt-6.1-sol

4 files changed, 301 insertions(+), 67 deletions(-)

guest

guest/DESIGN.md+4-4

...	...	@@ -1,10 +1,10 @@
1	1	# Guest display stream
2	2
3		The executable is freestanding C99. `cursor.c` contains pixel conversion and dirty-rectangle detection; `windows.c` owns desktop capture and transport; `capture.c` implements optional window surfaces. Zig supplies the cross compiler and Windows headers, with no Zig or C runtime in the executable. x86 and x64 target Windows 7; ARM64 targets Windows 10. PE subsystem versions and imported DLLs are checked by `python3 guest/test.py`. Runtime validation uses Windows 10 and 11; Windows 7 runtime behavior is unverified.
	3	The executable is freestanding C99. `cursor.c` contains pixel conversion and dirty-rectangle detection; `windows.c` owns desktop capture and transport; `capture.c` implements modern desktop capture and optional window surfaces. Zig supplies the cross compiler and Windows headers, with no Zig or C runtime in the executable. x86 and x64 target Windows 7; ARM64 targets Windows 10. PE subsystem versions and imported DLLs are checked by `python3 guest/test.py`. Runtime validation uses Windows 10 and 11; Windows 7 runtime behavior is unverified.
4	4
5	5	Build with `python3 guest/build.py --arch x86_64` (also `x86` and `aarch64`). In an elevated PowerShell session inside the logged-in administrator's desktop, run `./install.ps1 -Binary ./build/snowglobe-guest-x86_64.exe`. The installer uses Task Scheduler COM APIs available to PowerShell 2, installs into protected Program Files, and starts an elevated task at that user's logon. `-Remove` removes that task and executable. Elevation is required by the VirtIO serial driver's device ACL; capture must run in the interactive session, not session 0. The guest needs a compatible VirtIO serial driver and the channel named in `protocol.json`. A standard-user launcher requires a privileged handle broker; it is not part of this first backend.
6	6
7		`BitBlt` copies the primary desktop into a CPU DIB without drawing the cursor. RGB comparisons ignore the unused fourth byte. Unchanged pixels produce no screen packets; changed pixels produce one bounding rectangle. Cursor polling runs every millisecond; desktop capture targets 60 Hz. The host's existing encoder supplies WebRTC video. Secure desktops, RDP sessions that differ from the physical console, and capture failures return the viewer to QEMU capture.
	7	Windows 10 build 19041 and later use Windows Graphics Capture for the primary monitor with cursor capture disabled. Older or unsupported systems use `BitBlt` into a CPU DIB without drawing the cursor. Only the latest available capture frames are read back; an unchanged desktop needs no readback. RGB comparisons ignore the unused fourth byte. Unchanged pixels produce no screen packets; changed pixels produce one bounding rectangle. Cursor polling runs every millisecond; desktop capture targets 60 Hz. The host's existing encoder supplies WebRTC video. Secure desktops, RDP sessions that differ from the physical console, and capture failures return the viewer to QEMU capture.
8	8
9	9	`GetCursorInfo` supplies the shape, visibility, and observed position. `GetIconInfo` and black/white `DrawIconEx` renders recover RGBA and binary XOR planes, including colored XOR. Nonbinary color XOR and unsupported cursor sizes explicitly fall back to the browser's default cursor, preserving video. Animation timing uses the optional `GetCursorFrameInfo` user32 export, resolved by name. Missing or unusable animation metadata falls back to a static frame. Animation starts a browser-local clock; Windows does not expose the current animation phase through this interface. Ordinary shapes use the browser cursor; XOR shapes use local layers with difference blending. Neither follows streamed positions.
10	10
...	...	@@ -29,9 +29,9 @@ The host broker validates a running VM's libvirt-owned channel path. One guest r
29	29
30	30	## Separate windows experiment
31	31
32		Windows 10 build 19041 and later can enable separate windows from the viewer’s View menu. Windows Graphics Capture supplies full unoccluded surfaces with cursor capture disabled. Its dependencies are loaded dynamically, preserving the Windows 7 import boundary. Desktop and taskbar surfaces use `PrintWindow` because Windows Graphics Capture rejects those shell HWNDs. Changed rectangles update a single host atlas and one existing video encoder. Metadata describes the atlas and stacking; the browser composes it and moves cached surfaces immediately during native caption drags. Atlas geometry is matched to the decoded RTP frame.
	32	Windows 10 build 19041 and later can enable separate windows from the viewer’s View menu. Windows Graphics Capture supplies full unoccluded surfaces with cursor capture disabled. Its dependencies are loaded dynamically, preserving the Windows 7 import boundary. Desktop and taskbar surfaces initialize with a `PrintWindow` worker because Windows Graphics Capture rejects those shell HWNDs. Subsequent updates copy visible desktop spans, retaining covered pixels for local window dragging. Shell surfaces initialize before window capture sessions start, keeping capture indicators out of their initial pixels. Changed rectangles update a single host atlas and one existing video encoder. Metadata describes the atlas and stacking; the browser composes it and moves cached surfaces immediately during native caption drags. Atlas geometry is matched to the decoded RTP frame.
33	33
34		The experiment is off by default. Unsupported, layered, or oversized window sets fall back to desktop video. Secure desktops and session changes suspend guest capture. `PrintWindow` is synchronous; an unresponsive shell can interrupt the guest stream and trigger the host timeout. Protected content and custom caption behavior remain subject to the application’s capture support.
	34	The experiment is off by default. Unsupported, layered, translucent, shell-popup, or oversized window sets close all window sessions and fall back to desktop video. Frame-pool resizing follows captured content dimensions; a window whose first or resized surface never arrives also falls back. Secure desktops and session changes suspend guest capture. Shell painting runs off the capture thread and falls back after 500 ms; its bounded drawing resources remain owned until the worker exits. Windows capture indicators remain subject to the OS; fallback disposes the sessions instead of leaving indicators attached to inactive windows. Protected content and custom caption behavior remain subject to the application’s capture support.
35	35
36	36	## Later backends
37	37

guest/capture.c+275-51

...	...	@@ -37,22 +37,35 @@ static const GUID dxgi_iid = {0x54ec77fa,0x1377,0x44e6,{0x8c,0x32,0x88,0xfd,0x5f
37	37	static const GUID access_iid = {0xa9b3d012,0x3df2,0x4ee3,{0xb8,0xd1,0x86,0x95,0xf4,0x57,0xd3,0xc1}};
38	38	static const GUID texture_iid = {0x6f15aaf2,0xd208,0x4e89,{0x9a,0xb4,0x48,0x95,0x35,0xd3,0x4f,0x9c}};
39	39
	40	struct shell_paint {
	41	HWND hwnd;
	42	HANDLE thread;
	43	struct shell_paint *next;
	44	HDC dc;
	45	HBITMAP dib, selected;
	46	uint8_t *bits;
	47	size_t pixels;
	48	DWORD started;
	49	};
	50
40	51	struct window {
41	52	HWND hwnd;
42	53	uint32_t id, width, height;
43	54	struct size pool_size;
44	55	void pool, session;
45	56	ID3D11Texture2D *staging;
46		uint8_t before, bits;
47		HDC dc;
48		HBITMAP dib, selected;
	57	uint8_t *before;
	58	struct shell_paint *paint;
49	59	int initial, seen, shell;
	60	DWORD waiting;
50	61	};
51		static struct window windows[SG_MAX_WINDOWS];
	62	static struct window windows[SG_MAX_WINDOWS], desktop;
	63	static RECT bounds[SG_MAX_WINDOWS], capture_bounds[SG_MAX_WINDOWS];
52	64	static uint32_t layout[SG_WINDOWS_WORDS + SG_MAX_WINDOWS * SG_WINDOW_WORDS];
53	65	static uint32_t prior[SG_WINDOWS_WORDS + SG_MAX_WINDOWS * SG_WINDOW_WORDS];
54	66	static uint32_t count, next_id;
55		static size_t allocated;
	67	static size_t allocated, painted;
	68	static struct shell_paint *retired_paints;
56	69	static int invalid;
57	70	static HANDLE heap;
58	71	/* Capture workers outlive session disposal; retain their DLLs until process exit. */
...	...	@@ -65,7 +78,7 @@ static delete_string_fn delete_string;
65	78	static factory_fn get_factory;
66	79	static uninitialize_fn uninitialize;
67	80	static attribute_fn attribute;
68		static DWORD retry;
	81	static DWORD retry, desktop_retry;
69	82
70	83	static void release(void *object) {
71	84	if (object) METHOD(object, 2, release_fn)(object);
...	...	@@ -78,13 +91,41 @@ static void close_object(void *object) {
78	91	}
79	92	release(object);
80	93	}
	94	static void free_paint(struct shell_paint *paint) {
	95	if (paint->thread) CloseHandle(paint->thread);
	96	if (paint->selected) SelectObject(paint->dc, paint->selected);
	97	if (paint->dib) DeleteObject(paint->dib);
	98	if (paint->dc) DeleteDC(paint->dc);
	99	painted -= paint->pixels;
	100	HeapFree(heap, 0, paint);
	101	}
	102	static DWORD WINAPI paint_shell(void *data) {
	103	struct shell_paint *paint = data;
	104	BOOL success = PrintWindow(paint->hwnd, paint->dc, 2);
	105	GdiFlush();
	106	return success;
	107	}
	108	static void reap_paints(void) {
	109	struct shell_paint **link = &retired_paints;
	110	while (*link) {
	111	struct shell_paint paint = link;
	112	if (WaitForSingleObject(paint->thread, 0) == WAIT_OBJECT_0) {
	113	*link = paint->next; free_paint(paint);
	114	} else link = &paint->next;
	115	}
	116	}
	117	static void detach_paint(struct window *window) {
	118	struct shell_paint *paint = window->paint;
	119	if (!paint) return;
	120	window->paint = NULL;
	121	if (!paint->thread \|\| WaitForSingleObject(paint->thread, 0) == WAIT_OBJECT_0) free_paint(paint);
	122	else { paint->next = retired_paints; retired_paints = paint; }
	123	}
	124
81	125	static void drop(struct window *window) {
82	126	close_object(window->session); close_object(window->pool);
83	127	if (window->staging) ID3D11Texture2D_Release(window->staging);
84		if (window->selected) SelectObject(window->dc, window->selected);
85		if (window->dib) DeleteObject(window->dib);
86		if (window->dc) DeleteDC(window->dc);
87		window->dc = NULL; window->dib = window->selected = NULL; window->bits = NULL;
	128	detach_paint(window);
88	129	if (window->before) { allocated -= (size_t)window->width * window->height; HeapFree(heap, 0, window->before); }
89	130	window->hwnd = NULL; window->session = window->pool = NULL;
90	131	window->staging = NULL; window->before = NULL;
...	...	@@ -103,26 +144,33 @@ static int factory(PCWSTR name, const GUID iid, void *result) {
103	144	}
104	145
105	146	void sg_windows_stop(void) {
	147	reap_paints();
106	148	uint32_t i;
107	149	for (i = 0; i < SG_MAX_WINDOWS; ++i) drop(windows + i);
	150	count = 0;
	151	retry = 0;
	152	for (i = 0; i < sizeof(prior) / sizeof(prior[0]); ++i) prior[i] = 0;
	153	}
	154
	155	void sg_capture_stop(void) {
	156	sg_windows_stop();
	157	drop(&desktop);
	158	desktop_retry = 0;
108	159	release(pool_factory); release(interop); release(capture_device);
109	160	if (context) ID3D11DeviceContext_Release(context);
110	161	if (device) ID3D11Device_Release(device);
111	162	pool_factory = interop = capture_device = NULL; context = NULL; device = NULL;
112	163	if (uninitialize) uninitialize();
113	164	uninitialize = NULL;
114		count = 0;
115		retry = 0;
116		for (i = 0; i < sizeof(prior) / sizeof(prior[0]); ++i) prior[i] = 0;
117	165	}
118	166
119		int sg_windows_start(void) {
	167	int sg_capture_start(void) {
120	168	initialize_fn initialize;
121	169	typedef HRESULT (WINAPI create_fn)(IDXGIAdapter , D3D_DRIVER_TYPE, HMODULE, UINT, const D3D_FEATURE_LEVEL , UINT, UINT, ID3D11Device , D3D_FEATURE_LEVEL , ID3D11DeviceContext **);
122	170	create_fn create;
123	171	device_fn wrap;
124	172	IDXGIDevice *dxgi = NULL;
125		sg_windows_stop();
	173	if (device) return 1;
126	174	heap = GetProcessHeap();
127	175	if (!runtime) runtime = LoadLibraryW(L"combase.dll");
128	176	if (!d3d) d3d = LoadLibraryW(L"d3d11.dll");
...	...	@@ -148,7 +196,7 @@ int sg_windows_start(void) {
148	196	if (FAILED(status)) goto failed;
149	197	return 1;
150	198	failed:
151		sg_windows_stop();
	199	sg_capture_stop();
152	200	return 0;
153	201	}
154	202
...	...	@@ -173,11 +221,20 @@ static BOOL CALLBACK enumerate(HWND hwnd, LPARAM unused) {
173	221	GetClassNameW(hwnd, class_name, 32);
174	222	windows[i].shell = hwnd == GetShellWindow() \|\|
175	223	!lstrcmpW(class_name, L"Shell_TrayWnd") \|\| !lstrcmpW(class_name, L"Shell_SecondaryTrayWnd") \|\| !lstrcmpW(class_name, L"WorkerW");
176		if (FAILED(attribute(hwnd, 9, &rect, sizeof(rect)))) GetWindowRect(hwnd, &rect);
	224	if ((!lstrcmpW(class_name, L"ApplicationFrameWindow") && !GetWindowTextLengthW(hwnd)) \|\| !lstrcmpW(class_name, L"Windows.UI.Core.CoreWindow") \|\| !lstrcmpW(class_name, L"Xaml_WindowedPopupClass") \|\| !lstrcmpW(class_name, L"XamlExplorerHostIslandWindow")) { invalid = 1; return FALSE; }
	225	if (!windows[i].shell && (GetWindowLongW(hwnd, GWL_STYLE) & WS_CAPTION) != WS_CAPTION) { invalid = 1; return FALSE; }
	226	capture_bounds[count] = rect;
	227	if (!windows[i].shell) attribute(hwnd, 9, &rect, sizeof(rect));
	228	if (windows[i].before && !windows[i].initial &&
	229	(windows[i].width != (uint32_t)(rect.right - rect.left) \|\| windows[i].height != (uint32_t)(rect.bottom - rect.top))) {
	230	windows[i].initial = 1; windows[i].waiting = GetTickCount();
	231	}
	232	bounds[count] = rect;
177	233	uint32_t record = layout + SG_WINDOWS_WORDS + count++ SG_WINDOW_WORDS;
178	234	record[SG_WINDOW_ID] = windows[i].id;
179	235	record[SG_WINDOW_X] = (uint32_t)rect.left; record[SG_WINDOW_Y] = (uint32_t)rect.top;
180		record[SG_WINDOW_WIDTH] = (uint32_t)(rect.right - rect.left); record[SG_WINDOW_HEIGHT] = (uint32_t)(rect.bottom - rect.top);
	236	record[SG_WINDOW_WIDTH] = windows[i].before && !windows[i].shell ? windows[i].width : (uint32_t)(rect.right - rect.left);
	237	record[SG_WINDOW_HEIGHT] = windows[i].before && !windows[i].shell ? windows[i].height : (uint32_t)(rect.bottom - rect.top);
181	238	for (i = SG_WINDOW_CAPTION_LEFT; i <= SG_WINDOW_CAPTION_BOTTOM; ++i) record[i] = 0;
182	239	if ((GetWindowLongW(hwnd, GWL_STYLE) & WS_CAPTION) == WS_CAPTION) {
183	240	TITLEBARINFO title = {0};
...	...	@@ -187,6 +244,16 @@ static BOOL CALLBACK enumerate(HWND hwnd, LPARAM unused) {
187	244	record[SG_WINDOW_CAPTION_TOP] = (uint32_t)title.rcTitleBar.top;
188	245	record[SG_WINDOW_CAPTION_RIGHT] = (uint32_t)(title.rcTitleBar.right - GetSystemMetrics(SM_CXSIZE) * 3);
189	246	record[SG_WINDOW_CAPTION_BOTTOM] = (uint32_t)title.rcTitleBar.bottom;
	247	LONG left = (LONG)record[SG_WINDOW_CAPTION_LEFT], right = (LONG)record[SG_WINDOW_CAPTION_RIGHT];
	248	LONG y = (title.rcTitleBar.top + title.rcTitleBar.bottom) / 2;
	249	for (int sample = 0; sample < 3; ++sample) {
	250	LONG x = left + 1 + sample * (right - left - 2) / 2;
	251	DWORD_PTR hit;
	252	if (right <= left \|\| !SendMessageTimeoutW(hwnd, WM_NCHITTEST, 0, MAKELPARAM(x, y), SMTO_ABORTIFHUNG \| SMTO_BLOCK, 1, &hit) \|\| hit != HTCAPTION) {
	253	for (i = SG_WINDOW_CAPTION_LEFT; i <= SG_WINDOW_CAPTION_BOTTOM; ++i) record[i] = 0;
	254	break;
	255	}
	256	}
190	257	}
191	258	}
192	259	return TRUE;
...	...	@@ -196,46 +263,147 @@ static int start_window(struct window *window) {
196	263	void item = NULL, cursor = NULL;
197	264	struct size size;
198	265	int success = 0;
199		if (FAILED(METHOD(interop, 3, window_fn)(interop, window->hwnd, &item_iid, &item))) goto done;
	266	HWND target = window->hwnd ? window->hwnd : (HWND)MonitorFromPoint((POINT){0, 0}, MONITOR_DEFAULTTOPRIMARY);
	267	if (FAILED(METHOD(interop, window->hwnd ? 3 : 4, window_fn)(interop, target, &item_iid, &item))) goto done;
200	268	if (FAILED(METHOD(item, 7, size_fn)(item, &size)) \|\| size.width <= 0 \|\| size.height <= 0 \|\| size.width > (INT32)SG_MAX_DIMENSION \|\| size.height > (INT32)SG_MAX_DIMENSION) goto done;
201	269	if (FAILED(METHOD(pool_factory, 6, pool_fn)(pool_factory, capture_device, DXGI_FORMAT_B8G8R8A8_UNORM, 2, size, &window->pool))) goto done;
202	270	if (FAILED(METHOD(window->pool, 10, session_fn)(window->pool, item, &window->session))) goto done;
203	271	if (FAILED(METHOD(window->session, 0, query_fn)(window->session, &cursor_iid, &cursor))) goto done;
204	272	if (FAILED(METHOD(cursor, 7, bool_fn)(cursor, FALSE)) \|\| FAILED(METHOD(window->session, 6, close_fn)(window->session))) goto done;
205	273	window->pool_size = size;
	274	window->waiting = GetTickCount();
206	275	success = 1;
207	276	done:
208	277	release(cursor); release(item);
209	278	return success;
210	279	}
211	280
212		int sg_windows_poll(sg_emit emit, int refresh) {
	281	static int latest_frame(struct window window, void *frame) {
	282	void *next = NULL;
	283	for (int i = 0; i < 2; ++i) {
	284	if (FAILED(METHOD(window->pool, 7, object_fn)(window->pool, &next))) return 0;
	285	if (!next) return 1;
	286	close_object(*frame);
	287	*frame = next;
	288	}
	289	return 1;
	290	}
	291
	292	int sg_desktop_capture(uint8_t *pixels, uint32_t width, uint32_t height, int refresh) {
	293	reap_paints();
	294	void frame = NULL, surface = NULL, *access = NULL;
	295	ID3D11Texture2D *texture = NULL;
	296	D3D11_TEXTURE2D_DESC desc;
	297	D3D11_MAPPED_SUBRESOURCE mapped;
	298	struct size size;
	299	int result = -1;
	300	if (desktop_retry && (DWORD)(GetTickCount() - desktop_retry) < 1000) return -1;
	301	if (!device && !sg_capture_start()) { desktop_retry = GetTickCount(); return -1; }
	302	if (!desktop.pool && !start_window(&desktop)) goto done;
	303	if (!latest_frame(&desktop, &frame)) goto done;
	304	result = 0;
	305	if (!frame) {
	306	if (refresh && desktop.staging && desktop.width == width && desktop.height == height) goto readback;
	307	if ((DWORD)(GetTickCount() - desktop.waiting) > 500 && (!desktop.staging \|\| desktop.width != width \|\| desktop.height != height)) result = -1;
	308	goto done;
	309	}
	310	if (FAILED(METHOD(frame, 8, size_fn)(frame, &size))) { result = -1; goto done; }
	311	if (size.width != (INT32)width \|\| size.height != (INT32)height) { result = -1; goto done; }
	312	if (desktop.pool_size.width != size.width \|\| desktop.pool_size.height != size.height) {
	313	if (FAILED(METHOD(desktop.pool, 6, recreate_fn)(desktop.pool, capture_device, DXGI_FORMAT_B8G8R8A8_UNORM, 2, size))) { result = -1; goto done; }
	314	desktop.pool_size = size;
	315	goto done;
	316	}
	317	result = -1;
	318	if (FAILED(METHOD(frame, 6, object_fn)(frame, &surface)) \|\|
	319	FAILED(METHOD(surface, 0, query_fn)(surface, &access_iid, &access)) \|\|
	320	FAILED(METHOD(access, 3, query_fn)(access, &texture_iid, (void **)&texture))) goto done;
	321	ID3D11Texture2D_GetDesc(texture, &desc);
	322	if (desc.Width < width \|\| desc.Height < height) goto done;
	323	if (desktop.width != width \|\| desktop.height != height) {
	324	if (desktop.staging) ID3D11Texture2D_Release(desktop.staging);
	325	desktop.staging = NULL;
	326	desc.Width = width; desc.Height = height;
	327	desc.BindFlags = 0; desc.MiscFlags = 0; desc.Usage = D3D11_USAGE_STAGING; desc.CPUAccessFlags = D3D11_CPU_ACCESS_READ;
	328	if (FAILED(ID3D11Device_CreateTexture2D(device, &desc, NULL, &desktop.staging))) goto done;
	329	desktop.width = width; desktop.height = height;
	330	}
	331	D3D11_BOX box = {0, 0, 0, width, height, 1};
	332	ID3D11DeviceContext_CopySubresourceRegion(context, (ID3D11Resource )desktop.staging, 0, 0, 0, 0, (ID3D11Resource )texture, 0, &box);
	333	readback:
	334	result = -1;
	335	if (FAILED(ID3D11DeviceContext_Map(context, (ID3D11Resource *)desktop.staging, 0, D3D11_MAP_READ, 0, &mapped))) goto done;
	336	for (uint32_t y = 0; y < height; ++y) {
	337	uint8_t src = (uint8_t )mapped.pData + (size_t)y * mapped.RowPitch;
	338	uint8_t dst = pixels + (size_t)y width * 4;
	339	for (uint32_t x = 0; x < width * 4; ++x) dst[x] = src[x];
	340	}
	341	ID3D11DeviceContext_Unmap(context, (ID3D11Resource *)desktop.staging, 0);
	342	result = 1;
	343	done:
	344	if (texture) ID3D11Texture2D_Release(texture);
	345	release(access); release(surface); close_object(frame);
	346	if (result < 0) { drop(&desktop); desktop_retry = GetTickCount(); }
	347	else desktop_retry = 0;
	348	return result;
	349	}
	350
	351	static int atlas_fits(void) {
	352	uint32_t width = layout[SG_WINDOWS_WIDTH], height = 0, x = 0, row_height = 0, last_id = 0;
	353	for (uint32_t i = 0; i < count; ++i) {
	354	uint32_t w = layout[SG_WINDOWS_WORDS + i * SG_WINDOW_WORDS + SG_WINDOW_WIDTH];
	355	if (w > width) width = w;
	356	}
	357	for (uint32_t i = 0; i < count; ++i) {
	358	uint32_t *next = NULL;
	359	for (uint32_t n = 0; n < count; ++n) {
	360	uint32_t record = layout + SG_WINDOWS_WORDS + n SG_WINDOW_WORDS;
	361	if (record[SG_WINDOW_ID] > last_id && (!next \|\| record[SG_WINDOW_ID] < next[SG_WINDOW_ID])) next = record;
	362	}
	363	uint32_t w = next[SG_WINDOW_WIDTH], h = next[SG_WINDOW_HEIGHT];
	364	if (!w \|\| !h \|\| w > SG_MAX_DIMENSION \|\| h > SG_MAX_DIMENSION) return 0;
	365	if (x + w > width) { x = 0; height += row_height; row_height = 0; }
	366	x += w;
	367	if (h > row_height) row_height = h;
	368	last_id = next[SG_WINDOW_ID];
	369	}
	370	height += row_height;
	371	return width <= SG_MAX_DIMENSION && height <= SG_MAX_DIMENSION && (size_t)width * height <= SG_MAX_PIXELS;
	372	}
	373
	374	int sg_windows_poll(sg_emit emit, int refresh, const uint8_t *pixels, uint32_t width, uint32_t height) {
213	375	uint32_t i, n, words[SG_SURFACE_WORDS] = {0};
214	376	if (retry && (DWORD)(GetTickCount() - retry) < 1000 && !refresh) return 1;
215	377	retry = 0;
216	378	count = 0; invalid = 0;
217		layout[SG_WINDOWS_WIDTH] = (uint32_t)GetSystemMetrics(SM_CXSCREEN);
218		layout[SG_WINDOWS_HEIGHT] = (uint32_t)GetSystemMetrics(SM_CYSCREEN);
	379	layout[SG_WINDOWS_WIDTH] = width;
	380	layout[SG_WINDOWS_HEIGHT] = height;
219	381	for (i = 0; i < SG_MAX_WINDOWS; ++i) windows[i].seen = 0;
220		EnumWindows(enumerate, 0);
	382	HWND foreground = GetForegroundWindow();
	383	if (foreground && foreground != GetShellWindow() && foreground != FindWindowW(L"Shell_TrayWnd", NULL) &&
	384	(GetWindowLongW(foreground, GWL_STYLE) & WS_CAPTION) != WS_CAPTION) invalid = 1;
	385	if (!invalid) EnumWindows(enumerate, 0);
221	386	for (i = 0; i < SG_MAX_WINDOWS; ++i) if (windows[i].hwnd && !windows[i].seen) drop(windows + i);
	387	if (!invalid && !atlas_fits()) invalid = 1;
222	388	layout[SG_WINDOWS_COUNT] = invalid ? 0 : count;
223	389	n = SG_WINDOWS_WORDS + layout[SG_WINDOWS_COUNT] * SG_WINDOW_WORDS;
224	390	int changed = refresh;
225	391	for (i = 0; i < n; ++i) if (layout[i] != prior[i]) changed = 1;
226	392	if (changed && !emit(SG_WINDOWS, layout, n, NULL, 0)) return 0;
227	393	for (i = 0; i < n; ++i) prior[i] = layout[i];
228		if (invalid) return 1;
229		for (i = 0; i < SG_MAX_WINDOWS; ++i) {
230		struct window *window = windows + i;
	394	if (invalid) { sg_windows_stop(); retry = GetTickCount(); return 1; }
	395	for (i = 0; i < SG_MAX_WINDOWS * 2; ++i) {
	396	struct window *window = windows + i % SG_MAX_WINDOWS;
	397	if (window->shell != (i < SG_MAX_WINDOWS)) continue;
231	398	void frame = NULL, surface = NULL, *access = NULL;
232	399	ID3D11Texture2D *texture = NULL;
233	400	D3D11_TEXTURE2D_DESC desc;
	401	D3D11_BOX box;
234	402	D3D11_MAPPED_SUBRESOURCE mapped;
235	403	struct size size;
236	404	struct sg_rect rect;
237	405	uint32_t y;
238		int success = 1;
	406	int success = 1, mapped_texture = 0;
239	407	if (!window->hwnd) continue;
240	408	uint32_t *record = NULL;
241	409	for (n = 0; n < count; ++n) {
...	...	@@ -244,22 +412,24 @@ int sg_windows_poll(sg_emit emit, int refresh) {
244	412	}
245	413	if (!record) continue;
246	414	if (window->shell) {
247		RECT bounds;
248		if (!GetWindowRect(window->hwnd, &bounds)) goto frame_failed;
249		size.width = bounds.right - bounds.left; size.height = bounds.bottom - bounds.top;
	415	size.width = capture_bounds[n].right - capture_bounds[n].left;
	416	size.height = capture_bounds[n].bottom - capture_bounds[n].top;
250	417	mapped.pData = NULL;
251	418	} else {
252	419	if (!window->pool && !start_window(window)) goto frame_failed;
253		struct size target = {(INT32)record[SG_WINDOW_WIDTH], (INT32)record[SG_WINDOW_HEIGHT]};
254		if (window->pool_size.width != target.width \|\| window->pool_size.height != target.height) {
255		if (FAILED(METHOD(window->pool, 6, recreate_fn)(window->pool, capture_device, DXGI_FORMAT_B8G8R8A8_UNORM, 2, target))) goto frame_failed;
256		window->pool_size = target;
257		window->initial = 1;
	420	if (!latest_frame(window, &frame)) goto frame_failed;
	421	if (!frame) {
	422	if (window->initial && (DWORD)(GetTickCount() - window->waiting) > 500) goto frame_failed;
	423	continue;
258	424	}
259		if (FAILED(METHOD(window->pool, 7, object_fn)(window->pool, &frame))) goto frame_failed;
260		if (!frame) continue;
261	425	if (FAILED(METHOD(frame, 8, size_fn)(frame, &size))) goto frame_failed;
262		if (size.width != target.width \|\| size.height != target.height) goto frame_done;
	426	if (window->pool_size.width != size.width \|\| window->pool_size.height != size.height) {
	427	if (size.width <= 0 \|\| size.height <= 0 \|\| size.width > (INT32)SG_MAX_DIMENSION \|\| size.height > (INT32)SG_MAX_DIMENSION) goto frame_failed;
	428	if (FAILED(METHOD(window->pool, 6, recreate_fn)(window->pool, capture_device, DXGI_FORMAT_B8G8R8A8_UNORM, 2, size))) goto frame_failed;
	429	window->pool_size = size;
	430	window->initial = 1; window->waiting = GetTickCount();
	431	goto frame_done;
	432	}
263	433	if (FAILED(METHOD(frame, 6, object_fn)(frame, &surface)) \|\|
264	434	FAILED(METHOD(surface, 0, query_fn)(surface, &access_iid, &access)) \|\|
265	435	FAILED(METHOD(access, 3, query_fn)(access, &texture_iid, (void **)&texture))) goto frame_failed;
...	...	@@ -268,13 +438,26 @@ int sg_windows_poll(sg_emit emit, int refresh) {
268	438	if (texture) {
269	439	ID3D11Texture2D_GetDesc(texture, &desc);
270	440	if (desc.Width < (UINT)size.width \|\| desc.Height < (UINT)size.height) goto frame_failed;
	441	LONG left = 0, top = 0;
	442	LONG right = bounds[n].right - bounds[n].left, bottom = bounds[n].bottom - bounds[n].top;
	443	if (size.width == capture_bounds[n].right - capture_bounds[n].left && size.height == capture_bounds[n].bottom - capture_bounds[n].top) {
	444	left = bounds[n].left - capture_bounds[n].left; top = bounds[n].top - capture_bounds[n].top;
	445	right += left; bottom += top;
	446	} else if (size.width != right \|\| size.height != bottom) goto frame_failed;
	447	if (left < 0 \|\| top < 0 \|\| right > size.width \|\| bottom > size.height \|\| right <= left \|\| bottom <= top) goto frame_failed;
	448	box = (D3D11_BOX){(UINT)left, (UINT)top, 0, (UINT)right, (UINT)bottom, 1};
	449	size.width = right - left; size.height = bottom - top;
	450	}
	451	if (record[SG_WINDOW_WIDTH] != (uint32_t)size.width \|\| record[SG_WINDOW_HEIGHT] != (uint32_t)size.height) {
	452	record[SG_WINDOW_WIDTH] = (uint32_t)size.width; record[SG_WINDOW_HEIGHT] = (uint32_t)size.height;
	453	if (!atlas_fits()) goto frame_failed;
	454	if (!emit(SG_WINDOWS, layout, SG_WINDOWS_WORDS + count * SG_WINDOW_WORDS, NULL, 0)) { success = 0; goto frame_done; }
	455	for (n = 0; n < SG_WINDOWS_WORDS + count * SG_WINDOW_WORDS; ++n) prior[n] = layout[n];
271	456	}
272	457	if (window->width != (uint32_t)size.width \|\| window->height != (uint32_t)size.height) {
273	458	if (window->staging) ID3D11Texture2D_Release(window->staging);
274	459	window->staging = NULL;
275		if (window->selected) SelectObject(window->dc, window->selected);
276		if (window->dib) DeleteObject(window->dib);
277		window->dib = window->selected = NULL; window->bits = NULL;
	460	detach_paint(window);
278	461	if (window->before) { allocated -= (size_t)window->width * window->height; HeapFree(heap, 0, window->before); window->before = NULL; }
279	462	if (allocated + (size_t)size.width * size.height > SG_MAX_PIXELS) goto frame_failed;
280	463	window->width = (uint32_t)size.width; window->height = (uint32_t)size.height;
...	...	@@ -283,12 +466,21 @@ int sg_windows_poll(sg_emit emit, int refresh) {
283	466	allocated += (size_t)window->width * window->height;
284	467	if (window->shell) {
285	468	BITMAPINFO info = {0};
286		if (!window->dc) window->dc = CreateCompatibleDC(NULL);
	469	size_t pixels = (size_t)window->width * window->height;
	470	if (painted + pixels > SG_MAX_PIXELS) goto frame_failed;
	471	window->paint = HeapAlloc(heap, HEAP_ZERO_MEMORY, sizeof(struct shell_paint));
	472	if (!window->paint) goto frame_failed;
	473	struct shell_paint *paint = window->paint;
	474	paint->pixels = pixels; painted += pixels; paint->hwnd = window->hwnd;
	475	paint->dc = CreateCompatibleDC(NULL);
287	476	info.bmiHeader.biSize = sizeof(BITMAPINFOHEADER); info.bmiHeader.biWidth = size.width;
288	477	info.bmiHeader.biHeight = -size.height; info.bmiHeader.biPlanes = 1; info.bmiHeader.biBitCount = 32;
289		window->dib = CreateDIBSection(window->dc, &info, DIB_RGB_COLORS, (void **)&window->bits, NULL, 0);
290		if (!window->dc \|\| !window->dib) goto frame_failed;
291		window->selected = SelectObject(window->dc, window->dib);
	478	paint->dib = CreateDIBSection(paint->dc, &info, DIB_RGB_COLORS, (void **)&paint->bits, NULL, 0);
	479	if (!paint->dc \|\| !paint->dib) goto frame_failed;
	480	paint->selected = SelectObject(paint->dc, paint->dib);
	481	paint->started = GetTickCount();
	482	paint->thread = CreateThread(NULL, 0, paint_shell, paint, 0, NULL);
	483	if (!paint->thread) goto frame_failed;
292	484	} else {
293	485	desc.Width = window->width; desc.Height = window->height;
294	486	desc.BindFlags = 0; desc.MiscFlags = 0; desc.Usage = D3D11_USAGE_STAGING; desc.CPUAccessFlags = D3D11_CPU_ACCESS_READ;
...	...	@@ -297,14 +489,45 @@ int sg_windows_poll(sg_emit emit, int refresh) {
297	489	window->initial = 1;
298	490	}
299	491	if (window->shell) {
300		DWORD_PTR result;
301		if (!SendMessageTimeoutW(window->hwnd, WM_NULL, 0, 0, SMTO_ABORTIFHUNG \| SMTO_BLOCK, 20, &result) \|\| !PrintWindow(window->hwnd, window->dc, 2)) goto frame_failed;
302		GdiFlush();
303		mapped.pData = window->bits; mapped.RowPitch = window->width * 4;
	492	if (WaitForSingleObject(window->paint->thread, 0) != WAIT_OBJECT_0) {
	493	if ((DWORD)(GetTickCount() - window->paint->started) >= 500) goto frame_failed;
	494	return 1;
	495	}
	496	DWORD painted_ok;
	497	if (!GetExitCodeThread(window->paint->thread, &painted_ok) \|\| !painted_ok) goto frame_failed;
	498	for (y = 0; y < window->height; ++y) {
	499	LONG screen_y = (LONG)record[SG_WINDOW_Y] + (LONG)y;
	500	if (screen_y < 0 \|\| screen_y >= (LONG)layout[SG_WINDOWS_HEIGHT]) continue;
	501	LONG x = (LONG)record[SG_WINDOW_X], right = x + (LONG)window->width;
	502	if (x < 0) x = 0;
	503	if (right > (LONG)layout[SG_WINDOWS_WIDTH]) right = (LONG)layout[SG_WINDOWS_WIDTH];
	504	while (x < right) {
	505	LONG stop = right;
	506	for (n = 0; n < count; ++n) {
	507	uint32_t other = layout + SG_WINDOWS_WORDS + n SG_WINDOW_WORDS;
	508	if (other[SG_WINDOW_ID] == window->id) break;
	509	LONG left = capture_bounds[n].left - 2, top = capture_bounds[n].top - 2;
	510	LONG edge = capture_bounds[n].right + 2;
	511	if (screen_y < top \|\| screen_y >= capture_bounds[n].bottom + 2) continue;
	512	if (left <= x && edge > x) { x = edge; stop = x; break; }
	513	if (left > x && left < stop) stop = left;
	514	}
	515	if (x >= stop) continue;
	516	uint8_t dst = window->paint->bits + ((size_t)y window->width + (x - (LONG)record[SG_WINDOW_X])) * 4;
	517	const uint8_t src = pixels + ((size_t)screen_y layout[SG_WINDOWS_WIDTH] + x) * 4;
	518	for (LONG byte = 0; byte < (stop - x) * 4; ++byte) dst[byte] = src[byte];
	519	x = stop;
	520	}
	521	}
	522	mapped.pData = window->paint->bits; mapped.RowPitch = window->width * 4;
304	523	} else {
305		D3D11_BOX box = {0, 0, 0, window->width, window->height, 1};
306	524	ID3D11DeviceContext_CopySubresourceRegion(context, (ID3D11Resource )window->staging, 0, 0, 0, 0, (ID3D11Resource )texture, 0, &box);
307	525	if (FAILED(ID3D11DeviceContext_Map(context, (ID3D11Resource *)window->staging, 0, D3D11_MAP_READ, 0, &mapped))) goto frame_failed;
	526	mapped_texture = 1;
	527	for (y = 8; y + 8 < window->height; ++y) {
	528	const uint8_t row = (const uint8_t )mapped.pData + (size_t)y * mapped.RowPitch;
	529	for (uint32_t x = 8; x + 8 < window->width; ++x) if (row[x * 4 + 3] != 255) goto frame_failed;
	530	}
308	531	}
309	532	rect.x = window->width; rect.y = window->height; rect.width = rect.height = 0;
310	533	if (refresh \|\| window->initial) { rect.x = rect.y = 0; rect.width = window->width; rect.height = window->height; }
...	...	@@ -330,17 +553,18 @@ int sg_windows_poll(sg_emit emit, int refresh) {
330	553	}
331	554	window->initial = 0;
332	555	}
333		if (texture) ID3D11DeviceContext_Unmap(context, (ID3D11Resource *)window->staging, 0);
334	556	goto frame_done;
335	557	frame_failed:
336	558	invalid = 1;
337	559	frame_done:
	560	if (mapped_texture) ID3D11DeviceContext_Unmap(context, (ID3D11Resource *)window->staging, 0);
338	561	if (texture) ID3D11Texture2D_Release(texture);
339	562	release(access); release(surface); close_object(frame);
340	563	if (!success) return 0;
341	564	if (invalid) { drop(window); break; }
342	565	}
343	566	if (invalid) {
	567	sg_windows_stop();
344	568	retry = GetTickCount();
345	569	uint32_t disabled[SG_WINDOWS_WORDS] = {layout[SG_WINDOWS_WIDTH], layout[SG_WINDOWS_HEIGHT], 0};
346	570	if (!emit(SG_WINDOWS, disabled, SG_WINDOWS_WORDS, NULL, 0)) return 0;

guest/capture.h+4-2

...	...	@@ -3,7 +3,9 @@
3	3	#include "cursor.h"
4	4
5	5	typedef int (sg_emit)(uint32_t type, const uint32_t words, uint32_t count, const uint8_t *pixels, uint32_t stride);
6		int sg_windows_start(void);
7		int sg_windows_poll(sg_emit emit, int refresh);
	6	int sg_capture_start(void);
	7	int sg_windows_poll(sg_emit emit, int refresh, const uint8_t *pixels, uint32_t width, uint32_t height);
	8	int sg_desktop_capture(uint8_t *pixels, uint32_t width, uint32_t height, int refresh);
	9	void sg_capture_stop(void);
8	10	void sg_windows_stop(void);
9	11	#endif

guest/windows.c+18-10

...	...	@@ -194,7 +194,7 @@ static void stream(DWORD session) {
194	194	if (capturing != request[15]) {
195	195	capturing = request[15];
196	196	initial = 1; last_cursor = NULL; last_flags = ~0u;
197		if (!capturing) { sg_windows_stop(); windows_active = 0; }
	197	if (!capturing) { sg_capture_stop(); windows_active = 0; }
198	198	}
199	199	} else {
200	200	int enabled = request[15] && modern_windows;
...	...	@@ -222,7 +222,7 @@ static void stream(DWORD session) {
222	222	if (!active \|\| !GetCursorInfo(&cursor)) {
223	223	if (!suspended && !packet(SG_SUSPEND, 0, NULL, 0)) goto done;
224	224	suspended = 1; initial = 1; last_cursor = NULL; last_flags = ~0u;
225		if (windows_active) sg_windows_stop();
	225	sg_capture_stop();
226	226	windows_active = 0;
227	227	Sleep(100); continue;
228	228	}
...	...	@@ -270,14 +270,20 @@ static void stream(DWORD session) {
270	270	if (!before \|\| !dib) goto done;
271	271	previous = SelectObject(dc, dib); width = w; height = h; initial = 1;
272	272	}
273		if (!BitBlt(dc, 0, 0, (int)width, (int)height, screen, 0, 0, SRCCOPY \| CAPTUREBLT)) {
	273	int captured = modern_windows ? sg_desktop_capture(pixels, width, height, initial) : -1;
	274	if (!captured) {
	275	if (initial) continue;
	276	goto window_capture;
	277	}
	278	if (captured < 0 && !BitBlt(dc, 0, 0, (int)width, (int)height, screen, 0, 0, SRCCOPY \| CAPTUREBLT)) {
274	279	if (!suspended && !packet(SG_SUSPEND, 0, NULL, 0)) goto done;
275	280	suspended = 1; initial = 1; Sleep(100); continue;
276	281	}
277		GdiFlush();
278		QueryPerformanceCounter(&scanned);
	282	if (captured < 0) GdiFlush();
	283	QueryPerformanceCounter(&now);
279	284	++stats[SG_CAPTURE_STATS_FRAMES];
280		stats[SG_CAPTURE_STATS_CAPTURE_US] += microseconds(scanned.QuadPart - started.QuadPart, frequency.QuadPart);
	285	stats[SG_CAPTURE_STATS_CAPTURE_US] += microseconds(now.QuadPart - started.QuadPart, frequency.QuadPart);
	286	scanned = now;
281	287	if (initial) { rect.x = 0; rect.y = 0; rect.width = width; rect.height = height; }
282	288	else if (!sg_difference(before, pixels, width, height, &rect)) rect.width = rect.height = 0;
283	289	QueryPerformanceCounter(&written);
...	...	@@ -290,25 +296,26 @@ static void stream(DWORD session) {
290	296	size_t offset = ((size_t)y * width + rect.x) * 4, i;
291	297	for (i = 0; i < rect.width * 4; ++i) before[offset + i] = pixels[offset + i];
292	298	}
	299	window_capture:
293	300	if (windows_enabled) {
294	301	if (!windows_active) {
295		windows_active = sg_windows_start();
	302	windows_active = sg_capture_start();
296	303	if (!windows_active) {
297	304	uint32_t disabled[SG_WINDOWS_WORDS] = {0};
298	305	if (!emit(SG_WINDOWS, disabled, SG_WINDOWS_WORDS, NULL, 0)) goto done;
299	306	windows_enabled = 0;
300	307	}
301	308	}
302		if (windows_active && !sg_windows_poll(emit, initial)) goto done;
	309	if (windows_active && !sg_windows_poll(emit, initial, pixels, width, height)) goto done;
303	310	}
304	311	QueryPerformanceCounter(&now);
305		stats[SG_CAPTURE_STATS_WRITE_US] += microseconds(now.QuadPart - written.QuadPart, frequency.QuadPart);
	312	if (captured) stats[SG_CAPTURE_STATS_WRITE_US] += microseconds(now.QuadPart - written.QuadPart, frequency.QuadPart);
306	313	initial = 0; suspended = 0;
307	314	}
308	315	Sleep(1);
309	316	}
310	317	done:
311		sg_windows_stop();
	318	sg_capture_stop();
312	319	if (begin_period && end_period) end_period(1);
313	320	if (winmm) FreeLibrary(winmm);
314	321	CancelIo(output);
...	...	@@ -337,6 +344,7 @@ void mainCRTStartup(void) {
337	344	/* This optional user32 export supplies ANI step timing; static capture uses only documented APIs. */
338	345	cursor_frame = (cursor_frame_fn)symbol;
339	346	for (;;) {
	347	sg_capture_stop();
340	348	if (session != WTSGetActiveConsoleSessionId()) { Sleep(1000); continue; }
341	349	output = CreateFileW(L"\\\\.\\Global\\" SG_CHANNEL, GENERIC_READ \| GENERIC_WRITE, 0, NULL, OPEN_EXISTING, FILE_FLAG_OVERLAPPED, NULL);
342	350	if (output != INVALID_HANDLE_VALUE) { stream(session); CloseHandle(output); }