feat: snip a region into chat, and make links clickable #14

2026-10-04T07:13:32Z

adrianbonpin commented

2026-10-04 07:13:32 +00:00

Two changes: screenshot input gated on the model really reporting vision, and a fix for URLs that arrived as code.

Links

The screenshot showed three bare GitHub URLs rendered in a fenced block (dark panel, Copy button, monospace), so they could not be opened.

Root cause, verified against the Vicinae source:

markdown-model.cpp attaches only the table and strikethrough GFM extensions, so autolink is off and a bare URL is plain dead text.
Clickability comes from CMARK_NODE_LINK becoming an <a href>, which text-selection-controller.cpp opens on release via linkAt then openLink.
So [label](url) is the only clickable form, and a URL inside a fence is the worst presentation available.
promoteStandaloneCodeSpans was itself promoting a lone URL in backticks into a fence, making this worse.

Fix: restoreUrlLinks recovers three narrow shapes (a fence whose body is all URLs, a lone URL span, a line that is only a URL) into markdown links. Link restoration runs before promotion so a recovered URL is never re-fenced. Anything else is copied through, so a curl invocation containing an address is never rewritten. The default prompt now also says never to fence a URL.

Screenshots

capture.ts shells out to an auto-detected backend: grimblast, grim + slurp, spectacle, maim, scrot, flameshot, or screencapture on macOS. There is no screenshot call in @vicinae/api, and Clipboard moves no binary (the proto is {text, html, urls}), so a bitmap can never be read out of the clipboard. The extension therefore owns the capture and reads the file it wrote.
Cancel is detected by whether an image landed on disk, not by exit code: the tools disagree, and at least one exits 0 on cancel.
grim -g "$(slurp)" is expressed as two argv calls instead of a shell pipeline.
In chat: Snip a Region in the action panel (cmd+shift+S) attaches to the current conversation. Standalone: Snip to Lam starts a new one. Nothing is sent on capture; the view holds the image until you ask something about it, and Return sends with a default question if you have not typed one.
vision.ts caches the POST /api/show capability answer per host and model, so switching model invalidates it by construction. A failed check is unknown, never no, because reporting a dropped connection as a fact about the user's model sends them shopping for a model they do not need. Actions still fail closed: only a confirmed yes offers a snip.
Only the path and mime type reach storage; base64 is read at send time and attached to its own turn only (no replay, the same reasoning as dropping tool output). MAX_STORED_CHARS_PER_TURN would otherwise be blown by one 1080p capture.
Capture is never a model-callable tool. It stays on the action panel and the keybinding, so it only runs because a person asked.
Set Up Lam Agent clears the vision cache on re-test and shows an "Image Input" row for the selected model.

Verification

npx tsc --noEmit, npx eslint ., npx vitest run (244 tests, 15 files), and npm run build (entrypoints now include src/snip.tsx).

Not verified: live interaction with the running launcher, and the non-macOS capture backends (no Linux or Wayland host here). Those are covered by unit tests against an injected PATH and a fake run, so the selection order and argv are pinned, but the binaries themselves have not been executed.

Two changes: screenshot input gated on the model really reporting vision, and a fix for URLs that arrived as code. ## Links The screenshot showed three bare GitHub URLs rendered in a fenced block (dark panel, Copy button, monospace), so they could not be opened. Root cause, verified against the Vicinae source: - `markdown-model.cpp` attaches only the `table` and `strikethrough` GFM extensions, so **`autolink` is off** and a bare URL is plain dead text. - Clickability comes from `CMARK_NODE_LINK` becoming an `<a href>`, which `text-selection-controller.cpp` opens on release via `linkAt` then `openLink`. - So `[label](url)` is the only clickable form, and a URL inside a fence is the worst presentation available. - `promoteStandaloneCodeSpans` was **itself promoting** a lone URL in backticks into a fence, making this worse. Fix: `restoreUrlLinks` recovers three narrow shapes (a fence whose body is all URLs, a lone URL span, a line that is only a URL) into markdown links. Link restoration runs before promotion so a recovered URL is never re-fenced. Anything else is copied through, so a `curl` invocation containing an address is never rewritten. The default prompt now also says never to fence a URL. ## Screenshots - `capture.ts` shells out to an auto-detected backend: `grimblast`, `grim` + `slurp`, `spectacle`, `maim`, `scrot`, `flameshot`, or `screencapture` on macOS. There is no screenshot call in `@vicinae/api`, and `Clipboard` moves no binary (the proto is `{text, html, urls}`), so a bitmap can never be read out of the clipboard. The extension therefore owns the capture and reads the file it wrote. - Cancel is detected by whether an image landed **on disk**, not by exit code: the tools disagree, and at least one exits `0` on cancel. - `grim -g "$(slurp)"` is expressed as two argv calls instead of a shell pipeline. - In chat: **Snip a Region** in the action panel (`cmd+shift+S`) attaches to the current conversation. Standalone: **Snip to Lam** starts a new one. Nothing is sent on capture; the view holds the image until you ask something about it, and Return sends with a default question if you have not typed one. - `vision.ts` caches the `POST /api/show` capability answer per host and model, so switching model invalidates it by construction. A failed check is `unknown`, never `no`, because reporting a dropped connection as a fact about the user's model sends them shopping for a model they do not need. Actions still fail closed: only a confirmed `yes` offers a snip. - Only the path and mime type reach storage; base64 is read at send time and attached to its own turn only (no replay, the same reasoning as dropping tool output). `MAX_STORED_CHARS_PER_TURN` would otherwise be blown by one 1080p capture. - Capture is **never** a model-callable tool. It stays on the action panel and the keybinding, so it only runs because a person asked. - `Set Up Lam Agent` clears the vision cache on re-test and shows an "Image Input" row for the selected model. ## Verification `npx tsc --noEmit`, `npx eslint .`, `npx vitest run` (244 tests, 15 files), and `npm run build` (entrypoints now include `src/snip.tsx`). Not verified: live interaction with the running launcher, and the non-macOS capture backends (no Linux or Wayland host here). Those are covered by unit tests against an injected `PATH` and a fake `run`, so the selection order and argv are pinned, but the binaries themselves have not been executed.

adrianbonpin added 1 commit 2026-10-04 07:13:33 +00:00

feat: snip a region into chat, and make links clickable 5e083d8142

Two changes: screenshot input gated on the model really reporting vision,
and a fix for URLs that arrived as code.

Links
- A fence whose body is all URLs, a lone `https://...` inline span, and a
  line that is only a URL are rewritten into [url](url) markdown links.
  Vicinae attaches only the table and strikethrough GFM extensions, so
  autolink is off and a bare URL is dead text; clickability comes from
  MarkdownModel turning CMARK_NODE_LINK into an <a href>. A fenced URL was
  a dark panel with a Copy button and no way to open it.
- promoteStandaloneCodeSpans was itself promoting a lone URL span into a
  fence, so it made this worse. Link restoration now runs first.
- The default prompt says never to fence a URL.
- Anything that is not one of those three shapes is copied through, so a
  curl invocation containing an address is never rewritten.

Screenshots
- capture.ts shells out to an auto-detected tool: grimblast, grim + slurp,
  spectacle, maim, scrot, flameshot, or screencapture on macOS. There is
  no screenshot call in @vicinae/api and Clipboard moves no binary, so the
  extension owns the capture and reads the file it wrote rather than
  guessing at the clipboard. Cancel is detected by whether an image landed
  on disk, because the tools disagree on exit codes.
- In chat: "Snip a Region" in the action panel (cmd+shift+S) attaches to
  the current conversation. Standalone: "Snip to Lam" starts a new one.
- vision.ts caches the /api/show capability answer per host and model. A
  failed check is "unknown", never "no", so a dropped connection is not
  reported as a fact about the user's model. Actions still fail closed.
- Only the path is stored, base64 is read at send time, and the image goes
  on its own turn only. Capture is never a model-callable tool.

Verified: 244 tests, tsc, eslint, and vici build (both entrypoints).

adrianbonpin added 1 commit 2026-10-04 07:27:01 +00:00

fix: take the launcher window out of the shot when snipping 5f359778b1

The snip was selecting a region around the launcher panel sitting in the
middle of the screen. `activateEntrypoint` calls `showWindow()` when it
activates a command from a closed launcher, so a view command's window is
already up before any extension code runs. Capturing from inside the chat
view could never have worked.

- `snip` is now a `no-view` command. It calls `closeMainWindow()` first,
  waits out the 50ms hide that `ui-service.hpp` schedules, then captures
  with nothing on screen, then brings the window back by launching `chat`
  with the file. `settleWindow()` exists because a capture started the
  instant the close resolves could still find the window there.
- The chat's action no longer captures inline. It launches `snip` and
  passes the session id, so there is one capture path, one capability
  check, and one place that deals with the window.
- The hand-off relays through `snip` rather than going straight back to
  `chat` on purpose: `ActivateEntrypointOptions::toggleIfAlreadyActive`
  defaults to true, so launching the *already active* view command closes
  the window instead of reopening it. A launch cannot override that flag,
  but relaying through a different no-view command makes the toggle branch
  unreachable.
- The action is hidden while streaming, since the hand-off unloads the
  view and would abandon an answer mid-sentence.
- Vision is settled before the selector appears, so a model that cannot
  read images never costs a selection gesture. `no` and `unknown` get
  different HUD wording.
- `lib/snip.ts` holds the hand-off contract; `lib/hooks/useSnip.ts` is
  gone, as nothing captures from a view any more.

Verified: 252 tests, tsc, eslint, prettier, and vici build. The deployed
manifest now reports `snip mode=no-view`.

adrianbonpin added 1 commit 2026-10-04 07:29:02 +00:00

fix: tell a refused screen capture apart from a cancelled one ada98968ac

The snip failed with "could not create image from rect". That message is
macOS's signature for a denied Screen Recording permission, not a cancel,
and it is the only thing distinguishing the two: both write no file, and
screencapture -i has been reported to exit 0 when the permission is
missing. The previous code could not tell them apart, so a permission
problem would either look like Escape or, as here, surface as a bare
"Command failed".

- `RunCommand` now returns stdout and stderr, and `captureRegion`
  collects every message a tool produced, clean exit or not.
- Success is still decided by the file landing on disk, so a tool that
  warns on stderr and writes the image anyway is still a success.
- Failure is triaged from stderr: the macOS denial message becomes
  `CapturePermissionError` with the setting to change (TCC attributes the
  request to Vicinae, since it launched the extension runtime, so the
  permission belongs to Vicinae and needs an app restart). Any other
  complaint becomes a `CaptureToolError` rather than being swallowed.
  Silence with no image is a cancel, which is the common case.
- `describeCaptureFailure` in `lib/snip.ts` keeps both actionable hints,
  which the old catch clause dropped for everything but a missing tool.

Diagnosed from the process tree: the extension runtime (pid 66329) is a
child of Vicinae (pid 66327), so the grant goes on Vicinae.

Verified: 257 tests, tsc, eslint, prettier, and vici build.

adrianbonpin added 1 commit 2026-10-04 07:34:13 +00:00

fix: stop the chat view hanging on "Loading conversations…" after a snip e00b49bbe5

The hand-off launch deadlocked its own startup. `resumed` was initialised to
`Boolean(initialSession)`, which is false for a snip, and the idle-resume
effect returned early whenever there was a hand-off, so nothing ever set it.
The render gate needed both `resumed` and `prepared`, so a successful snip
left the view spinning and there was no way to reach the composer.

Two effects guarding two booleans had already produced one bug here, so this
fixes the shape rather than the value:

- One startup effect and one `startup` flag, cleared in a `finally`. Every
  branch ends in the same place, including a thrown one, so no path can leave
  the view loading.
- The decision tree moves to `lib/startup.ts` as a pure `resolveStartup`,
  which takes its storage and file reads as injected ports. It is now covered
  by 16 tests, including the case that broke: a hand-off must not fall through
  to the idle resume, and must still finish startup.
- `startup.ts` joins the guard that keeps the logic layer free of Vicinae
  imports.
- ChatView is prettier-clean; it had been carrying pre-existing drift and has
  been rewritten twice this session anyway.

Verified: 273 tests, tsc, eslint, prettier, and vici build.

adrianbonpin added 1 commit 2026-10-04 07:37:16 +00:00

feat: render the attached screenshot instead of just naming it c1212192c5

The transcript said "_Screenshot attached_" and left it at that, on a note in
the test file claiming Vicinae could not resolve a local image. It can. That
claim was true of a URL the model invented, not of a file this extension
captured itself.

Checked against the renderer rather than assumed:

- `imageProviderUrl` in `markdown-model.cpp` routes `file://`, and a
  scheme-less string, through `ImageURL::local`. Anything it does not
  recognise comes back empty, which is why a bare path is not good enough.
- `MdImage.qml` draws it block-level and centred, using the alt text as an
  italic caption, and caps the rendered height at 200px regardless of the
  `raycast-width`/`raycast-height` hint it also accepts.

So `attachments.ts` gains `imageSource`, which builds a `file://` URL with
`pathToFileURL` so a path with a space or non-ascii still resolves, and
`markdown.ts` gains `renderAttachments`, used for a turn's attachments and
for an unsent pending snip. That second use is the one that matters most:
the capture is now visible before the question about it exists.

The alt text carries the label, so a deleted file degrades to what the
transcript used to say. The 200px cap is a Vicinae limit, so the action panel
keeps Open Screenshot for the full image.

Tests that asserted the old behaviour are inverted, and imageSource is
covered for spaces and non-ascii.

Verified: 280 tests, tsc, eslint, prettier, and vici build.

adrianbonpin added 1 commit 2026-10-04 07:41:50 +00:00

fix: lay the screenshot preview out to the image, not to the request 2c0653c23c

The preview rendered, but with dead space around it. `MdImage.qml` sizes its
container to the request rather than to the image, so with no hint it is the
full pane width and 200px tall and the image is aspect-fitted inside that. A
wide capture sat in a letterbox: measured against the real captures in the
support directory, a 4:1 crop wasted about 106px of the 200px container and a
6.7:1 crop about 140px, which is what made the transcript look gappy and put
the caption far below the image.

- `readAttachment` now reads the PNG `IHDR` chunk, so `Attachment` carries the
  real dimensions and they are persisted with the turn.
- `previewBox` computes an exact box from those dimensions, scaled to fit
  400x200 with the scale capped at 1 so a small capture is not upscaled, and
  `imageSource` emits it as the `raycast-width`/`raycast-height` hints
  `parseImageSize` looks for. The path is resolved from the URL before the
  query, so the hints never reach the file name.
- The alt text is now "Screenshot". It is drawn centred under the image, so the
  sentence "Screenshot attached" read as a stray line of prose floating in the
  middle of the pane.
- A pending note no longer stacks on top of the empty-transcript placeholder.
  Both said the same thing, one line under the other, once a snip was attached.

A capture with no readable header simply falls back to the old sizing.

Verified: 293 tests, tsc, eslint, prettier, and vici build.

adrianbonpin added 1 commit 2026-10-04 07:44:33 +00:00

feat: let the model answer with tables, lists and headings 35d88f635c

The model refused to write a table and described one instead: a bulleted
list of rows with the column names in inline code. That was our instruction,
not a rendering limit. The default prompt asked for brevity in a narrow
window and then said "skip large headings and wide tables", so the model
obeyed and flattened the answer.

Vicinae renders tables properly. The screenshot that prompted this is a
rendered one: `MdTable.qml` is wired to block type 5, and `markdown-model.cpp`
attaches the `table` GFM extension. Nothing was missing.

- The prompt now names what actually renders (tables, bulleted and numbered
  lists, headings, blockquotes, bold, italics, links, code) and asks for the
  shape that fits, with a cap of about four narrow columns because the detail
  pane really is narrow. It also says outright not to hand back a table as a
  bulleted list of rows.
- `separateTables` adds the blank line a table needs in front of it. GFM tables
  cannot interrupt a paragraph, so a table written straight under a line of
  prose renders as a paragraph of pipes and dashes. Only a genuine table start
  is touched, which is a row followed by a delimiter row; prose with a pipe, a
  table already at a block boundary, and a fenced example are all left alone.
- A regression test asserts a real table survives the whole repair pass
  untouched: pipes are not fences, not links, and not lone spans.

Verified: 304 tests, tsc, eslint, prettier, and vici build.

adrianbonpin added 2 commits 2026-10-04 07:52:26 +00:00

fix: stop reporting a timezone's name as the user's city 400f81fa6c

get_location printed "City: Manila", derived from the last segment of the
IANA zone name. That is the timezone database's representative place, not
the user's: Asia/Manila is one zone covering the entire Philippines,
Asia/Kolkata covers all of India, Europe/Oslo all of Norway. It was wrong
for anyone outside the zone's namesake, and confidently so, because a model
asked about the weather uses it without hesitating. The README claimed the
value was region-level, so the output also contradicted its own docs.

- The zone's label is still reported, as `Timezone area`, and named for what
  it is. Both the field and the tool note say outright that it is not the
  user's city, and the tool description repeats it.
- The locale region subtag is now used, which is a declaration rather than an
  inference: a locale of en-PH is the Philippines because the user set it.
- A Location preference was added. It is the only exact place this extension
  has, it is reported verbatim and first, and when it is set the note tells
  the model to prefer it over anything inferred.
- The continent grouping from the zone name was dropped: "Asia" next to a
  region of "Philippines" reads as two answers to the same question.
- ToolContext takes the place as an injected value, keeping tools.ts free of
  Vicinae imports as the module wiring test requires.

Verified: 316 tests, tsc, eslint, prettier, and vici build.

feat: opt-in IP geolocation for get_location 51c16b678f

Adds the one thing the extension could not answer before: roughly where
the user actually is, rather than which zone their clock is set to.

- `lib/geo.ts` resolves a place from ipapi.co. Chosen because it is HTTPS
  and needs no API key; ip-api.com, the other obvious free option, is
  HTTP-only on its free tier, which is not a place to send an IP address.
- Off by default, and skipped entirely when the user has set Location: the
  better answer is already there, so spending their privacy on a worse one
  would be indefensible. `ipPlace` is only wired into ToolContext when the
  preference is on, so an off setting cannot reach the network at all.
- Cached for 6 hours. A city does not change between questions, and every
  lookup hands the provider another copy of the address. Set Up Lam Agent
  clears it on re-test, alongside the vision cache.
- Failures are returned, not swallowed: a rate limit or an offline machine
  says so in the tool output while the region answer stays available. The
  provider signals some failures with error: true and HTTP 200, so the body
  is checked as well as the status.
- The tool output always names the provider and calls the result city-level
  rather than exact. It also flags an IP timezone that disagrees with the
  device clock, which usually means a VPN.
- The README claimed no geolocation service ever saw the user's address.
  That is now conditional, so the section was rewritten rather than left to
  quietly become untrue.

Deliberately not done: GPS. CoreLocation has no CLI on macOS and GeoClue's
usual provider resolves nearby Wi-Fi through a Mozilla service anyway.

Verified: 341 tests (25 new), tsc, eslint, prettier, and vici build. I did
not make a live API call, because doing so would send this machine's IP to
the provider as the exact flow being opted into.

adrianbonpin added 1 commit 2026-10-04 07:59:14 +00:00

fix: ipapi.co stopped answering, so the lookup never resolved af325b4a62

Turning IP Geolocation on changed nothing: every lookup failed, and the
tool output still said "no Location is set in preferences", which is a
different problem and sent the user looking in the wrong place.

ipapi.co no longer serves an unauthenticated caller. Verified directly:

  no User-Agent    -> HTTP 429 {"reason":"RateLimited",
                                 "message":"Please sign up for a paid plan"}
  browser UA       -> HTTP 403, a Cloudflare "Just a moment..." challenge

So the provider is now ipwho.is, which answers 200 without a key. It
signals refusal with success: false on an HTTP 200, so the body is checked
as well as the status, and it nests the timezone under an object, which the
parser now accepts alongside a bare string. A named User-Agent is sent:
node's default is bare, and "who is asking" is the one thing a geolocation
provider has a fair interest in.

The failure branch is also fixed. A failed lookup and an unset preference
are different problems, and the wording no longer conflates them. The note
tells the model to pass the failure on, because the model is the only
channel that reaches the user and it had been paraphrasing the failure away.

Accuracy is documented honestly rather than as "city-level". An IP locates
the ISP's egress, not the person: measured from one machine, an IPv4 query
resolved to Cebu City and an IPv6 query moments later to Talisay in
Calabarzon, about 500km apart. That is now in the README, the preference
description and the tool output, and it is the argument for setting
Location instead.

Verified live with the exact request and field access the extension uses:
HTTP 200, "Talisay, Calabarzon (Region IV-A), Philippines", Asia/Manila.
344 tests, tsc, eslint, prettier, and vici build.

adrianbonpin added 1 commit 2026-10-04 08:08:29 +00:00

fix: end a spent tool budget in an answer, not a notice about the loop 1ae44adc7b

Asked for the fish in every Valheim biome, the turn researched for five
rounds and then displayed "Stopped after 5 rounds of tool calls without a
final answer." Three things compounded:

- MAX_TOOL_ITERATIONS of 5 is too few for a question that needs several
  searches and a few pages. It is a safety net against looping, not a
  budget anyone chose.
- The loop discarded everything. `content` was scoped to the iteration, so
  every preamble the model wrote was thrown away and the turn ended with an
  app-generated message about its own control flow.
- The model was never told it was running out, so it kept searching until
  the hard stop instead of answering with what it had.

The budget is now spent on a closing round with the tools taken away and an
instruction to answer from what was gathered, naming anything it could not
confirm. So the worst case is a partial answer that says what is missing.
If the closing round produces nothing, the last preamble is used, and only
then the notice. The cap is 8 because the extra round is not more searching,
it is the answer.

Verified: 349 tests, tsc, eslint, prettier, and vici build.

adrianbonpin merged commit 576c5946f7 into prod

2026-10-05 08:38:55 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: adrianbonpin/lam-agent-vicinae#14