adrianbonpin/lam-agent-vicinae
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: snip a region into chat, and make links clickable

Browse Source 
Two changes: screenshot input gated on the model really reporting vision,
and a fix for URLs that arrived as code.

Links
- A fence whose body is all URLs, a lone `https://...` inline span, and a
  line that is only a URL are rewritten into [url](url) markdown links.
  Vicinae attaches only the table and strikethrough GFM extensions, so
  autolink is off and a bare URL is dead text; clickability comes from
  MarkdownModel turning CMARK_NODE_LINK into an <a href>. A fenced URL was
  a dark panel with a Copy button and no way to open it.
- promoteStandaloneCodeSpans was itself promoting a lone URL span into a
  fence, so it made this worse. Link restoration now runs first.
- The default prompt says never to fence a URL.
- Anything that is not one of those three shapes is copied through, so a
  curl invocation containing an address is never rewritten.

Screenshots
- capture.ts shells out to an auto-detected tool: grimblast, grim + slurp,
  spectacle, maim, scrot, flameshot, or screencapture on macOS. There is
  no screenshot call in @vicinae/api and Clipboard moves no binary, so the
  extension owns the capture and reads the file it wrote rather than
  guessing at the clipboard. Cancel is detected by whether an image landed
  on disk, because the tools disagree on exit codes.
- In chat: "Snip a Region" in the action panel (cmd+shift+S) attaches to
  the current conversation. Standalone: "Snip to Lam" starts a new one.
- vision.ts caches the /api/show capability answer per host and model. A
  failed check is "unknown", never "no", so a dropped connection is not
  reported as a fact about the user's model. Actions still fail closed.
- Only the path is stored, base64 is read at send time, and the image goes
  on its own turn only. Capture is never a model-callable tool.

Verified: 244 tests, tsc, eslint, and vici build (both entrypoints).
This commit is contained in:
adrianbonpin
2026-10-04 15:13:15 +08:00
parent f5d1ced4d6
commit 5e083d8142
29 changed files with 1744 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+42 -5
View File
@@ -5,7 +5,7 @@ Agentic AI chat inside the [Vicinae](https://vicinae.com) launcher, powered by [
Hit a global shortcut, type a question, get a streamed answer without leaving whatever you were doing.
<!-- prettier-ignore -->
> **Status:** early. Chat, streaming, persistent sessions and the web tools work. Anything that *acts* on your machine (opening links, clipboard, launching apps) is deliberately not here yet. See [Roadmap](#roadmap).
> **Status:** early. Chat, streaming, persistent sessions, the web tools and screen capture work. Anything else that *acts* on your machine (opening links, clipboard, launching apps) is deliberately not here yet. See [Roadmap](#roadmap).
---
@@ -16,15 +16,29 @@ Hit a global shortcut, type a question, get a streamed answer without leaving wh
- **Persistent conversations.** Every chat is saved locally. The sidebar lists them by recency, and reopening the command picks up where you left off, unless it has been idle longer than you allow.
- **Readable code.** Fenced blocks render in a dark panel with the language named in a header, syntax colouring, and a Copy button.
- **Tools.** Lam decides when to call them: current time, location context, web search, and page fetching.
- **Chat only.** Lam answers questions. It does not touch your files or run commands.
- **Screenshots, two ways.** Snip a region into the conversation you are already in from the action panel, or run **Snip to Lam** to start a new chat about it. Neither is offered unless the current model can actually see images.
- **Chat only.** Lam answers questions. It does not touch your files or run commands. The one exception is capture, and it only ever happens because you pressed the key for it.
## Requirements
- Vicinae with the extension runtime enabled (Node.js 20+ on `PATH`)
- An [Ollama](https://ollama.com) account and API key, which you can [create here](https://ollama.com/settings/keys)
- A screenshot tool, if you want to snip. A vision model, if you want Lam to read the result
Ollama Cloud is the default. A local Ollama server works too: point **Ollama Host** at `http://localhost:11434`. Note that the web tools are a cloud feature and are not available from a local server.
### Snipping needs a capture tool
Vicinae has no screenshot API, so the extension shells out. It auto-detects the first of these that is installed, in this order:
| Platform | Tools, in preference order |
| --- | --- |
| Linux (Wayland) | `grimblast`, then `grim` + `slurp`, then `spectacle` |
| Linux (X11) | `maim` + `slop`, then `scrot`, then `flameshot` |
| macOS | `screencapture`, which ships with the system |
If none is found, the chat says so and names what to install. Nothing to configure: the detection happens per platform at capture time.
## Install
```sh
@@ -57,6 +71,7 @@ To change the model later, either edit the **Model** preference directly, or run
| `↩` | Send the message in the search bar |
| `↩` with an empty bar | Open the highlighted conversation |
| `⌘ N` | New chat |
| `⌘ ⇧ S` | Snip a region and attach it to this chat (only with a vision model) |
| `⌘ ⇧ ⌫` | Delete the highlighted conversation (also in **Chat Sessions**) |
Prompts are single line. An earlier `⌘ ↩` multi-line composer was removed. See [Architecture](#architecture) for why a `Form.TextArea` cannot be laid out cleanly in a launcher. The `Tab` shortcut went with it: the left column *is* the conversation switcher now, so an accessory dropdown would only duplicate it.
@@ -67,14 +82,17 @@ Assign these in **Vicinae Settings → Extensions → Lam Agent**, or bind them
```sh
vicinae 'vicinae://launch/lam-agent/chat?toggle=true'
vicinae 'vicinae://launch/lam-agent/snip'
```
```ini
# Hyprland
bind = $mainAlt, Space, exec, vicinae 'vicinae://launch/lam-agent/chat?toggle=true'
bind = $mainAlt, S, exec, vicinae 'vicinae://launch/lam-agent/snip'
# niri
Mod+Shift+Space repeat=false { spawn "vicinae" "vicinae://launch/lam-agent/chat?toggle=true"; }
Mod+Shift+S repeat=false { spawn "vicinae" "vicinae://launch/lam-agent/snip"; }
```
`fallbackText` pre-fills the prompt, which makes a handy "ask about this" command:
@@ -88,8 +106,9 @@ vicinae 'vicinae://launch/lam-agent/chat?fallbackText=summarise%20this'
| Command | Purpose |
| --- | --- |
| **Chat with Lam** | The chat surface. Bind this to your global shortcut. |
| **Snip to Lam** | Capture a region and start a new chat about it. Bind this to a second shortcut for one-key "snip and ask". |
| **Chat Sessions** | Browse, reopen, and delete past conversations. |
| **Set Up Lam Agent** | Verify the connection and list available models. |
| **Set Up Lam Agent** | Verify the connection, list available models, and check whether the selected model can see images. |
## Configuration
@@ -107,7 +126,7 @@ vicinae 'vicinae://launch/lam-agent/chat?fallbackText=summarise%20this'
## Tools
Lam calls these on its own when they would improve an answer. All four are read-only.
Lam calls these on its own when they would improve an answer. All four are read-only, and all four are the entire list of things Lam is allowed to do without being asked twice. Screen capture is deliberately not among them.
| Tool | What it does |
| --- | --- |
@@ -125,19 +144,26 @@ Web search results and fetched pages can be large. Lam truncates them before the
```
src/
chat.tsx # "chat" command entrypoint, thin wrapper
snip.tsx # "snip" command entrypoint, captures then chats
sessions.tsx # browse, reopen, delete conversations
setup.tsx # connection check + live model list
lib/
ollama.ts # the only file that talks HTTP
agent.ts # the tool-calling loop
tools.ts # tool schemas + dispatch
capture.ts # screen capture backends, shelled out
attachments.ts # image files -> base64 payloads
vision.ts # cached "can this model see?" answer
sessions.ts # LocalStorage persistence
history.ts # turns -> API messages
location.ts # time/place, derived locally
thread.ts # turns -> flat message list
markdown.ts # messages -> detail-panel markdown
code.ts # code fences and links, repaired
preferences.ts # preferences, normalised
hooks/useChat.ts # the chat state machine
hooks/useSnip.ts # capture -> pending attachment
hooks/useVision.ts # capability check, for gating snip actions
components/ChatView.tsx # the chat surface
```
@@ -161,12 +187,22 @@ A few decisions worth knowing about before you change things:
**Code fences are labelled so Vicinae can colour them.** `MdCodeBlock.qml` already provides the dark panel, a header naming the language, and a Copy button, so none of that is built here. The language string is what turns on highlighting: `syntax::highlight` resolves it with `definitionForName(language)` and then `definitionForFileName("file." + language)`. So `bash` and `ts` colour correctly, while an unrecognised string such as `terminal` still renders a label but produces no highlighting at all, so a wrong label is worse than none. `code.ts` therefore labels bare fences conservatively, only when a signal is clear, and the default prompt asks for a language on every fence.
**A URL is only clickable as a markdown link, so `code.ts` takes them back out of code.** Vicinae's parser attaches the `table` and `strikethrough` GFM extensions and nothing else, so the `autolink` extension is off: a bare URL is plain dead text. Clickability comes from `MarkdownModel` turning `CMARK_NODE_LINK` into an `<a href>`, which the selection controller opens on release. A URL the model fenced is therefore the worst possible presentation, a dark panel with a Copy button and no way to open the address. `restoreUrlLinks` recovers three narrow shapes (a fence whose entire body is URLs, a lone `` `https://...` `` span, and a line that is only a URL) into `[url](url)` links, and the default prompt now says never to fence a URL in the first place. Anything that is not one of those three shapes is copied through, so a `curl` invocation containing an address is never rewritten.
**Streamed deltas are buffered in refs and flushed to state on a timer** (`FLUSH_INTERVAL_MS`). Re-rendering the transcript per token makes the launcher visibly janky.
**The logic layer has no Vicinae imports.** `agent.ts`, `ollama.ts`, `tools.ts` and friends are plain TypeScript, which is why they unit test without a runtime. A test enforces this.
**Sessions are capped twice over**, once per turn and once per session, because `LocalStorage` has no documented ceiling.
**Screen capture is owned, not borrowed.** There is no screenshot call anywhere in `@vicinae/api`, and `Clipboard` moves text, html, urls and a file path and nothing binary, so a bitmap can never be read back out of the clipboard either. The only way to get a region of the screen is to shell out, which is fair game because extensions run under Node. The useful consequence is that the extension picks the output path and reads the file it wrote, rather than guessing what some other application left on the clipboard. Cancel detection leans on the file for the same reason: every tool reports an abandoned selection differently, and at least one exits `0` on cancel, so whether an image actually landed on disk is the only signal they agree on. Snips go in the extension's support directory rather than a temp directory, because a stored conversation keeps the path and a path that survives a reboot is the difference between an old chat showing its screenshot and pointing at nothing.
**Image support is a fact from the API, not a guess from the model name, and it is cached.** `GET /api/tags` returns names, sizes and quantization levels and says nothing about capability, so the check is a `POST /api/show` that reads the `vision` flag. That is a round trip per model, so the answer is persisted under a key built from the host and model name, which means switching model invalidates it by construction. The state is a tri-state rather than a boolean: a failed request is `"unknown"`, never `"no"`, because telling someone their model cannot see when the network merely dropped is a false claim about their setup that sends them shopping for a model they do not need. The actions still fail closed, so only a confirmed `"yes"` offers a snip.
**The capture is never a tool.** It stays in the action panel and on a keybinding, so it only ever runs because a person asked for it. A model that could decide to screenshot the screen is a different risk profile from one that answers questions, which is the same line the read-only tools already sit behind.
**An attached image is sent once and never replayed.** It is the largest thing a request can carry, so `buildMessages` attaches it to the question it belongs to and drops it afterwards, exactly as it already drops tool output. The cost is that a follow-up about a screenshot needs the image attached again; the alternative is re-sending every screenshot in the conversation on every turn. Only the path and mime type go into storage: `MAX_STORED_CHARS_PER_TURN` exists so one turn cannot dominate `LocalStorage`, and base64 would blow past it on a single 1080p capture.
## Development
```sh
@@ -183,7 +219,8 @@ npm test
- **Actions.** Opening links, clipboard access, launching apps. These need an approval step first. An assistant that can silently act on your machine is a different risk profile from one that answers questions, and this release does not pretend otherwise.
- **`@` mentions** for files and clipboard content.
- **Per-conversation model** overrides.
- **Image input**, which Ollama's chat API already supports.
- **Annotations on a snip.** The capture backends are there, but drawing on the result before sending it needs a second tool and a wait for the editor to save.
- **Follow-ups that keep the image.** Replaying an attachment is a token cost that has to be made opt-in first.
## License