Compare commits
7
Commits
prod
...
35d88f635c
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
35d88f635c | ||
|
|
2c0653c23c | ||
|
|
c1212192c5 | ||
|
|
e00b49bbe5 | ||
|
|
ada98968ac | ||
|
|
5f359778b1 | ||
|
|
5e083d8142 |
@@ -5,7 +5,7 @@ Agentic AI chat inside the [Vicinae](https://vicinae.com) launcher, powered by [
|
|||||||
Hit a global shortcut, type a question, get a streamed answer without leaving whatever you were doing.
|
Hit a global shortcut, type a question, get a streamed answer without leaving whatever you were doing.
|
||||||
|
|
||||||
<!-- prettier-ignore -->
|
<!-- prettier-ignore -->
|
||||||
> **Status:** early. Chat, streaming, persistent sessions and the web tools work. Anything that *acts* on your machine (opening links, clipboard, launching apps) is deliberately not here yet. See [Roadmap](#roadmap).
|
> **Status:** early. Chat, streaming, persistent sessions, the web tools and screen capture work. Anything else that *acts* on your machine (opening links, clipboard, launching apps) is deliberately not here yet. See [Roadmap](#roadmap).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -15,16 +15,35 @@ Hit a global shortcut, type a question, get a streamed answer without leaving wh
|
|||||||
- **Streaming answers.** Tokens render as they arrive, in a markdown side panel.
|
- **Streaming answers.** Tokens render as they arrive, in a markdown side panel.
|
||||||
- **Persistent conversations.** Every chat is saved locally. The sidebar lists them by recency, and reopening the command picks up where you left off, unless it has been idle longer than you allow.
|
- **Persistent conversations.** Every chat is saved locally. The sidebar lists them by recency, and reopening the command picks up where you left off, unless it has been idle longer than you allow.
|
||||||
- **Readable code.** Fenced blocks render in a dark panel with the language named in a header, syntax colouring, and a Copy button.
|
- **Readable code.** Fenced blocks render in a dark panel with the language named in a header, syntax colouring, and a Copy button.
|
||||||
|
- **Real markdown.** Tables, bulleted and numbered lists, headings, blockquotes, bold, italics and links all render natively, and the default prompt asks for the shape that fits the answer rather than flattening everything into prose.
|
||||||
- **Tools.** Lam decides when to call them: current time, location context, web search, and page fetching.
|
- **Tools.** Lam decides when to call them: current time, location context, web search, and page fetching.
|
||||||
- **Chat only.** Lam answers questions. It does not touch your files or run commands.
|
- **Screenshots, two ways.** Snip a region into the conversation you are already in from the action panel, or run **Snip to Lam** to start a new chat about it. The launcher window gets out of the way first, so it never ends up in the shot, and the capture appears in the transcript as a thumbnail with Open Screenshot for the full image. Neither is offered unless the current model can actually see images.
|
||||||
|
- **Chat only.** Lam answers questions. It does not touch your files or run commands. The one exception is capture, and it only ever happens because you pressed the key for it.
|
||||||
|
|
||||||
## Requirements
|
## Requirements
|
||||||
|
|
||||||
- Vicinae with the extension runtime enabled (Node.js 20+ on `PATH`)
|
- Vicinae with the extension runtime enabled (Node.js 20+ on `PATH`)
|
||||||
- An [Ollama](https://ollama.com) account and API key, which you can [create here](https://ollama.com/settings/keys)
|
- An [Ollama](https://ollama.com) account and API key, which you can [create here](https://ollama.com/settings/keys)
|
||||||
|
- A screenshot tool, if you want to snip. A vision model, if you want Lam to read the result
|
||||||
|
|
||||||
Ollama Cloud is the default. A local Ollama server works too: point **Ollama Host** at `http://localhost:11434`. Note that the web tools are a cloud feature and are not available from a local server.
|
Ollama Cloud is the default. A local Ollama server works too: point **Ollama Host** at `http://localhost:11434`. Note that the web tools are a cloud feature and are not available from a local server.
|
||||||
|
|
||||||
|
### Snipping needs a capture tool
|
||||||
|
|
||||||
|
Vicinae has no screenshot API, so the extension shells out. It auto-detects the first of these that is installed, in this order:
|
||||||
|
|
||||||
|
| Platform | Tools, in preference order |
|
||||||
|
| --- | --- |
|
||||||
|
| Linux (Wayland) | `grimblast`, then `grim` + `slurp`, then `spectacle` |
|
||||||
|
| Linux (X11) | `maim` + `slop`, then `scrot`, then `flameshot` |
|
||||||
|
| macOS | `screencapture`, which ships with the system |
|
||||||
|
|
||||||
|
If none is found, the chat says so and names what to install. Nothing to configure: the detection happens per platform at capture time.
|
||||||
|
|
||||||
|
**On macOS, Vicinae itself needs Screen Recording permission.** macOS attributes a capture request to the app responsible for launching the process that made it, which is Vicinae rather than the extension, so the grant goes on Vicinae: **System Settings, Privacy & Security, Screen & System Audio Recording**, then quit and reopen Vicinae (a TCC change does not apply to a running app).
|
||||||
|
|
||||||
|
Without it, `screencapture` prints `could not create image from rect` and writes nothing. That is awkward, because a cancelled selection also writes nothing, and `screencapture -i` has been reported to exit `0` when the permission is missing. So the file alone cannot tell the two apart. `capture.ts` therefore reads stderr as well: that specific message becomes a permission error naming the setting to change, and only silence with no image is treated as a cancel.
|
||||||
|
|
||||||
## Install
|
## Install
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
@@ -57,6 +76,7 @@ To change the model later, either edit the **Model** preference directly, or run
|
|||||||
| `↩` | Send the message in the search bar |
|
| `↩` | Send the message in the search bar |
|
||||||
| `↩` with an empty bar | Open the highlighted conversation |
|
| `↩` with an empty bar | Open the highlighted conversation |
|
||||||
| `⌘ N` | New chat |
|
| `⌘ N` | New chat |
|
||||||
|
| `⌘ ⇧ S` | Snip a region and attach it to this chat (only with a vision model) |
|
||||||
| `⌘ ⇧ ⌫` | Delete the highlighted conversation (also in **Chat Sessions**) |
|
| `⌘ ⇧ ⌫` | Delete the highlighted conversation (also in **Chat Sessions**) |
|
||||||
|
|
||||||
Prompts are single line. An earlier `⌘ ↩` multi-line composer was removed. See [Architecture](#architecture) for why a `Form.TextArea` cannot be laid out cleanly in a launcher. The `Tab` shortcut went with it: the left column *is* the conversation switcher now, so an accessory dropdown would only duplicate it.
|
Prompts are single line. An earlier `⌘ ↩` multi-line composer was removed. See [Architecture](#architecture) for why a `Form.TextArea` cannot be laid out cleanly in a launcher. The `Tab` shortcut went with it: the left column *is* the conversation switcher now, so an accessory dropdown would only duplicate it.
|
||||||
@@ -67,14 +87,17 @@ Assign these in **Vicinae Settings → Extensions → Lam Agent**, or bind them
|
|||||||
|
|
||||||
```sh
|
```sh
|
||||||
vicinae 'vicinae://launch/lam-agent/chat?toggle=true'
|
vicinae 'vicinae://launch/lam-agent/chat?toggle=true'
|
||||||
|
vicinae 'vicinae://launch/lam-agent/snip'
|
||||||
```
|
```
|
||||||
|
|
||||||
```ini
|
```ini
|
||||||
# Hyprland
|
# Hyprland
|
||||||
bind = $mainAlt, Space, exec, vicinae 'vicinae://launch/lam-agent/chat?toggle=true'
|
bind = $mainAlt, Space, exec, vicinae 'vicinae://launch/lam-agent/chat?toggle=true'
|
||||||
|
bind = $mainAlt, S, exec, vicinae 'vicinae://launch/lam-agent/snip'
|
||||||
|
|
||||||
# niri
|
# niri
|
||||||
Mod+Shift+Space repeat=false { spawn "vicinae" "vicinae://launch/lam-agent/chat?toggle=true"; }
|
Mod+Shift+Space repeat=false { spawn "vicinae" "vicinae://launch/lam-agent/chat?toggle=true"; }
|
||||||
|
Mod+Shift+S repeat=false { spawn "vicinae" "vicinae://launch/lam-agent/snip"; }
|
||||||
```
|
```
|
||||||
|
|
||||||
`fallbackText` pre-fills the prompt, which makes a handy "ask about this" command:
|
`fallbackText` pre-fills the prompt, which makes a handy "ask about this" command:
|
||||||
@@ -88,8 +111,9 @@ vicinae 'vicinae://launch/lam-agent/chat?fallbackText=summarise%20this'
|
|||||||
| Command | Purpose |
|
| Command | Purpose |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| **Chat with Lam** | The chat surface. Bind this to your global shortcut. |
|
| **Chat with Lam** | The chat surface. Bind this to your global shortcut. |
|
||||||
|
| **Snip to Lam** | Capture a region and start a new chat about it. A `no-view` command, so the launcher window closes before the selector appears. Bind this to a second shortcut for one-key "snip and ask". |
|
||||||
| **Chat Sessions** | Browse, reopen, and delete past conversations. |
|
| **Chat Sessions** | Browse, reopen, and delete past conversations. |
|
||||||
| **Set Up Lam Agent** | Verify the connection and list available models. |
|
| **Set Up Lam Agent** | Verify the connection, list available models, and check whether the selected model can see images. |
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
@@ -107,7 +131,7 @@ vicinae 'vicinae://launch/lam-agent/chat?fallbackText=summarise%20this'
|
|||||||
|
|
||||||
## Tools
|
## Tools
|
||||||
|
|
||||||
Lam calls these on its own when they would improve an answer. All four are read-only.
|
Lam calls these on its own when they would improve an answer. All four are read-only, and all four are the entire list of things Lam is allowed to do without being asked twice. Screen capture is deliberately not among them.
|
||||||
|
|
||||||
| Tool | What it does |
|
| Tool | What it does |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
@@ -125,19 +149,26 @@ Web search results and fetched pages can be large. Lam truncates them before the
|
|||||||
```
|
```
|
||||||
src/
|
src/
|
||||||
chat.tsx # "chat" command entrypoint, thin wrapper
|
chat.tsx # "chat" command entrypoint, thin wrapper
|
||||||
|
snip.tsx # "snip" command entrypoint, no-view: captures then hands over
|
||||||
sessions.tsx # browse, reopen, delete conversations
|
sessions.tsx # browse, reopen, delete conversations
|
||||||
setup.tsx # connection check + live model list
|
setup.tsx # connection check + live model list
|
||||||
lib/
|
lib/
|
||||||
ollama.ts # the only file that talks HTTP
|
ollama.ts # the only file that talks HTTP
|
||||||
agent.ts # the tool-calling loop
|
agent.ts # the tool-calling loop
|
||||||
tools.ts # tool schemas + dispatch
|
tools.ts # tool schemas + dispatch
|
||||||
|
capture.ts # screen capture backends, shelled out
|
||||||
|
attachments.ts # image files -> base64 payloads
|
||||||
|
vision.ts # cached "can this model see?" answer
|
||||||
sessions.ts # LocalStorage persistence
|
sessions.ts # LocalStorage persistence
|
||||||
history.ts # turns -> API messages
|
history.ts # turns -> API messages
|
||||||
location.ts # time/place, derived locally
|
location.ts # time/place, derived locally
|
||||||
thread.ts # turns -> flat message list
|
thread.ts # turns -> flat message list
|
||||||
markdown.ts # messages -> detail-panel markdown
|
markdown.ts # messages -> detail-panel markdown
|
||||||
|
code.ts # code fences and links, repaired
|
||||||
preferences.ts # preferences, normalised
|
preferences.ts # preferences, normalised
|
||||||
hooks/useChat.ts # the chat state machine
|
hooks/useChat.ts # the chat state machine
|
||||||
|
hooks/useVision.ts # capability check, for gating the snip action
|
||||||
|
snip.ts # the snip -> chat handoff contract
|
||||||
components/ChatView.tsx # the chat surface
|
components/ChatView.tsx # the chat surface
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -157,16 +188,42 @@ A few decisions worth knowing about before you change things:
|
|||||||
|
|
||||||
**The transcript reads newest message first.** The search bar sits at the top of the window, so the most recent message, an answer once one has been written, is the first line under it rather than below the fold. Within a single exchange that means the answer appears above the question it responds to. `buildThread` produces that order, and `buildThreadMarkdown` regroups messages back into exchanges so the horizontal rule separates one exchange from the next instead of dividing every message.
|
**The transcript reads newest message first.** The search bar sits at the top of the window, so the most recent message, an answer once one has been written, is the first line under it rather than below the fold. Within a single exchange that means the answer appears above the question it responds to. `buildThread` produces that order, and `buildThreadMarkdown` regroups messages back into exchanges so the horizontal rule separates one exchange from the next instead of dividing every message.
|
||||||
|
|
||||||
**The transcript is text-only, and nothing in it is bold except an error.** The answer is what gets read, so it is plain body text carrying no label; the question is a blockquote, which the renderer draws dimmed, italic, and with a left rule. A `**You**` / `**Lam**` header on each exchange gave both speakers identical weight and made the answer sit behind chrome. Tool calls collapse to one italic line (_Tools: `web_search`_) rather than a bold header plus a list, so they stay below the answer too. Raycast solves the same problem with alignment, but markdown offers no alignment control.
|
**The transcript is text plus the attached images, and nothing in it is bold except an error.** The answer is what gets read, so it is plain body text carrying no label; the question is a blockquote, which the renderer draws dimmed, italic, and with a left rule. A `**You**` / `**Lam**` header on each exchange gave both speakers identical weight and made the answer sit behind chrome. Tool calls collapse to one italic line (_Tools: `web_search`_) rather than a bold header plus a list, so they stay below the answer too. Raycast solves the same problem with alignment, but markdown offers no alignment control.
|
||||||
|
|
||||||
|
**An attached screenshot is rendered, not described.** This is a correction: the transcript used to say `_Screenshot attached_` and leave it at that, on the belief that Vicinae could not resolve a local image. It can. `imageProviderUrl` in `markdown-model.cpp` routes `file://` (and a scheme-less string) through `ImageURL::local`, and `MdImage.qml` draws it block-level and centred with the alt text as an italic caption underneath.
|
||||||
|
|
||||||
|
**The preview is asked for as an exact box, which is why the PNG header is parsed.** `MdImage.qml` sizes its container to the *request* rather than to the image: with no hint it is the full pane width and 200px tall, and the image is aspect-fitted inside that, so a wide capture sits in a letterbox with dead space above and below. Measured on real captures here, a 4:1 crop wasted about 106px of a 200px container and a 6.7:1 crop wasted about 140px. `previewBox` therefore computes a box from the image's real dimensions, scales it to fit 400x200, and emits it as the `raycast-width` and `raycast-height` hints the renderer reads, so the container matches the image and nothing is wasted. The scale is capped at 1 so a small capture is not blown up.
|
||||||
|
|
||||||
|
The rendered height stays capped at 200px whatever is asked for (`Math.min(_rawH, 200, maxImageHeight)`), so this is a thumbnail rather than something readable, which is why the action panel still offers Open Screenshot for the full image. The alt text is short for the same reason it is not empty: whatever is there is drawn centred under the image, so a sentence reads as a stray line of prose, and it is the only thing left if the capture file has since been deleted. A preview of an unsent attachment is shown the same way, so a snip is visible before the question about it exists.
|
||||||
|
|
||||||
**Code fences are labelled so Vicinae can colour them.** `MdCodeBlock.qml` already provides the dark panel, a header naming the language, and a Copy button, so none of that is built here. The language string is what turns on highlighting: `syntax::highlight` resolves it with `definitionForName(language)` and then `definitionForFileName("file." + language)`. So `bash` and `ts` colour correctly, while an unrecognised string such as `terminal` still renders a label but produces no highlighting at all, so a wrong label is worse than none. `code.ts` therefore labels bare fences conservatively, only when a signal is clear, and the default prompt asks for a language on every fence.
|
**Code fences are labelled so Vicinae can colour them.** `MdCodeBlock.qml` already provides the dark panel, a header naming the language, and a Copy button, so none of that is built here. The language string is what turns on highlighting: `syntax::highlight` resolves it with `definitionForName(language)` and then `definitionForFileName("file." + language)`. So `bash` and `ts` colour correctly, while an unrecognised string such as `terminal` still renders a label but produces no highlighting at all, so a wrong label is worse than none. `code.ts` therefore labels bare fences conservatively, only when a signal is clear, and the default prompt asks for a language on every fence.
|
||||||
|
|
||||||
|
**A URL is only clickable as a markdown link, so `code.ts` takes them back out of code.** Vicinae's parser attaches the `table` and `strikethrough` GFM extensions and nothing else, so the `autolink` extension is off: a bare URL is plain dead text. Clickability comes from `MarkdownModel` turning `CMARK_NODE_LINK` into an `<a href>`, which the selection controller opens on release. A URL the model fenced is therefore the worst possible presentation, a dark panel with a Copy button and no way to open the address. `restoreUrlLinks` recovers three narrow shapes (a fence whose entire body is URLs, a lone `` `https://...` `` span, and a line that is only a URL) into `[url](url)` links, and the default prompt now says never to fence a URL in the first place. Anything that is not one of those three shapes is copied through, so a `curl` invocation containing an address is never rewritten.
|
||||||
|
|
||||||
|
**A table needs a blank line in front of it, and the prompt alone does not guarantee one.** Vicinae renders tables properly (`MdTable.qml`, block type 5), but GFM tables cannot interrupt a paragraph, so a table written directly under a line of prose is not a table at all: it renders as a paragraph of pipes and dashes. The blank line looks unnecessary, so models leave it out. `separateTables` adds it, and only in front of a genuine table start, which is a row followed by a delimiter row. Prose containing a pipe is left alone, a table already at a block boundary is left alone, and a fenced example is never touched.
|
||||||
|
|
||||||
|
**The prompt used to forbid tables, which is worth not repeating.** It asked for brevity in a narrow window and then said "skip large headings and wide tables", and a model follows that instruction by describing the table it was about to write instead of writing it: a bulleted list of rows, with the column names in inline code. Brevity is about length, not about which markdown is allowed, so the guidance now names what renders and asks for the shape that fits, with a cap of about four narrow columns because the detail pane really is narrow.
|
||||||
|
|
||||||
**Streamed deltas are buffered in refs and flushed to state on a timer** (`FLUSH_INTERVAL_MS`). Re-rendering the transcript per token makes the launcher visibly janky.
|
**Streamed deltas are buffered in refs and flushed to state on a timer** (`FLUSH_INTERVAL_MS`). Re-rendering the transcript per token makes the launcher visibly janky.
|
||||||
|
|
||||||
**The logic layer has no Vicinae imports.** `agent.ts`, `ollama.ts`, `tools.ts` and friends are plain TypeScript, which is why they unit test without a runtime. A test enforces this.
|
**The logic layer has no Vicinae imports.** `agent.ts`, `ollama.ts`, `tools.ts` and friends are plain TypeScript, which is why they unit test without a runtime. A test enforces this.
|
||||||
|
|
||||||
**Sessions are capped twice over**, once per turn and once per session, because `LocalStorage` has no documented ceiling.
|
**Sessions are capped twice over**, once per turn and once per session, because `LocalStorage` has no documented ceiling.
|
||||||
|
|
||||||
|
**Screen capture is owned, not borrowed.** There is no screenshot call anywhere in `@vicinae/api`, and `Clipboard` moves text, html, urls and a file path and nothing binary, so a bitmap can never be read back out of the clipboard either. The only way to get a region of the screen is to shell out, which is fair game because extensions run under Node. The useful consequence is that the extension picks the output path and reads the file it wrote, rather than guessing what some other application left on the clipboard.
|
||||||
|
|
||||||
|
Deciding whether a capture *worked* takes two signals, not one. The image landing on disk is what makes it a success, because the backends disagree about exit codes and at least one reports success on a cancelled selection. But the file cannot distinguish a cancel from a refused permission, since both leave nothing behind and macOS can exit `0` on the refusal, so the refusal is recognised by its stderr message instead. Silence with no image is the user pressing Escape; a complaint with no image is a failure the user is told about. Snips go in the extension's support directory rather than a temp directory, because a stored conversation keeps the path and a path that survives a reboot is the difference between an old chat showing its screenshot and pointing at nothing.
|
||||||
|
|
||||||
|
**Capture runs in a `no-view` command, because the chat view owns a window.** This is the part that is easy to get wrong. `activateEntrypoint` calls `showWindow()` when it activates a command from a closed launcher, so the window is already on screen before any extension code runs, and a region selector started from a view is dragging a rectangle around a panel parked in the middle of the screen. So `snip` is declared `no-view`, calls `closeMainWindow()` as its first statement, captures with nothing in the way, and only then brings the window back by launching `chat` with the file. That command is also the single capture path: the chat's own action launches `snip` rather than capturing inline, so there is one implementation and one place that deals with the window.
|
||||||
|
|
||||||
|
The hand-off goes through `snip` specifically rather than straight to `chat` on purpose. `activateEntrypoint` treats a launch of the command that is *already active* as a toggle and closes the window instead of reopening it (`toggleIfAlreadyActive` defaults to true, and a launch cannot override it). Relaying through a different, no-view command means never asking for `chat` while `chat` is active, so the toggle branch is unreachable. The chat's own snip action is also hidden while a turn is streaming, because the hand-off unloads the view and would abandon an answer mid-sentence.
|
||||||
|
|
||||||
|
**Image support is a fact from the API, not a guess from the model name, and it is cached.** `GET /api/tags` returns names, sizes and quantization levels and says nothing about capability, so the check is a `POST /api/show` that reads the `vision` flag. That is a round trip per model, so the answer is persisted under a key built from the host and model name, which means switching model invalidates it by construction. The state is a tri-state rather than a boolean: a failed request is `"unknown"`, never `"no"`, because telling someone their model cannot see when the network merely dropped is a false claim about their setup that sends them shopping for a model they do not need. The snip command settles this *before* the selector appears, so a model that cannot read images never costs the user a selection gesture, and the two outcomes get different HUD wording. The in-chat action still fails closed: only a confirmed `"yes"` offers it.
|
||||||
|
|
||||||
|
**The capture is never a tool.** It stays in the action panel and on a keybinding, so it only ever runs because a person asked for it. A model that could decide to screenshot the screen is a different risk profile from one that answers questions, which is the same line the read-only tools already sit behind.
|
||||||
|
|
||||||
|
**An attached image is sent once and never replayed.** It is the largest thing a request can carry, so `buildMessages` attaches it to the question it belongs to and drops it afterwards, exactly as it already drops tool output. The cost is that a follow-up about a screenshot needs the image attached again; the alternative is re-sending every screenshot in the conversation on every turn. Only the path and mime type go into storage: `MAX_STORED_CHARS_PER_TURN` exists so one turn cannot dominate `LocalStorage`, and base64 would blow past it on a single 1080p capture.
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
@@ -183,7 +240,8 @@ npm test
|
|||||||
- **Actions.** Opening links, clipboard access, launching apps. These need an approval step first. An assistant that can silently act on your machine is a different risk profile from one that answers questions, and this release does not pretend otherwise.
|
- **Actions.** Opening links, clipboard access, launching apps. These need an approval step first. An assistant that can silently act on your machine is a different risk profile from one that answers questions, and this release does not pretend otherwise.
|
||||||
- **`@` mentions** for files and clipboard content.
|
- **`@` mentions** for files and clipboard content.
|
||||||
- **Per-conversation model** overrides.
|
- **Per-conversation model** overrides.
|
||||||
- **Image input**, which Ollama's chat API already supports.
|
- **Annotations on a snip.** The capture backends are there, but drawing on the result before sending it needs a second tool and a wait for the editor to save.
|
||||||
|
- **Follow-ups that keep the image.** Replaying an attachment is a token cost that has to be made opt-in first.
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
|
|||||||
@@ -15,6 +15,13 @@
|
|||||||
"description": "Ask Lam anything. Answers stream in as they are generated, your session history is kept, and Lam can search the web when it needs current information.",
|
"description": "Ask Lam anything. Answers stream in as they are generated, your session history is kept, and Lam can search the web when it needs current information.",
|
||||||
"mode": "view"
|
"mode": "view"
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"name": "snip",
|
||||||
|
"title": "Snip to Lam",
|
||||||
|
"subtitle": "Lam Agent",
|
||||||
|
"description": "Capture a region of the screen and start a new chat about it. The launcher window closes first so it cannot end up in the shot. Offered only when the model in preferences supports image input.",
|
||||||
|
"mode": "no-view"
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"name": "sessions",
|
"name": "sessions",
|
||||||
"title": "Chat Sessions",
|
"title": "Chat Sessions",
|
||||||
|
|||||||
@@ -46,3 +46,26 @@ export function __setPreferenceValues(values: Record<string, unknown>): void {
|
|||||||
export function getPreferenceValues<T = Record<string, unknown>>(): T {
|
export function getPreferenceValues<T = Record<string, unknown>>(): T {
|
||||||
return preferenceOverride as T;
|
return preferenceOverride as T;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The runtime `environment`. Only `supportPath` is exercised, by the snip
|
||||||
|
* directory, and a stable fake path keeps those assertions independent of where
|
||||||
|
* the test happens to run.
|
||||||
|
*/
|
||||||
|
export const environment = {
|
||||||
|
supportPath: "/tmp/lam-agent-support",
|
||||||
|
assetsPath: "/tmp/lam-agent-assets",
|
||||||
|
extensionName: "lam-agent",
|
||||||
|
ownerOrAuthorName: "adrianbonpin",
|
||||||
|
commandName: "chat",
|
||||||
|
commandMode: "view" as const,
|
||||||
|
isDevelopment: true,
|
||||||
|
appearance: "dark" as const,
|
||||||
|
theme: "dark" as const,
|
||||||
|
textSize: "medium" as const,
|
||||||
|
launchType: "userInitiated" as const,
|
||||||
|
raycastVersion: "0.0.0",
|
||||||
|
vicinaeVersion: { tag: "test", commit: "test" },
|
||||||
|
isRaycast: false,
|
||||||
|
canAccess: () => true,
|
||||||
|
};
|
||||||
|
|||||||
@@ -165,7 +165,17 @@ describe("module wiring", () => {
|
|||||||
// These modules must stay renderer-free: they are pure logic, they are what
|
// These modules must stay renderer-free: they are pure logic, they are what
|
||||||
// the unit tests cover, and nothing in them should need a Vicinae runtime.
|
// the unit tests cover, and nothing in them should need a Vicinae runtime.
|
||||||
// (`sessions.ts` is deliberately excluded: it legitimately needs LocalStorage.)
|
// (`sessions.ts` is deliberately excluded: it legitimately needs LocalStorage.)
|
||||||
for (const file of ["agent.ts", "ollama.ts", "tools.ts", "location.ts", "history.ts", "markdown.ts"]) {
|
for (const file of [
|
||||||
|
"agent.ts",
|
||||||
|
"ollama.ts",
|
||||||
|
"tools.ts",
|
||||||
|
"location.ts",
|
||||||
|
"history.ts",
|
||||||
|
"markdown.ts",
|
||||||
|
"capture.ts",
|
||||||
|
"attachments.ts",
|
||||||
|
"startup.ts",
|
||||||
|
]) {
|
||||||
const source = readFileSync(path.resolve(__dirname, `../lib/${file}`), "utf8");
|
const source = readFileSync(path.resolve(__dirname, `../lib/${file}`), "utf8");
|
||||||
expect(source, `${file} should not import @vicinae/api`).not.toMatch(/from\s+"@vicinae\/api"/);
|
expect(source, `${file} should not import @vicinae/api`).not.toMatch(/from\s+"@vicinae\/api"/);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,177 @@
|
|||||||
|
import { mkdtemp, rm, writeFile } from "node:fs/promises";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { afterEach, describe, expect, it } from "vitest";
|
||||||
|
import { imageSource, mimeForPath, pngSize, previewBox, readAttachment } from "../lib/attachments";
|
||||||
|
|
||||||
|
const tempDirs: string[] = [];
|
||||||
|
|
||||||
|
/** A 24-byte PNG header, which is all `pngSize` reads. */
|
||||||
|
function pngHeader(width: number, height: number): Buffer {
|
||||||
|
const bytes = Buffer.alloc(24);
|
||||||
|
Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]).copy(bytes, 0);
|
||||||
|
bytes.writeUInt32BE(13, 8);
|
||||||
|
bytes.write("IHDR", 12, "ascii");
|
||||||
|
bytes.writeUInt32BE(width, 16);
|
||||||
|
bytes.writeUInt32BE(height, 20);
|
||||||
|
return bytes;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function tempDir(): Promise<string> {
|
||||||
|
const dir = await mkdtemp(join(tmpdir(), "lam-attach-"));
|
||||||
|
tempDirs.push(dir);
|
||||||
|
return dir;
|
||||||
|
}
|
||||||
|
|
||||||
|
afterEach(async () => {
|
||||||
|
await Promise.all(tempDirs.splice(0).map((dir) => rm(dir, { recursive: true, force: true })));
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("mimeForPath", () => {
|
||||||
|
it("names the image types Ollama accepts", () => {
|
||||||
|
expect(mimeForPath("/tmp/snip.png")).toBe("image/png");
|
||||||
|
expect(mimeForPath("/tmp/photo.JPG")).toBe("image/jpeg");
|
||||||
|
expect(mimeForPath("/tmp/photo.jpeg")).toBe("image/jpeg");
|
||||||
|
expect(mimeForPath("/tmp/shot.webp")).toBe("image/webp");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not guess a type for anything else", () => {
|
||||||
|
expect(mimeForPath("/tmp/notes.txt")).toBe("application/octet-stream");
|
||||||
|
expect(mimeForPath("/tmp/noextension")).toBe("application/octet-stream");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("pngSize", () => {
|
||||||
|
it("reads the dimensions out of the IHDR chunk", () => {
|
||||||
|
expect(pngSize(pngHeader(1600, 400))).toEqual({ width: 1600, height: 400 });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses anything that is not a PNG", () => {
|
||||||
|
// Guessing at another format would put a wrong aspect ratio into the layout.
|
||||||
|
expect(pngSize(Buffer.from("not a png at all, just some text"))).toBeUndefined();
|
||||||
|
expect(pngSize(Buffer.alloc(0))).toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses a header that is present but truncated", () => {
|
||||||
|
expect(pngSize(pngHeader(10, 10).subarray(0, 20))).toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses degenerate dimensions", () => {
|
||||||
|
expect(pngSize(pngHeader(0, 100))).toBeUndefined();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("previewBox", () => {
|
||||||
|
it("fits a wide capture to the width limit and keeps the aspect ratio", () => {
|
||||||
|
// This is the case that produced the dead space: 4:1 at full pane width was
|
||||||
|
// letterboxed into a 200px-tall container it barely used.
|
||||||
|
expect(previewBox(1600, 400)).toEqual({ width: 400, height: 100 });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("fits a 16:9 capture to the height limit instead", () => {
|
||||||
|
expect(previewBox(1920, 1080)).toEqual({ width: 356, height: 200 });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("fits a portrait capture by its height", () => {
|
||||||
|
expect(previewBox(400, 1600)).toEqual({ width: 50, height: 200 });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not blow a small capture up into a blurry one", () => {
|
||||||
|
expect(previewBox(100, 50)).toEqual({ width: 100, height: 50 });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("has nothing to say when the size is unknown", () => {
|
||||||
|
expect(previewBox(undefined, undefined)).toBeUndefined();
|
||||||
|
expect(previewBox(1600, undefined)).toBeUndefined();
|
||||||
|
expect(previewBox(1600, 0)).toBeUndefined();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("imageSource", () => {
|
||||||
|
it("produces a file URL, which is the only scheme the renderer resolves for a local file", () => {
|
||||||
|
expect(imageSource({ path: "/tmp/snip.png" })).toBe("file:///tmp/snip.png");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("escapes a path with a space, which the user's own screenshot directory has", () => {
|
||||||
|
expect(imageSource({ path: "/Users/x/Pictures/Screenshot 2026-10-04.png" })).toBe(
|
||||||
|
"file:///Users/x/Pictures/Screenshot%202026-10-04.png",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("escapes non-ascii", () => {
|
||||||
|
expect(imageSource({ path: "/tmp/caf\u00e9.png" })).toBe("file:///tmp/caf%C3%A9.png");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps a bare path out of the markdown, because that resolves to nothing", () => {
|
||||||
|
expect(imageSource({ path: "/tmp/snip.png" })).not.toBe("/tmp/snip.png");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("carries the size hints the renderer reads", () => {
|
||||||
|
// `parseImageSize` looks for these names, and resolves the path from the URL
|
||||||
|
// before the query, so the hints never reach the file name.
|
||||||
|
expect(imageSource({ path: "/tmp/snip.png", width: 1600, height: 400 })).toBe(
|
||||||
|
"file:///tmp/snip.png?raycast-width=400&raycast-height=100",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("omits the hints when the dimensions are unknown", () => {
|
||||||
|
expect(imageSource({ path: "/tmp/snip.png", width: undefined, height: undefined })).toBe(
|
||||||
|
"file:///tmp/snip.png",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("readAttachment", () => {
|
||||||
|
it("reads the bytes as bare base64, with no data URL prefix", async () => {
|
||||||
|
const dir = await tempDir();
|
||||||
|
const path = join(dir, "snip.png");
|
||||||
|
await writeFile(path, Buffer.from([0x89, 0x50, 0x4e, 0x47]));
|
||||||
|
|
||||||
|
const attachment = await readAttachment(path);
|
||||||
|
|
||||||
|
// Ollama wants the payload bare. A `data:image/png;base64,` prefix makes it
|
||||||
|
// reject the image, and the mistake is invisible until a request comes back
|
||||||
|
// with an error that says nothing about encoding.
|
||||||
|
expect(attachment.base64).toBe("iVBORw==");
|
||||||
|
expect(attachment.base64).not.toContain("data:");
|
||||||
|
expect(attachment.mime).toBe("image/png");
|
||||||
|
expect(attachment.path).toBe(path);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("round-trips a payload larger than one base64 padding boundary", async () => {
|
||||||
|
const dir = await tempDir();
|
||||||
|
const path = join(dir, "snip.png");
|
||||||
|
const bytes = Buffer.alloc(300, 7);
|
||||||
|
await writeFile(path, bytes);
|
||||||
|
|
||||||
|
const attachment = await readAttachment(path);
|
||||||
|
|
||||||
|
expect(Buffer.from(attachment.base64, "base64")).toEqual(bytes);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps the dimensions, which the transcript needs long after this returns", async () => {
|
||||||
|
const dir = await tempDir();
|
||||||
|
const path = join(dir, "snip.png");
|
||||||
|
await writeFile(path, pngHeader(1600, 400));
|
||||||
|
|
||||||
|
const attachment = await readAttachment(path);
|
||||||
|
|
||||||
|
expect(attachment.width).toBe(1600);
|
||||||
|
expect(attachment.height).toBe(400);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("omits the dimensions for an image it cannot measure", async () => {
|
||||||
|
const dir = await tempDir();
|
||||||
|
const path = join(dir, "snip.png");
|
||||||
|
await writeFile(path, Buffer.from([0x89, 0x50, 0x4e, 0x47]));
|
||||||
|
|
||||||
|
const attachment = await readAttachment(path);
|
||||||
|
|
||||||
|
expect(attachment.width).toBeUndefined();
|
||||||
|
expect(attachment.height).toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("fails loudly when the file is gone", async () => {
|
||||||
|
await expect(readAttachment("/nonexistent/snip.png")).rejects.toThrow();
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,329 @@
|
|||||||
|
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||||
|
import {
|
||||||
|
CAPTURE_BACKENDS,
|
||||||
|
CapturePermissionError,
|
||||||
|
CaptureToolError,
|
||||||
|
NoCaptureToolError,
|
||||||
|
captureRegion,
|
||||||
|
selectBackend,
|
||||||
|
snipFileName,
|
||||||
|
type CaptureBackend,
|
||||||
|
} from "../lib/capture";
|
||||||
|
|
||||||
|
/** A `PATH` containing exactly the named tools. */
|
||||||
|
function pathWith(...tools: string[]) {
|
||||||
|
const present = new Set(tools);
|
||||||
|
return async (tool: string) => present.has(tool);
|
||||||
|
}
|
||||||
|
|
||||||
|
function backendNamed(name: string): CaptureBackend {
|
||||||
|
const backend = CAPTURE_BACKENDS.find((candidate) => candidate.name === name);
|
||||||
|
if (!backend) throw new Error(`no backend named ${name}`);
|
||||||
|
return backend;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A tool that ran, said nothing, and produced no image. */
|
||||||
|
function silent(): { stdout: Buffer; stderr: string } {
|
||||||
|
return { stdout: Buffer.alloc(0), stderr: "" };
|
||||||
|
}
|
||||||
|
|
||||||
|
const tempDirs: string[] = [];
|
||||||
|
|
||||||
|
async function tempDir(): Promise<string> {
|
||||||
|
const dir = await mkdtemp(join(tmpdir(), "lam-snip-"));
|
||||||
|
tempDirs.push(dir);
|
||||||
|
return dir;
|
||||||
|
}
|
||||||
|
|
||||||
|
afterEach(async () => {
|
||||||
|
await Promise.all(tempDirs.splice(0).map((dir) => rm(dir, { recursive: true, force: true })));
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("snipFileName", () => {
|
||||||
|
it("is sortable and filesystem-safe", () => {
|
||||||
|
expect(snipFileName(new Date(2026, 9, 4, 14, 59, 26))).toBe("snip-20261004-145926.png");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("zero-pads every field so names sort as text", () => {
|
||||||
|
expect(snipFileName(new Date(2026, 0, 1, 0, 0, 0))).toBe("snip-20260101-000000.png");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("selectBackend", () => {
|
||||||
|
it("returns nothing when no tool is installed", async () => {
|
||||||
|
expect(await selectBackend(pathWith())).toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("prefers grimblast on a Hyprland machine", async () => {
|
||||||
|
const backend = await selectBackend(pathWith("grimblast", "grim", "slurp", "maim"));
|
||||||
|
expect(backend?.name).toBe("grimblast");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("falls back to grim + slurp when grimblast is absent", async () => {
|
||||||
|
const backend = await selectBackend(pathWith("grim", "slurp", "maim"));
|
||||||
|
expect(backend?.name).toBe("grim + slurp");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("needs both halves of a pair, not just one", async () => {
|
||||||
|
// grim without slurp cannot select a region, so it must not be chosen.
|
||||||
|
const backend = await selectBackend(pathWith("grim", "maim"));
|
||||||
|
expect(backend?.name).toBe("maim");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("falls back through the X11 tools", async () => {
|
||||||
|
expect((await selectBackend(pathWith("maim")))?.name).toBe("maim");
|
||||||
|
expect((await selectBackend(pathWith("scrot")))?.name).toBe("scrot");
|
||||||
|
expect((await selectBackend(pathWith("flameshot")))?.name).toBe("flameshot");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("recognises the macOS built-in tool", async () => {
|
||||||
|
expect((await selectBackend(pathWith("screencapture")))?.name).toBe("screencapture");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("can be given a shorter candidate list, which is how macOS is restricted", async () => {
|
||||||
|
const onlyScreencapture = CAPTURE_BACKENDS.filter((b) => b.name === "screencapture");
|
||||||
|
const backend = await selectBackend(pathWith("screencapture", "maim"), onlyScreencapture);
|
||||||
|
|
||||||
|
expect(backend?.name).toBe("screencapture");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("backend commands", () => {
|
||||||
|
it("runs slurp before grim and passes the geometry through", async () => {
|
||||||
|
const calls: Array<{ tool: string; args: string[] }> = [];
|
||||||
|
const run = vi.fn(async (tool: string, args: string[]) => {
|
||||||
|
calls.push({ tool, args });
|
||||||
|
return tool === "slurp" ? { stdout: Buffer.from("120,340 800x600\n"), stderr: "" } : silent();
|
||||||
|
});
|
||||||
|
|
||||||
|
await backendNamed("grim + slurp").capture({ output: "/tmp/snip.png", run });
|
||||||
|
|
||||||
|
expect(calls).toEqual([
|
||||||
|
{ tool: "slurp", args: [] },
|
||||||
|
{ tool: "grim", args: ["-g", "120,340 800x600", "/tmp/snip.png"] },
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not shell out to grim when the selection was cancelled", async () => {
|
||||||
|
const run = vi.fn(async () => silent());
|
||||||
|
|
||||||
|
await backendNamed("grim + slurp").capture({ output: "/tmp/snip.png", run });
|
||||||
|
|
||||||
|
expect(run).toHaveBeenCalledTimes(1);
|
||||||
|
expect(run).toHaveBeenCalledWith("slurp", [], expect.anything());
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps a shell pipeline out of grim by using argv", () => {
|
||||||
|
// `grim -g "$(slurp)"` is the documented one-liner, but a shell here would
|
||||||
|
// be a needless injection surface and would not survive a path with spaces.
|
||||||
|
expect(backendNamed("grim + slurp").requires).toEqual(["grim", "slurp"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("writes flameshot's stdout, because it refuses an output path", async () => {
|
||||||
|
const dir = await tempDir();
|
||||||
|
const output = join(dir, "snip.png");
|
||||||
|
const png = Buffer.from([0x89, 0x50, 0x4e, 0x47]);
|
||||||
|
|
||||||
|
await backendNamed("flameshot").capture({ output, run: async () => ({ stdout: png, stderr: "" }) });
|
||||||
|
|
||||||
|
expect(await readFile(output)).toEqual(png);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("writes nothing when flameshot returned no image", async () => {
|
||||||
|
const dir = await tempDir();
|
||||||
|
const output = join(dir, "snip.png");
|
||||||
|
|
||||||
|
await backendNamed("flameshot").capture({ output, run: async () => silent() });
|
||||||
|
|
||||||
|
await expect(readFile(output)).rejects.toThrow();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("asks screencapture for an interactive region and no sound", async () => {
|
||||||
|
const run = vi.fn(async () => silent());
|
||||||
|
|
||||||
|
await backendNamed("screencapture").capture({ output: "/tmp/snip.png", run });
|
||||||
|
|
||||||
|
expect(run).toHaveBeenCalledWith("screencapture", ["-i", "-x", "/tmp/snip.png"], expect.anything());
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("captureRegion", () => {
|
||||||
|
/** A `run` that writes a real file, which is what the success path checks for. */
|
||||||
|
function writingRun(bytes = Buffer.from([0x89, 0x50])) {
|
||||||
|
return async (_tool: string, args: string[]) => {
|
||||||
|
await writeFile(args[args.length - 1], bytes);
|
||||||
|
return silent();
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
it("writes into the directory it was given and reports the path", async () => {
|
||||||
|
const dir = await tempDir();
|
||||||
|
const output = join(dir, "snip-20261004-145926.png");
|
||||||
|
|
||||||
|
const result = await captureRegion({
|
||||||
|
outputDir: dir,
|
||||||
|
platform: "darwin",
|
||||||
|
now: new Date(2026, 9, 4, 14, 59, 26),
|
||||||
|
hasCommand: pathWith("screencapture"),
|
||||||
|
run: writingRun(Buffer.from([0x89, 0x50])),
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(result).toEqual({ path: output, bytes: 2, backend: "screencapture" });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("creates the output directory on first use", async () => {
|
||||||
|
const dir = await tempDir();
|
||||||
|
const nested = join(dir, "snips");
|
||||||
|
const now = new Date(2026, 9, 4, 14, 59, 26);
|
||||||
|
|
||||||
|
await captureRegion({
|
||||||
|
outputDir: nested,
|
||||||
|
platform: "darwin",
|
||||||
|
now,
|
||||||
|
hasCommand: pathWith("screencapture"),
|
||||||
|
run: writingRun(Buffer.from([0x89])),
|
||||||
|
});
|
||||||
|
|
||||||
|
expect((await readFile(join(nested, snipFileName(now)))).length).toBeGreaterThan(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("returns null when the user backed out and said nothing", async () => {
|
||||||
|
const dir = await tempDir();
|
||||||
|
|
||||||
|
const result = await captureRegion({
|
||||||
|
outputDir: dir,
|
||||||
|
platform: "darwin",
|
||||||
|
hasCommand: pathWith("screencapture"),
|
||||||
|
run: async () => silent(),
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(result).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("treats an empty file as a cancel rather than an attachment", async () => {
|
||||||
|
// At least one tool exits 0 on cancel, so exit codes cannot be trusted. A
|
||||||
|
// zero-byte PNG would be sent to the model as a broken image.
|
||||||
|
const dir = await tempDir();
|
||||||
|
|
||||||
|
const result = await captureRegion({
|
||||||
|
outputDir: dir,
|
||||||
|
platform: "darwin",
|
||||||
|
hasCommand: pathWith("screencapture"),
|
||||||
|
run: writingRun(Buffer.alloc(0)),
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(result).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("still succeeds when the tool also complained on stderr", async () => {
|
||||||
|
// Some tools warn on stderr and write the image anyway. The file is the
|
||||||
|
// source of truth, so a warning must not turn a good capture into an error.
|
||||||
|
const dir = await tempDir();
|
||||||
|
|
||||||
|
const result = await captureRegion({
|
||||||
|
outputDir: dir,
|
||||||
|
platform: "darwin",
|
||||||
|
hasCommand: pathWith("screencapture"),
|
||||||
|
run: async (_tool, args) => {
|
||||||
|
await writeFile(args[args.length - 1], Buffer.from([0x89]));
|
||||||
|
return { stdout: Buffer.alloc(0), stderr: "warning: something\n" };
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(result?.bytes).toBe(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("macOS Screen Recording permission", () => {
|
||||||
|
// The message is the only difference between a refused permission and a
|
||||||
|
// cancelled selection: both leave no file behind, and screencapture has been
|
||||||
|
// reported to exit 0 when the permission is missing.
|
||||||
|
const denial = "could not create image from rect\n";
|
||||||
|
|
||||||
|
it("reports a permission problem instead of a cancel when the tool exits non-zero", async () => {
|
||||||
|
const dir = await tempDir();
|
||||||
|
|
||||||
|
await expect(
|
||||||
|
captureRegion({
|
||||||
|
outputDir: dir,
|
||||||
|
platform: "darwin",
|
||||||
|
hasCommand: pathWith("screencapture"),
|
||||||
|
run: async () => {
|
||||||
|
throw new CaptureToolError("screencapture", denial);
|
||||||
|
},
|
||||||
|
}),
|
||||||
|
).rejects.toThrow(CapturePermissionError);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("reports a permission problem even when the tool exits 0", async () => {
|
||||||
|
// The documented exit-0 shape. Without reading stderr this is
|
||||||
|
// indistinguishable from Escape, which is how a silently useless feature
|
||||||
|
// gets shipped.
|
||||||
|
const dir = await tempDir();
|
||||||
|
|
||||||
|
await expect(
|
||||||
|
captureRegion({
|
||||||
|
outputDir: dir,
|
||||||
|
platform: "darwin",
|
||||||
|
hasCommand: pathWith("screencapture"),
|
||||||
|
run: async () => ({ stdout: Buffer.alloc(0), stderr: denial }),
|
||||||
|
}),
|
||||||
|
).rejects.toThrow(CapturePermissionError);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("names Vicinae rather than the extension, because that is who holds the grant", async () => {
|
||||||
|
const dir = await tempDir();
|
||||||
|
const attempt = () =>
|
||||||
|
captureRegion({
|
||||||
|
outputDir: dir,
|
||||||
|
platform: "darwin",
|
||||||
|
hasCommand: pathWith("screencapture"),
|
||||||
|
run: async () => {
|
||||||
|
throw new CaptureToolError("screencapture", denial);
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
await expect(attempt()).rejects.toMatchObject({
|
||||||
|
name: "CapturePermissionError",
|
||||||
|
hint: expect.stringContaining("Vicinae"),
|
||||||
|
});
|
||||||
|
await expect(attempt()).rejects.toMatchObject({
|
||||||
|
hint: expect.stringContaining("Screen Recording"),
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("reports any other explanation rather than swallowing it", async () => {
|
||||||
|
const dir = await tempDir();
|
||||||
|
|
||||||
|
await expect(
|
||||||
|
captureRegion({
|
||||||
|
outputDir: dir,
|
||||||
|
platform: "linux",
|
||||||
|
hasCommand: pathWith("maim"),
|
||||||
|
run: async () => {
|
||||||
|
throw new CaptureToolError("maim", "maim: cannot open display\n");
|
||||||
|
},
|
||||||
|
}),
|
||||||
|
).rejects.toThrow(CaptureToolError);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("explains what to install when nothing is available", async () => {
|
||||||
|
const dir = await tempDir();
|
||||||
|
|
||||||
|
await expect(
|
||||||
|
captureRegion({ outputDir: dir, platform: "linux", hasCommand: pathWith() }),
|
||||||
|
).rejects.toThrow(NoCaptureToolError);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not invent a backend on macOS", async () => {
|
||||||
|
const dir = await tempDir();
|
||||||
|
|
||||||
|
// maim is available in this fake PATH but is meaningless on darwin.
|
||||||
|
await expect(
|
||||||
|
captureRegion({ outputDir: dir, platform: "darwin", hasCommand: pathWith("maim") }),
|
||||||
|
).rejects.toThrow(NoCaptureToolError);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -4,6 +4,8 @@ import {
|
|||||||
labelCodeFences,
|
labelCodeFences,
|
||||||
promoteStandaloneCodeSpans,
|
promoteStandaloneCodeSpans,
|
||||||
repairCodeBlocks,
|
repairCodeBlocks,
|
||||||
|
restoreUrlLinks,
|
||||||
|
separateTables,
|
||||||
} from "../lib/code";
|
} from "../lib/code";
|
||||||
|
|
||||||
describe("inferLanguage", () => {
|
describe("inferLanguage", () => {
|
||||||
@@ -132,11 +134,207 @@ describe("promoteStandaloneCodeSpans", () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe("restoreUrlLinks", () => {
|
||||||
|
it("unwraps a fence that is nothing but URLs", () => {
|
||||||
|
// The reported case: "here are the links" arriving as a code block, so the
|
||||||
|
// one thing a URL is for, being clicked, does not work.
|
||||||
|
const input = [
|
||||||
|
"```",
|
||||||
|
"https://github.com/jasonkneen/openclicky",
|
||||||
|
"https://github.com/unn-Known1/clickyX",
|
||||||
|
"```",
|
||||||
|
].join("\n");
|
||||||
|
|
||||||
|
expect(restoreUrlLinks(input)).toBe(
|
||||||
|
[
|
||||||
|
"[https://github.com/jasonkneen/openclicky](https://github.com/jasonkneen/openclicky)",
|
||||||
|
"[https://github.com/unn-Known1/clickyX](https://github.com/unn-Known1/clickyX)",
|
||||||
|
].join("\n"),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("unwraps a URL fence even when the model labelled it", () => {
|
||||||
|
expect(restoreUrlLinks("```bash\nhttps://example.com\n```")).toBe(
|
||||||
|
"[https://example.com](https://example.com)",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("links a bare URL standing alone on a line", () => {
|
||||||
|
expect(restoreUrlLinks("https://example.com")).toBe("[https://example.com](https://example.com)");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("links a lone inline span holding a URL", () => {
|
||||||
|
expect(restoreUrlLinks("`https://example.com`")).toBe(
|
||||||
|
"[https://example.com](https://example.com)",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps a bulleted list of links a bulleted list", () => {
|
||||||
|
const input = "- https://a.example\n- https://b.example";
|
||||||
|
|
||||||
|
expect(restoreUrlLinks(input)).toBe(
|
||||||
|
"- [https://a.example](https://a.example)\n- [https://b.example](https://b.example)",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps an ordered list marker", () => {
|
||||||
|
expect(restoreUrlLinks("1. https://example.com")).toBe(
|
||||||
|
"1. [https://example.com](https://example.com)",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("drops sentence punctuation that is not part of the address", () => {
|
||||||
|
expect(restoreUrlLinks("See this:\n\nhttps://example.com.")).toContain(
|
||||||
|
"[https://example.com](https://example.com)",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("brackets a destination that contains parentheses", () => {
|
||||||
|
const url = "https://en.wikipedia.org/wiki/Foo_(bar)";
|
||||||
|
expect(restoreUrlLinks(url)).toBe(`[${url}](<${url}>)`);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("escapes brackets in the label, which an IPv6 literal contains", () => {
|
||||||
|
// Unescaped, the `]` in the label ends the link text early and the rest of
|
||||||
|
// the address is left as prose outside the link.
|
||||||
|
expect(restoreUrlLinks("http://[::1]:8080/")).toBe(
|
||||||
|
"[http://\\[::1\\]:8080/](http://[::1]:8080/)",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("never touches a fence holding real code", () => {
|
||||||
|
const input = "```bash\ncurl https://example.com | jq .\n```";
|
||||||
|
expect(restoreUrlLinks(input)).toBe(input);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("never touches a fence when one line is not a URL", () => {
|
||||||
|
const input = "```\nhttps://example.com\nnot a url\n```";
|
||||||
|
expect(restoreUrlLinks(input)).toBe(input);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("leaves a URL embedded in a sentence alone", () => {
|
||||||
|
const input = "The docs at https://example.com cover it.";
|
||||||
|
expect(restoreUrlLinks(input)).toBe(input);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("leaves a markdown link the model already wrote alone", () => {
|
||||||
|
const input = "[OpenClickly](https://github.com/jasonkneen/openclicky)";
|
||||||
|
expect(restoreUrlLinks(input)).toBe(input);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("preserves a fence that is never closed and is real code", () => {
|
||||||
|
expect(restoreUrlLinks("```\ngit log")).toBe("```\ngit log");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("is idempotent", () => {
|
||||||
|
const once = restoreUrlLinks("```\nhttps://example.com\n```");
|
||||||
|
expect(restoreUrlLinks(once)).toBe(once);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("is a no-op on ordinary prose", () => {
|
||||||
|
const input = "There are no addresses in this sentence.";
|
||||||
|
expect(restoreUrlLinks(input)).toBe(input);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("separateTables", () => {
|
||||||
|
/** The table from the answer that prompted this, verbatim. */
|
||||||
|
const table = [
|
||||||
|
"| snip | source | aspect | dead space before | box after |",
|
||||||
|
"| --- | --- | --- | --- | --- |",
|
||||||
|
"| `…153223` | 626×184 | 3.40 | **~82px** | 400×118 |",
|
||||||
|
].join("\n");
|
||||||
|
|
||||||
|
it("gives a table the blank line it needs to be a table", () => {
|
||||||
|
// GFM tables cannot interrupt a paragraph, so glued to the line above they
|
||||||
|
// render as a paragraph of pipes and dashes.
|
||||||
|
const repaired = separateTables(`Here are the numbers:\n${table}`);
|
||||||
|
|
||||||
|
expect(repaired).toBe(`Here are the numbers:\n\n${table}`);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("leaves a table alone when it already has one", () => {
|
||||||
|
const already = `Here are the numbers:\n\n${table}`;
|
||||||
|
expect(separateTables(already)).toBe(already);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("leaves a table alone when it opens the answer", () => {
|
||||||
|
expect(separateTables(table)).toBe(table);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("leaves a table alone when it follows a fence", () => {
|
||||||
|
const input = `\`\`\`bash\npwd\n\`\`\`\n${table}`;
|
||||||
|
expect(separateTables(input)).toBe(input);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not touch a table inside a fence", () => {
|
||||||
|
// A table shown as example code has to stay exactly as written, blank line
|
||||||
|
// or not.
|
||||||
|
const fenced = `\`\`\`markdown\n${table}\n\`\`\``;
|
||||||
|
expect(separateTables(fenced)).toBe(fenced);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("leaves prose containing a pipe alone", () => {
|
||||||
|
const input = "Use the pipe character:\nls | grep foo";
|
||||||
|
expect(separateTables(input)).toBe(input);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not insert a blank line before a data row", () => {
|
||||||
|
// Only the header starts a table. A blank line inside one would split it.
|
||||||
|
const input = `text\n${table}\n| a | b | c | d | e |`;
|
||||||
|
expect(separateTables(input)).toBe(`text\n\n${table}\n| a | b | c | d | e |`);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("recognises a one-dash delimiter, which GFM allows", () => {
|
||||||
|
const minimal = "| a | b |\n| - | - |\n| 1 | 2 |";
|
||||||
|
expect(separateTables(`text\n${minimal}`)).toBe(`text\n\n${minimal}`);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("recognises alignment colons", () => {
|
||||||
|
const aligned = "| a | b |\n|:---|:---:|\n| 1 | 2 |";
|
||||||
|
expect(separateTables(`text\n${aligned}`)).toBe(`text\n\n${aligned}`);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("is idempotent", () => {
|
||||||
|
const once = separateTables(`text\n${table}`);
|
||||||
|
expect(separateTables(once)).toBe(once);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
describe("repairCodeBlocks", () => {
|
describe("repairCodeBlocks", () => {
|
||||||
it("promotes a lone command and labels it in one pass", () => {
|
it("promotes a lone command and labels it in one pass", () => {
|
||||||
expect(repairCodeBlocks("`pwd`")).toBe("```bash\npwd\n```");
|
expect(repairCodeBlocks("`pwd`")).toBe("```bash\npwd\n```");
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("leaves a table exactly as the model wrote it", () => {
|
||||||
|
// The whole repair pass runs over every answer, and a table has to survive
|
||||||
|
// it untouched: pipes are not fences, not links, and not lone spans.
|
||||||
|
const table = [
|
||||||
|
"| snip | source | aspect | box after |",
|
||||||
|
"| --- | --- | --- | --- |",
|
||||||
|
"| `…153223` | 626×184 | 3.40 | 400×118 |",
|
||||||
|
"| [docs](https://example.com) | `pwd` | https://example.com | 400×60 |",
|
||||||
|
].join("\n");
|
||||||
|
|
||||||
|
expect(repairCodeBlocks(table)).toBe(table);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not fence a lone URL", () => {
|
||||||
|
// Restoring links runs before promotion, so a recovered URL never becomes a
|
||||||
|
// fence again. Fencing it here was the bug the user saw.
|
||||||
|
const repaired = repairCodeBlocks("`https://example.com`");
|
||||||
|
|
||||||
|
expect(repaired).not.toContain("```");
|
||||||
|
expect(repaired).toBe("[https://example.com](https://example.com)");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("unwraps a fenced list of links and leaves no fence behind", () => {
|
||||||
|
const repaired = repairCodeBlocks("```\nhttps://a.example\nhttps://b.example\n```");
|
||||||
|
|
||||||
|
expect(repaired).not.toContain("```");
|
||||||
|
expect(repaired).toContain("[https://a.example](https://a.example)");
|
||||||
|
});
|
||||||
|
|
||||||
it("still labels a bare multi-line fence", () => {
|
it("still labels a bare multi-line fence", () => {
|
||||||
expect(repairCodeBlocks("```\ngit status\n```")).toBe("```bash\ngit status\n```");
|
expect(repairCodeBlocks("```\ngit status\n```")).toBe("```bash\ngit status\n```");
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ import { buildMessages, composeSystemPrompt } from "../lib/history";
|
|||||||
import type { ChatTurn } from "../lib/types";
|
import type { ChatTurn } from "../lib/types";
|
||||||
|
|
||||||
function turn(prompt: string, response: string): ChatTurn {
|
function turn(prompt: string, response: string): ChatTurn {
|
||||||
return { id: prompt, prompt, response, thinking: "", model: "m", createdAt: 0, toolCalls: [] };
|
return { id: prompt, prompt, response, thinking: "", model: "m", createdAt: 0, toolCalls: [], attachments: [] };
|
||||||
}
|
}
|
||||||
|
|
||||||
describe("buildMessages", () => {
|
describe("buildMessages", () => {
|
||||||
@@ -66,3 +66,37 @@ describe("composeSystemPrompt", () => {
|
|||||||
expect(composeSystemPrompt("Be terse.", false)).not.toContain(RECENCY_HINT);
|
expect(composeSystemPrompt("Be terse.", false)).not.toContain(RECENCY_HINT);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe("buildMessages images", () => {
|
||||||
|
it("attaches images to the question being asked", () => {
|
||||||
|
const messages = buildMessages("SYSTEM", [], "what is this?", ["BASE64"]);
|
||||||
|
|
||||||
|
expect(messages[messages.length - 1]).toEqual({
|
||||||
|
role: "user",
|
||||||
|
content: "what is this?",
|
||||||
|
images: ["BASE64"],
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("omits the images key entirely when there is no attachment", () => {
|
||||||
|
// The key has to be absent, not an empty array: Ollama treats `images: []`
|
||||||
|
// differently from no images at all on some models.
|
||||||
|
const messages = buildMessages("SYSTEM", [], "hello");
|
||||||
|
|
||||||
|
expect(messages[messages.length - 1]).toEqual({ role: "user", content: "hello" });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("attaches several images to one question", () => {
|
||||||
|
const messages = buildMessages("SYSTEM", [], "compare these", ["A", "B"]);
|
||||||
|
|
||||||
|
expect(messages[messages.length - 1].images).toEqual(["A", "B"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not replay images from earlier turns", () => {
|
||||||
|
// A screenshot is the largest thing a request can carry, and re-sending it
|
||||||
|
// with every later question is exactly what dropping tool output avoids.
|
||||||
|
const messages = buildMessages("SYSTEM", [turn("earlier", "answer")], "follow up");
|
||||||
|
|
||||||
|
expect(messages.some((message) => message.images !== undefined)).toBe(false);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ function message(overrides: Partial<ThreadMessage> & Pick<ThreadMessage, "role">
|
|||||||
isStreaming: false,
|
isStreaming: false,
|
||||||
thinking: "",
|
thinking: "",
|
||||||
toolCalls: [],
|
toolCalls: [],
|
||||||
|
attachments: [],
|
||||||
...overrides,
|
...overrides,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
@@ -74,9 +75,9 @@ describe("buildThreadMarkdown", () => {
|
|||||||
expect(markdown.indexOf("second question")).toBeLessThan(markdown.indexOf("first answer"));
|
expect(markdown.indexOf("second question")).toBeLessThan(markdown.indexOf("first answer"));
|
||||||
});
|
});
|
||||||
|
|
||||||
it("renders no images, because markdown cannot resolve them", () => {
|
it("renders no images for a message that has no attachment", () => {
|
||||||
// Images render block-level and centred at up to 200px (`MdImage.qml`), and
|
// Images only ever come from an attachment this extension took itself. A URL
|
||||||
// every candidate icon source fails. Keep the transcript text-only.
|
// the model invented is not resolvable and renders as an empty block.
|
||||||
const markdown = buildThreadMarkdown([
|
const markdown = buildThreadMarkdown([
|
||||||
message({ role: "user", text: "q" }),
|
message({ role: "user", text: "q" }),
|
||||||
message({ role: "assistant", text: "a" }),
|
message({ role: "assistant", text: "a" }),
|
||||||
@@ -146,6 +147,34 @@ describe("buildThreadMarkdown", () => {
|
|||||||
expect(markdown).not.toContain("```bash");
|
expect(markdown).not.toContain("```bash");
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("turns a fenced list of URLs into links, because a fence is not clickable", () => {
|
||||||
|
// The reported case: asked for alternatives, the model answered with a code
|
||||||
|
// block of bare GitHub addresses. `MdCodeBlock.qml` draws that as a dark
|
||||||
|
// panel with a Copy button and no link handling, so the address could not be
|
||||||
|
// opened. Only `CMARK_NODE_LINK` becomes an `<a href>` the selection
|
||||||
|
// controller can activate.
|
||||||
|
const markdown = buildThreadMarkdown([
|
||||||
|
message({
|
||||||
|
role: "assistant",
|
||||||
|
text: "GitHub links:\n\n```\nhttps://github.com/jasonkneen/openclicky\nhttps://github.com/unn-Known1/clickyX\n```",
|
||||||
|
}),
|
||||||
|
]);
|
||||||
|
|
||||||
|
expect(markdown).not.toContain("```");
|
||||||
|
expect(markdown).toContain(
|
||||||
|
"[https://github.com/jasonkneen/openclicky](https://github.com/jasonkneen/openclicky)",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("links a lone URL the model put in inline code", () => {
|
||||||
|
const markdown = buildThreadMarkdown([
|
||||||
|
message({ role: "assistant", text: "`https://example.com`" }),
|
||||||
|
]);
|
||||||
|
|
||||||
|
expect(markdown).not.toContain("```");
|
||||||
|
expect(markdown).toContain("[https://example.com](https://example.com)");
|
||||||
|
});
|
||||||
|
|
||||||
it("lists tools on one quiet line, flagging failures", () => {
|
it("lists tools on one quiet line, flagging failures", () => {
|
||||||
const markdown = buildThreadMarkdown([
|
const markdown = buildThreadMarkdown([
|
||||||
message({
|
message({
|
||||||
@@ -212,7 +241,17 @@ describe("buildThreadMarkdown", () => {
|
|||||||
|
|
||||||
describe("sessionToPlainText", () => {
|
describe("sessionToPlainText", () => {
|
||||||
function turn(prompt: string, response: string, error?: string): ChatTurn {
|
function turn(prompt: string, response: string, error?: string): ChatTurn {
|
||||||
return { id: prompt, prompt, response, thinking: "", model: "m", createdAt: 0, toolCalls: [], error };
|
return {
|
||||||
|
id: prompt,
|
||||||
|
prompt,
|
||||||
|
response,
|
||||||
|
thinking: "",
|
||||||
|
model: "m",
|
||||||
|
createdAt: 0,
|
||||||
|
toolCalls: [],
|
||||||
|
attachments: [],
|
||||||
|
error,
|
||||||
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
it("returns an empty string for an empty conversation", () => {
|
it("returns an empty string for an empty conversation", () => {
|
||||||
@@ -247,3 +286,69 @@ describe("sessionToPlainText", () => {
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe("attachments in the transcript", () => {
|
||||||
|
const screenshot = { path: "/tmp/snip.png", mime: "image/png" };
|
||||||
|
|
||||||
|
it("shows the attachment instead of just naming it", () => {
|
||||||
|
// Vicinae resolves a local file through `ImageURL::local`, so the capture is
|
||||||
|
// rendered rather than described. `MdImage.qml` caps the height at 200px, so
|
||||||
|
// it is a thumbnail, and the alt text is what is left if the file is gone.
|
||||||
|
const markdown = buildThreadMarkdown([
|
||||||
|
message({ role: "user", text: "what is this?", attachments: [screenshot] }),
|
||||||
|
]);
|
||||||
|
|
||||||
|
expect(markdown).toContain("");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keeps the question above its image", () => {
|
||||||
|
const markdown = buildThreadMarkdown([
|
||||||
|
message({ role: "user", text: "what is this?", attachments: [screenshot] }),
|
||||||
|
]);
|
||||||
|
|
||||||
|
expect(markdown.indexOf("> what is this?")).toBeLessThan(markdown.indexOf("!["));
|
||||||
|
});
|
||||||
|
|
||||||
|
it("renders every attachment on a turn", () => {
|
||||||
|
const markdown = buildThreadMarkdown([
|
||||||
|
message({
|
||||||
|
role: "user",
|
||||||
|
text: "compare",
|
||||||
|
attachments: [screenshot, { path: "/tmp/second.png", mime: "image/png" }],
|
||||||
|
}),
|
||||||
|
]);
|
||||||
|
|
||||||
|
expect(markdown).toContain("file:///tmp/snip.png");
|
||||||
|
expect(markdown).toContain("file:///tmp/second.png");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("never puts an image on an answer, which has no attachments", () => {
|
||||||
|
const markdown = buildThreadMarkdown([
|
||||||
|
message({ role: "assistant", text: "a cat", attachments: [screenshot] }),
|
||||||
|
]);
|
||||||
|
|
||||||
|
expect(markdown).not.toContain("![");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("says nothing about attachments when there are none", () => {
|
||||||
|
const markdown = buildThreadMarkdown([message({ role: "user", text: "q" })]);
|
||||||
|
|
||||||
|
expect(markdown).not.toContain("Screenshot");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("notes the screenshot when a conversation is copied as plain text", () => {
|
||||||
|
const question = {
|
||||||
|
id: "t1",
|
||||||
|
prompt: "what is this?",
|
||||||
|
response: "a cat",
|
||||||
|
thinking: "",
|
||||||
|
model: "m",
|
||||||
|
createdAt: 0,
|
||||||
|
toolCalls: [],
|
||||||
|
attachments: [screenshot],
|
||||||
|
};
|
||||||
|
const text = sessionToPlainText([question]);
|
||||||
|
|
||||||
|
expect(text).toContain("_(screenshot attached)_");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
import { describe, expect, it } from "vitest";
|
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||||
import { normalizeHost, parseNdjsonStream } from "../lib/ollama";
|
import { modelCapabilities, normalizeHost, parseNdjsonStream, supportsVision } from "../lib/ollama";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The NDJSON reader is the part of the client most likely to break in a way
|
* The NDJSON reader is the part of the client most likely to break in a way
|
||||||
@@ -84,3 +84,70 @@ describe("parseNdjsonStream", () => {
|
|||||||
expect(await collect<unknown>([])).toEqual([]);
|
expect(await collect<unknown>([])).toEqual([]);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe("supportsVision", () => {
|
||||||
|
it("reads the capability Ollama reports for image input", () => {
|
||||||
|
expect(supportsVision(["completion", "tools", "vision"])).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("is false for a text-only model", () => {
|
||||||
|
expect(supportsVision(["completion", "tools"])).toBe(false);
|
||||||
|
expect(supportsVision([])).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not match a capability that merely contains the word", () => {
|
||||||
|
expect(supportsVision(["visions"])).toBe(false);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("modelCapabilities", () => {
|
||||||
|
const fetchMock = vi.fn();
|
||||||
|
|
||||||
|
beforeEach(() => {
|
||||||
|
fetchMock.mockReset();
|
||||||
|
vi.stubGlobal("fetch", fetchMock);
|
||||||
|
});
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
vi.unstubAllGlobals();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("posts to /api/show and returns the capability list", async () => {
|
||||||
|
fetchMock.mockResolvedValueOnce(
|
||||||
|
new Response(JSON.stringify({ capabilities: ["completion", "vision"] }), { status: 200 }),
|
||||||
|
);
|
||||||
|
|
||||||
|
await expect(
|
||||||
|
modelCapabilities({ host: "https://ollama.com", apiKey: "k", model: "gemma4" }),
|
||||||
|
).resolves.toEqual(["completion", "vision"]);
|
||||||
|
|
||||||
|
const [url, init] = fetchMock.mock.calls[0];
|
||||||
|
expect(url).toBe("https://ollama.com/api/show");
|
||||||
|
expect(init.headers.Authorization).toBe("Bearer k");
|
||||||
|
expect(JSON.parse(init.body)).toEqual({ model: "gemma4" });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("returns nothing when a model reports no capabilities", async () => {
|
||||||
|
fetchMock.mockResolvedValueOnce(new Response(JSON.stringify({}), { status: 200 }));
|
||||||
|
|
||||||
|
await expect(
|
||||||
|
modelCapabilities({ host: "https://ollama.com", apiKey: "k", model: "gemma4" }),
|
||||||
|
).resolves.toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("turns a 401 into the key hint rather than a bare status code", async () => {
|
||||||
|
fetchMock.mockResolvedValueOnce(new Response("unauthorized", { status: 401 }));
|
||||||
|
|
||||||
|
await expect(
|
||||||
|
modelCapabilities({ host: "https://ollama.com", apiKey: "bad", model: "gemma4" }),
|
||||||
|
).rejects.toMatchObject({ status: 401, hint: expect.stringContaining("ollama.com/settings/keys") });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("normalises the host so the path does not double up a slash", async () => {
|
||||||
|
fetchMock.mockResolvedValueOnce(new Response(JSON.stringify({ capabilities: [] }), { status: 200 }));
|
||||||
|
|
||||||
|
await modelCapabilities({ host: "http://localhost:11434/", apiKey: "", model: "gemma4" });
|
||||||
|
|
||||||
|
expect(fetchMock.mock.calls[0][0]).toBe("http://localhost:11434/api/show");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|||||||
@@ -24,6 +24,7 @@ function turn(index: number, response = `answer ${index}`): ChatTurn {
|
|||||||
model: "test-model",
|
model: "test-model",
|
||||||
createdAt: 1_000 + index,
|
createdAt: 1_000 + index,
|
||||||
toolCalls: [],
|
toolCalls: [],
|
||||||
|
attachments: [],
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,57 @@
|
|||||||
|
import { describe, expect, it } from "vitest";
|
||||||
|
import { readSnipContext, snipDirectory } from "../lib/snip";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The snip handoff crosses a command boundary as a serialized launch context, so
|
||||||
|
* every field arrives as unknown. This is the boundary where a wrong type would
|
||||||
|
* otherwise reach the file system as a path.
|
||||||
|
*/
|
||||||
|
|
||||||
|
describe("readSnipContext", () => {
|
||||||
|
it("reads the fields the snip command writes", () => {
|
||||||
|
expect(
|
||||||
|
readSnipContext({
|
||||||
|
snipPath: "/tmp/snip.png",
|
||||||
|
snipError: "no tool",
|
||||||
|
sessionId: "abc123",
|
||||||
|
}),
|
||||||
|
).toEqual({ snipPath: "/tmp/snip.png", snipError: "no tool", sessionId: "abc123" });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("returns nothing for a launch with no context at all", () => {
|
||||||
|
expect(readSnipContext(undefined)).toEqual({});
|
||||||
|
expect(readSnipContext(null)).toEqual({});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("ignores a context that is not an object", () => {
|
||||||
|
expect(readSnipContext("snip.png")).toEqual({});
|
||||||
|
expect(readSnipContext(42)).toEqual({});
|
||||||
|
expect(readSnipContext(["/tmp/snip.png"])).toEqual({});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("drops a value that is not a string rather than passing it on", () => {
|
||||||
|
// A number or object here would reach `readAttachment` as a path.
|
||||||
|
expect(readSnipContext({ snipPath: 42, sessionId: { id: "x" } })).toEqual({});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("treats an empty string as absent", () => {
|
||||||
|
expect(readSnipContext({ snipPath: "", sessionId: "" })).toEqual({});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("ignores unknown keys, so a stray field cannot widen the contract", () => {
|
||||||
|
expect(readSnipContext({ snipPath: "/tmp/snip.png", somethingElse: "/etc/passwd" })).toEqual({
|
||||||
|
snipPath: "/tmp/snip.png",
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("reads a payload that only names a conversation, which is a cancelled snip", () => {
|
||||||
|
expect(readSnipContext({ sessionId: "abc123" })).toEqual({ sessionId: "abc123" });
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("snipDirectory", () => {
|
||||||
|
it("keeps snips under the extension's support directory, not a temp directory", () => {
|
||||||
|
// A stored conversation holds the path, so it has to survive a reboot.
|
||||||
|
expect(snipDirectory()).toBe("/tmp/lam-agent-support/snips");
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,203 @@
|
|||||||
|
import { describe, expect, it, vi } from "vitest";
|
||||||
|
import { isHandoff, resolveStartup, type StartupPorts } from "../lib/startup";
|
||||||
|
import type { PendingAttachment, Session, SessionSummary } from "../lib/types";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The chat view's startup is the part that kept breaking: three asynchronous
|
||||||
|
* paths (a pushed session, a handed-over capture, an idle resume) that all have
|
||||||
|
* to end in a rendered chat. Two of those bugs shipped, so the decision tree is
|
||||||
|
* tested here rather than discovered from a screenshot of a spinner.
|
||||||
|
*/
|
||||||
|
|
||||||
|
const NOW = 1_000_000;
|
||||||
|
const HOUR = 3_600_000;
|
||||||
|
|
||||||
|
function session(id: string, updatedAt: number): Session {
|
||||||
|
return { id, title: id, model: "m", createdAt: updatedAt, updatedAt, turns: [] };
|
||||||
|
}
|
||||||
|
|
||||||
|
function summary(id: string, updatedAt: number): SessionSummary {
|
||||||
|
return { id, title: id, model: "m", createdAt: updatedAt, updatedAt, turnCount: 0, preview: "" };
|
||||||
|
}
|
||||||
|
|
||||||
|
function attachment(path: string): PendingAttachment {
|
||||||
|
return { path, mime: "image/png", base64: "AAAA" };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Ports that record what was asked for, so "did not resume" is checkable. */
|
||||||
|
function ports(overrides: Partial<StartupPorts> = {}) {
|
||||||
|
const loadSession = vi.fn(async (id: string) => session(id, NOW));
|
||||||
|
const listSessions = vi.fn(async () => [summary("latest", NOW)]);
|
||||||
|
const readAttachment = vi.fn(async (path: string) => attachment(path));
|
||||||
|
|
||||||
|
return {
|
||||||
|
loadSession,
|
||||||
|
listSessions,
|
||||||
|
readAttachment,
|
||||||
|
...overrides,
|
||||||
|
} satisfies StartupPorts;
|
||||||
|
}
|
||||||
|
|
||||||
|
function input(overrides: Partial<Parameters<typeof resolveStartup>[0]> = {}) {
|
||||||
|
return {
|
||||||
|
hasInitialSession: false,
|
||||||
|
now: NOW,
|
||||||
|
...overrides,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
describe("isHandoff", () => {
|
||||||
|
it("is true when a file was handed over", () => {
|
||||||
|
expect(isHandoff({ initialAttachmentPath: "/tmp/snip.png" })).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("is true when a conversation was named", () => {
|
||||||
|
expect(isHandoff({ initialSessionId: "abc" })).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("is false on an ordinary open", () => {
|
||||||
|
expect(isHandoff({})).toBe(false);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("resolveStartup", () => {
|
||||||
|
describe("a snip hand-off", () => {
|
||||||
|
it("reads the attachment and restores the named conversation", async () => {
|
||||||
|
const p = ports();
|
||||||
|
|
||||||
|
const result = await resolveStartup(
|
||||||
|
input({ initialSessionId: "abc", initialAttachmentPath: "/tmp/snip.png" }),
|
||||||
|
p,
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(result.session?.id).toBe("abc");
|
||||||
|
expect(result.attachment?.path).toBe("/tmp/snip.png");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not fall through to the idle resume", async () => {
|
||||||
|
// The bug this file exists for: a hand-off has to finish startup even
|
||||||
|
// though it took the resume path off the table. Guarding the resume with
|
||||||
|
// "and there was no hand-off" while forgetting to clear the readiness flag
|
||||||
|
// left the view on its loading row forever.
|
||||||
|
const p = ports();
|
||||||
|
|
||||||
|
await resolveStartup(input({ initialAttachmentPath: "/tmp/snip.png" }), p);
|
||||||
|
|
||||||
|
expect(p.listSessions).not.toHaveBeenCalled();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not attach the most recent conversation's session", async () => {
|
||||||
|
const p = ports();
|
||||||
|
|
||||||
|
const result = await resolveStartup(input({ initialAttachmentPath: "/tmp/snip.png" }), p);
|
||||||
|
|
||||||
|
expect(result.session).toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("still resolves when the named conversation has been deleted", async () => {
|
||||||
|
// A snip started from a conversation someone has since deleted should mean
|
||||||
|
// a fresh chat with the image, not a spinner or a crash.
|
||||||
|
const p = ports({ loadSession: vi.fn(async () => undefined) });
|
||||||
|
|
||||||
|
const result = await resolveStartup(
|
||||||
|
input({ initialSessionId: "gone", initialAttachmentPath: "/tmp/snip.png" }),
|
||||||
|
p,
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(result.session).toBeUndefined();
|
||||||
|
expect(result.attachment?.path).toBe("/tmp/snip.png");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("reports an unreadable attachment without refusing to open", async () => {
|
||||||
|
const p = ports({
|
||||||
|
readAttachment: vi.fn(async () => {
|
||||||
|
throw new Error("ENOENT: no such file");
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
|
||||||
|
const result = await resolveStartup(input({ initialAttachmentPath: "/tmp/gone.png" }), p);
|
||||||
|
|
||||||
|
expect(result.attachment).toBeUndefined();
|
||||||
|
expect(result.attachmentError).toContain("ENOENT");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("survives a storage failure while loading the named conversation", async () => {
|
||||||
|
const p = ports({
|
||||||
|
loadSession: vi.fn(async () => {
|
||||||
|
throw new Error("storage exploded");
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
|
||||||
|
const result = await resolveStartup(input({ initialSessionId: "abc" }), p);
|
||||||
|
|
||||||
|
expect(result.session).toBeUndefined();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("an ordinary open", () => {
|
||||||
|
it("resumes the most recent conversation when it is fresh", async () => {
|
||||||
|
const p = ports({ listSessions: vi.fn(async () => [summary("latest", NOW - 60_000)]) });
|
||||||
|
|
||||||
|
const result = await resolveStartup(input({ resumeTimeoutMs: HOUR }), p);
|
||||||
|
|
||||||
|
expect(result.session?.id).toBe("latest");
|
||||||
|
expect(p.loadSession).toHaveBeenCalledWith("latest");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("starts fresh when the newest conversation is older than the window", async () => {
|
||||||
|
const p = ports({ listSessions: vi.fn(async () => [summary("latest", NOW - 2 * HOUR)]) });
|
||||||
|
|
||||||
|
const result = await resolveStartup(input({ resumeTimeoutMs: HOUR }), p);
|
||||||
|
|
||||||
|
expect(result.session).toBeUndefined();
|
||||||
|
expect(p.loadSession).not.toHaveBeenCalled();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("treats exactly the window as fresh", async () => {
|
||||||
|
const p = ports({ listSessions: vi.fn(async () => [summary("latest", NOW - HOUR)]) });
|
||||||
|
|
||||||
|
const result = await resolveStartup(input({ resumeTimeoutMs: HOUR }), p);
|
||||||
|
|
||||||
|
expect(result.session?.id).toBe("latest");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("always resumes when the preference is 'never time out'", async () => {
|
||||||
|
const p = ports({ listSessions: vi.fn(async () => [summary("latest", NOW - 400 * HOUR)]) });
|
||||||
|
|
||||||
|
const result = await resolveStartup(input({ resumeTimeoutMs: undefined }), p);
|
||||||
|
|
||||||
|
expect(result.session?.id).toBe("latest");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("starts fresh when there are no conversations at all", async () => {
|
||||||
|
const p = ports({ listSessions: vi.fn(async () => []) });
|
||||||
|
|
||||||
|
const result = await resolveStartup(input({ resumeTimeoutMs: HOUR }), p);
|
||||||
|
|
||||||
|
expect(result).toEqual({});
|
||||||
|
expect(p.loadSession).not.toHaveBeenCalled();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("starts fresh rather than failing when the index cannot be read", async () => {
|
||||||
|
const p = ports({
|
||||||
|
listSessions: vi.fn(async () => {
|
||||||
|
throw new Error("corrupt index");
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
|
||||||
|
const result = await resolveStartup(input({ resumeTimeoutMs: HOUR }), p);
|
||||||
|
|
||||||
|
expect(result).toEqual({});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not resume when a session was pushed in by hand", async () => {
|
||||||
|
// The sessions command already knows which conversation to show.
|
||||||
|
const p = ports();
|
||||||
|
|
||||||
|
await resolveStartup(input({ hasInitialSession: true }), p);
|
||||||
|
|
||||||
|
expect(p.listSessions).not.toHaveBeenCalled();
|
||||||
|
expect(p.loadSession).not.toHaveBeenCalled();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -13,6 +13,7 @@ function turn(id: string, prompt: string, response: string, overrides: Partial<C
|
|||||||
model: "m",
|
model: "m",
|
||||||
createdAt: 100,
|
createdAt: 100,
|
||||||
toolCalls: [],
|
toolCalls: [],
|
||||||
|
attachments: [],
|
||||||
...overrides,
|
...overrides,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,121 @@
|
|||||||
|
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||||
|
import { __resetLocalStorage } from "../__mocks__/@vicinae/api";
|
||||||
|
import { forgetVisionSupport, visionSupport } from "../lib/vision";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The capability check decides whether a snip feature is offered at all, so the
|
||||||
|
* cases that matter are the ones where the answer is not a clean yes or no: a
|
||||||
|
* dead network must not be mistaken for "this model cannot see".
|
||||||
|
*/
|
||||||
|
|
||||||
|
const fetchMock = vi.fn();
|
||||||
|
|
||||||
|
function showResponse(capabilities: string[]): Response {
|
||||||
|
return new Response(JSON.stringify({ capabilities }), {
|
||||||
|
status: 200,
|
||||||
|
headers: { "Content-Type": "application/json" },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
beforeEach(() => {
|
||||||
|
__resetLocalStorage();
|
||||||
|
fetchMock.mockReset();
|
||||||
|
vi.stubGlobal("fetch", fetchMock);
|
||||||
|
});
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
vi.unstubAllGlobals();
|
||||||
|
});
|
||||||
|
|
||||||
|
const base = { host: "https://ollama.com", apiKey: "k", model: "gemma4" };
|
||||||
|
|
||||||
|
describe("visionSupport", () => {
|
||||||
|
it("reports yes when Ollama lists the vision capability", async () => {
|
||||||
|
fetchMock.mockResolvedValueOnce(showResponse(["completion", "vision"]));
|
||||||
|
await expect(visionSupport(base)).resolves.toBe("yes");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("reports no for a text-only model", async () => {
|
||||||
|
fetchMock.mockResolvedValueOnce(showResponse(["completion", "tools"]));
|
||||||
|
await expect(visionSupport(base)).resolves.toBe("no");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("asks /api/show for the model", async () => {
|
||||||
|
fetchMock.mockResolvedValueOnce(showResponse(["vision"]));
|
||||||
|
await visionSupport(base);
|
||||||
|
|
||||||
|
const [url, init] = fetchMock.mock.calls[0];
|
||||||
|
expect(url).toBe("https://ollama.com/api/show");
|
||||||
|
expect(JSON.parse(init.body)).toEqual({ model: "gemma4" });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("serves a repeat question from the cache", async () => {
|
||||||
|
fetchMock.mockResolvedValueOnce(showResponse(["vision"]));
|
||||||
|
await visionSupport(base);
|
||||||
|
await expect(visionSupport(base)).resolves.toBe("yes");
|
||||||
|
|
||||||
|
expect(fetchMock).toHaveBeenCalledTimes(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("caches a negative too, so a text model is not re-checked on every mount", async () => {
|
||||||
|
fetchMock.mockResolvedValueOnce(showResponse(["tools"]));
|
||||||
|
await visionSupport(base);
|
||||||
|
await expect(visionSupport(base)).resolves.toBe("no");
|
||||||
|
|
||||||
|
expect(fetchMock).toHaveBeenCalledTimes(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("keys the cache per model, so switching model re-checks", async () => {
|
||||||
|
fetchMock.mockResolvedValueOnce(showResponse(["vision"]));
|
||||||
|
await visionSupport(base);
|
||||||
|
|
||||||
|
fetchMock.mockResolvedValueOnce(showResponse(["tools"]));
|
||||||
|
await expect(visionSupport({ ...base, model: "gpt-oss:120b" })).resolves.toBe("no");
|
||||||
|
|
||||||
|
expect(fetchMock).toHaveBeenCalledTimes(2);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("re-checks when explicitly asked to, even with a cache entry", async () => {
|
||||||
|
fetchMock.mockResolvedValueOnce(showResponse(["tools"]));
|
||||||
|
await visionSupport(base);
|
||||||
|
|
||||||
|
fetchMock.mockResolvedValueOnce(showResponse(["vision"]));
|
||||||
|
await expect(visionSupport({ ...base, refresh: true })).resolves.toBe("yes");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("reports unknown rather than no when the request fails", async () => {
|
||||||
|
// A dead network is not evidence that a model cannot see, and treating it as
|
||||||
|
// such would hide the feature for a reason the user cannot see.
|
||||||
|
fetchMock.mockRejectedValueOnce(new Error("offline"));
|
||||||
|
await expect(visionSupport(base)).resolves.toBe("unknown");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not cache a failure", async () => {
|
||||||
|
fetchMock.mockRejectedValueOnce(new Error("offline"));
|
||||||
|
await visionSupport(base);
|
||||||
|
|
||||||
|
fetchMock.mockResolvedValueOnce(showResponse(["vision"]));
|
||||||
|
await expect(visionSupport(base)).resolves.toBe("yes");
|
||||||
|
|
||||||
|
expect(fetchMock).toHaveBeenCalledTimes(2);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("reports a rejected key as unknown rather than crashing", async () => {
|
||||||
|
fetchMock.mockResolvedValueOnce(new Response("unauthorized", { status: 401 }));
|
||||||
|
await expect(visionSupport(base)).resolves.toBe("unknown");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("forgetVisionSupport", () => {
|
||||||
|
it("drops every cached answer, so a re-pulled model is not pinned to no", async () => {
|
||||||
|
fetchMock.mockResolvedValueOnce(showResponse(["vision"]));
|
||||||
|
await visionSupport(base);
|
||||||
|
|
||||||
|
await forgetVisionSupport();
|
||||||
|
|
||||||
|
fetchMock.mockResolvedValueOnce(showResponse(["vision"]));
|
||||||
|
await visionSupport(base);
|
||||||
|
|
||||||
|
expect(fetchMock).toHaveBeenCalledTimes(2);
|
||||||
|
});
|
||||||
|
});
|
||||||
+14
-1
@@ -1,5 +1,6 @@
|
|||||||
import { type LaunchProps } from "@vicinae/api";
|
import { type LaunchProps } from "@vicinae/api";
|
||||||
import { ChatView } from "./lib/components/ChatView";
|
import { ChatView } from "./lib/components/ChatView";
|
||||||
|
import { readSnipContext } from "./lib/snip";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Entrypoint for the "chat" command.
|
* Entrypoint for the "chat" command.
|
||||||
@@ -11,7 +12,19 @@ import { ChatView } from "./lib/components/ChatView";
|
|||||||
* launcher-and-start-typing behaviour. `fallbackText` is populated when the
|
* launcher-and-start-typing behaviour. `fallbackText` is populated when the
|
||||||
* command is opened through a deeplink carrying text, so
|
* command is opened through a deeplink carrying text, so
|
||||||
* `vicinae://launch/lam-agent/chat?fallbackText=...` arrives pre-filled.
|
* `vicinae://launch/lam-agent/chat?fallbackText=...` arrives pre-filled.
|
||||||
|
*
|
||||||
|
* It is also the command the `snip` command returns through, carrying a captured
|
||||||
|
* file and, when the snip came from an open conversation, that conversation's id.
|
||||||
*/
|
*/
|
||||||
export default function ChatCommand(props: LaunchProps) {
|
export default function ChatCommand(props: LaunchProps) {
|
||||||
return <ChatView initialPrompt={props.fallbackText} />;
|
const snip = readSnipContext(props.launchContext);
|
||||||
|
|
||||||
|
return (
|
||||||
|
<ChatView
|
||||||
|
initialPrompt={props.fallbackText}
|
||||||
|
initialSessionId={snip.sessionId}
|
||||||
|
initialAttachmentPath={snip.snipPath}
|
||||||
|
launchNote={snip.snipError ? `**Could not take a screenshot:** ${snip.snipError}` : undefined}
|
||||||
|
/>
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,137 @@
|
|||||||
|
import { readFile } from "node:fs/promises";
|
||||||
|
import { extname } from "node:path";
|
||||||
|
import { pathToFileURL } from "node:url";
|
||||||
|
import type { Attachment, PendingAttachment } from "./types";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Reading attachments off disk, and describing them to the renderer.
|
||||||
|
*
|
||||||
|
* Kept separate from `capture.ts` because the two answer different questions:
|
||||||
|
* capture is about getting an image *onto* the disk, this is about getting it
|
||||||
|
* back off again and telling Vicinae how to draw it.
|
||||||
|
*
|
||||||
|
* Imports no Vicinae module, so it unit tests with no runtime.
|
||||||
|
*/
|
||||||
|
|
||||||
|
/** Extensions Ollama accepts. A snip is always PNG; the rest are here for a future picker. */
|
||||||
|
const MIME_TYPES: Record<string, string> = {
|
||||||
|
".png": "image/png",
|
||||||
|
".jpg": "image/jpeg",
|
||||||
|
".jpeg": "image/jpeg",
|
||||||
|
".webp": "image/webp",
|
||||||
|
".gif": "image/gif",
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The box a preview is fitted into.
|
||||||
|
*
|
||||||
|
* The height is not a preference: `MdImage.qml` clamps rendered height to 200px
|
||||||
|
* whatever is asked for (`Math.min(_rawH, 200, maxImageHeight)`), so a taller
|
||||||
|
* request is silently ignored. The width is here to keep a very wide capture
|
||||||
|
* from being scaled up past the detail pane it sits in.
|
||||||
|
*/
|
||||||
|
const PREVIEW_MAX_WIDTH = 400;
|
||||||
|
const PREVIEW_MAX_HEIGHT = 200;
|
||||||
|
|
||||||
|
export function mimeForPath(path: string): string {
|
||||||
|
return MIME_TYPES[extname(path).toLowerCase()] ?? "application/octet-stream";
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The eight bytes every PNG starts with. */
|
||||||
|
const PNG_SIGNATURE = [0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a];
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Pixel dimensions from a PNG's `IHDR` chunk, or `undefined` for anything else.
|
||||||
|
*
|
||||||
|
* Only PNG is handled, because that is the only thing a snip produces and
|
||||||
|
* guessing at other formats would be a lie. The header is 24 bytes, so this
|
||||||
|
* costs nothing on top of the read that is already happening.
|
||||||
|
*/
|
||||||
|
export function pngSize(bytes: Buffer): { width: number; height: number } | undefined {
|
||||||
|
if (bytes.length < 24) return undefined;
|
||||||
|
if (!PNG_SIGNATURE.every((byte, index) => bytes[index] === byte)) return undefined;
|
||||||
|
if (bytes.toString("ascii", 12, 16) !== "IHDR") return undefined;
|
||||||
|
|
||||||
|
const width = bytes.readUInt32BE(16);
|
||||||
|
const height = bytes.readUInt32BE(20);
|
||||||
|
|
||||||
|
return width > 0 && height > 0 ? { width, height } : undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The exact box to ask the renderer for, or `undefined` when the size is unknown.
|
||||||
|
*
|
||||||
|
* This is the whole point of reading the header. With no hint, `MdImage.qml`
|
||||||
|
* sizes its container to the pane width and 200px tall and then aspect-fits the
|
||||||
|
* image inside it, so a wide screenshot sits in a letterbox with dead space
|
||||||
|
* above and below it. Asking for a box that already matches the image's aspect
|
||||||
|
* ratio leaves none.
|
||||||
|
*
|
||||||
|
* The scale is capped at 1 so a small capture is not blown up into a blurry one.
|
||||||
|
*/
|
||||||
|
export function previewBox(
|
||||||
|
width: number | undefined,
|
||||||
|
height: number | undefined,
|
||||||
|
): { width: number; height: number } | undefined {
|
||||||
|
if (!width || !height || width <= 0 || height <= 0) return undefined;
|
||||||
|
|
||||||
|
const scale = Math.min(1, PREVIEW_MAX_WIDTH / width, PREVIEW_MAX_HEIGHT / height);
|
||||||
|
|
||||||
|
return {
|
||||||
|
width: Math.max(1, Math.round(width * scale)),
|
||||||
|
height: Math.max(1, Math.round(height * scale)),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A markdown image source for a local file.
|
||||||
|
*
|
||||||
|
* A `file://` URL rather than a bare path, because Vicinae's markdown renderer
|
||||||
|
* only resolves a scheme it recognises: `imageProviderUrl` in
|
||||||
|
* `markdown-model.cpp` routes `file://`, and a scheme-less string, through
|
||||||
|
* `ImageURL::local`, and anything it does not recognise it returns as empty.
|
||||||
|
* `pathToFileURL` does the escaping, so a capture under a directory with a space
|
||||||
|
* in its name still resolves.
|
||||||
|
*
|
||||||
|
* The query is read from the URL before it is resolved to a path, so the size
|
||||||
|
* hints travel here without ever reaching the file name. They are spelled the
|
||||||
|
* way Raycast spelled them, which is what the parser looks for.
|
||||||
|
*/
|
||||||
|
export function imageSource(attachment: Pick<Attachment, "path" | "width" | "height">): string {
|
||||||
|
const url = pathToFileURL(attachment.path);
|
||||||
|
const box = previewBox(attachment.width, attachment.height);
|
||||||
|
|
||||||
|
if (box) {
|
||||||
|
url.searchParams.set("raycast-width", String(box.width));
|
||||||
|
url.searchParams.set("raycast-height", String(box.height));
|
||||||
|
}
|
||||||
|
|
||||||
|
return url.href;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Reads an image into the shape a request needs.
|
||||||
|
*
|
||||||
|
* Ollama wants base64 with no `data:` URL prefix, which is why the payload is
|
||||||
|
* built here rather than handed straight to the renderer. The dimensions are
|
||||||
|
* kept for the same reason the path is: the transcript needs them long after
|
||||||
|
* this call is over, and the preview layout depends on them.
|
||||||
|
*/
|
||||||
|
export async function readAttachment(path: string): Promise<PendingAttachment> {
|
||||||
|
const bytes = await readFile(path);
|
||||||
|
const size = pngSize(bytes);
|
||||||
|
|
||||||
|
return {
|
||||||
|
path,
|
||||||
|
mime: mimeForPath(path),
|
||||||
|
base64: bytes.toString("base64"),
|
||||||
|
width: size?.width,
|
||||||
|
height: size?.height,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The persistable half of a pending attachment: everything except the base64 payload. */
|
||||||
|
export function toStoredAttachment(attachment: PendingAttachment): Attachment {
|
||||||
|
const { path, mime, width, height } = attachment;
|
||||||
|
return { path, mime, width, height };
|
||||||
|
}
|
||||||
@@ -0,0 +1,346 @@
|
|||||||
|
import { execFile } from "node:child_process";
|
||||||
|
import { mkdir, stat, writeFile } from "node:fs/promises";
|
||||||
|
import { join } from "node:path";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Screen capture for "snip to chat".
|
||||||
|
*
|
||||||
|
* Vicinae has no screenshot API. There is no capture call anywhere in
|
||||||
|
* `@vicinae/api`, and `Clipboard` moves text, html, urls and a file path and
|
||||||
|
* nothing binary, so a bitmap can never be read back *out* of the clipboard
|
||||||
|
* either. The only way to get a region of the screen is to shell out.
|
||||||
|
*
|
||||||
|
* That is acceptable here because Vicinae extensions run under Node ("all
|
||||||
|
* standard Node.js APIs are available"), which is why `node:child_process` is
|
||||||
|
* not a workaround so much as the supported route. The important consequence is
|
||||||
|
* that the capture is *owned* rather than borrowed: a tool writes a PNG to a
|
||||||
|
* path this module picks, and that path is what the chat stores. Nothing has to
|
||||||
|
* guess what some other application left on the clipboard.
|
||||||
|
*
|
||||||
|
* Every function here takes its environment as an argument and imports no
|
||||||
|
* Vicinae module, so the selection logic unit tests with no runtime and no
|
||||||
|
* installed tools.
|
||||||
|
*/
|
||||||
|
|
||||||
|
/** A PNG piped through stdout (flameshot) is easily several megabytes. */
|
||||||
|
const MAX_CAPTURE_BYTES = 64 * 1024 * 1024;
|
||||||
|
|
||||||
|
/** Long enough to pick a region unhurried, short enough that a wedged tool cannot hang the view forever. */
|
||||||
|
const CAPTURE_TIMEOUT_MS = 5 * 60 * 1000;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* What macOS says when Screen Recording permission is missing.
|
||||||
|
*
|
||||||
|
* This one pattern is why the module reads stderr at all. The message is the
|
||||||
|
* *only* difference between "the system refused to let us look" and "the user
|
||||||
|
* pressed Escape": both produce no file, and `screencapture -i` has been
|
||||||
|
* reported to exit `0` in the refusal case. Treating the refusal as a cancel
|
||||||
|
* leaves the user with a feature that silently does nothing.
|
||||||
|
*/
|
||||||
|
const SCREEN_RECORDING_DENIED = /could not create image from (rect|display|window)/i;
|
||||||
|
|
||||||
|
export interface RunResult {
|
||||||
|
stdout: Buffer;
|
||||||
|
stderr: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Runs a command. Injected so tests never spawn anything. */
|
||||||
|
export type RunCommand = (
|
||||||
|
tool: string,
|
||||||
|
args: string[],
|
||||||
|
options?: { signal?: AbortSignal },
|
||||||
|
) => Promise<RunResult>;
|
||||||
|
|
||||||
|
export interface CaptureBackend {
|
||||||
|
/** Shown to the user when a capture fails, so they know what to install. */
|
||||||
|
name: string;
|
||||||
|
/** Every one of these has to resolve on `PATH` for the backend to be usable. */
|
||||||
|
requires: string[];
|
||||||
|
/** Produces the image at `output`, or resolves without one when the user cancels. */
|
||||||
|
capture: (context: CaptureContext) => Promise<void>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CaptureContext {
|
||||||
|
output: string;
|
||||||
|
signal?: AbortSignal;
|
||||||
|
run: RunCommand;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CaptureResult {
|
||||||
|
/** Absolute path of the image that was written. */
|
||||||
|
path: string;
|
||||||
|
bytes: number;
|
||||||
|
/** The tool that produced it, for the transcript and for error messages. */
|
||||||
|
backend: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CaptureOptions {
|
||||||
|
/** Where images are kept. The caller supplies `environment.supportPath`, which keeps this module Vicinae-free. */
|
||||||
|
outputDir: string;
|
||||||
|
platform?: NodeJS.Platform;
|
||||||
|
signal?: AbortSignal;
|
||||||
|
now?: Date;
|
||||||
|
/** Overrides for tests. */
|
||||||
|
hasCommand?: (tool: string) => Promise<boolean>;
|
||||||
|
run?: RunCommand;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Raised when the machine has no capture tool at all, with something actionable to do about it. */
|
||||||
|
export class NoCaptureToolError extends Error {
|
||||||
|
readonly hint: string;
|
||||||
|
|
||||||
|
constructor(platform: NodeJS.Platform) {
|
||||||
|
super("No screen capture tool is installed.");
|
||||||
|
this.name = "NoCaptureToolError";
|
||||||
|
this.hint =
|
||||||
|
platform === "darwin"
|
||||||
|
? "macOS ships screencapture, so this should not happen. Check that it is on PATH."
|
||||||
|
: "Install one of: grim + slurp (Wayland), grimblast (Hyprland), spectacle, maim + slop (X11), scrot, or flameshot.";
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Raised when the system refuses to hand over the screen.
|
||||||
|
*
|
||||||
|
* Separate from a generic tool failure because it is the one capture error the
|
||||||
|
* user can actually fix, and because on macOS the fix is not obvious: the
|
||||||
|
* permission belongs to the *launcher*, since TCC attributes the request to the
|
||||||
|
* app responsible for spawning the extension runtime rather than to the
|
||||||
|
* extension itself.
|
||||||
|
*/
|
||||||
|
export class CapturePermissionError extends Error {
|
||||||
|
readonly hint: string;
|
||||||
|
|
||||||
|
constructor(platform: NodeJS.Platform) {
|
||||||
|
super("The system denied screen capture.");
|
||||||
|
this.name = "CapturePermissionError";
|
||||||
|
this.hint =
|
||||||
|
platform === "darwin"
|
||||||
|
? "Give Vicinae Screen Recording permission in System Settings under Privacy & Security, then quit Vicinae and open it again. The permission belongs to Vicinae rather than to the extension, because macOS attributes the request to the app that launched the extension runtime."
|
||||||
|
: "Give the launcher permission to record the screen, then restart it.";
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A capture tool that ran and failed for a reason it could explain. */
|
||||||
|
export class CaptureToolError extends Error {
|
||||||
|
readonly tool: string;
|
||||||
|
readonly stderr: string;
|
||||||
|
|
||||||
|
constructor(tool: string, stderr: string, cause?: unknown) {
|
||||||
|
const detail = stderr.trim().split("\n")[0] ?? "";
|
||||||
|
super(`${tool} failed.${detail ? ` ${detail}` : ""}`, cause === undefined ? undefined : { cause });
|
||||||
|
this.name = "CaptureToolError";
|
||||||
|
this.tool = tool;
|
||||||
|
this.stderr = stderr;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* `command -v` rather than spawning the tool itself: several of these block or
|
||||||
|
* open a window when run bare, and `--version` is not reliably supported.
|
||||||
|
*
|
||||||
|
* The tool names are literals from the table below and never come from the
|
||||||
|
* user, so there is nothing to escape.
|
||||||
|
*/
|
||||||
|
async function commandExists(tool: string): Promise<boolean> {
|
||||||
|
return new Promise((resolve) => {
|
||||||
|
execFile("sh", ["-c", `command -v ${tool}`], { timeout: 5_000 }, (error) => resolve(!error));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function runCommand(
|
||||||
|
tool: string,
|
||||||
|
args: string[],
|
||||||
|
options: { signal?: AbortSignal } = {},
|
||||||
|
): Promise<RunResult> {
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
execFile(
|
||||||
|
tool,
|
||||||
|
args,
|
||||||
|
{
|
||||||
|
signal: options.signal,
|
||||||
|
timeout: CAPTURE_TIMEOUT_MS,
|
||||||
|
maxBuffer: MAX_CAPTURE_BYTES,
|
||||||
|
encoding: "buffer",
|
||||||
|
},
|
||||||
|
(error, stdout, stderr) => {
|
||||||
|
const text = stderr?.toString("utf8") ?? "";
|
||||||
|
|
||||||
|
if (error) {
|
||||||
|
reject(new CaptureToolError(tool, text, error));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
resolve({ stdout: stdout as Buffer, stderr: text });
|
||||||
|
},
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* `grim -g "$(slurp)"`, expressed as two argv calls rather than a shell pipeline.
|
||||||
|
*
|
||||||
|
* slurp prints the chosen geometry and exits non-zero when the selection is
|
||||||
|
* cancelled, so an empty or absent geometry is the cancel signal.
|
||||||
|
*/
|
||||||
|
const grimSlurp: CaptureBackend = {
|
||||||
|
name: "grim + slurp",
|
||||||
|
requires: ["grim", "slurp"],
|
||||||
|
async capture({ output, signal, run }) {
|
||||||
|
const geometry = (await run("slurp", [], { signal })).stdout.toString("utf8").trim();
|
||||||
|
if (!geometry) return;
|
||||||
|
await run("grim", ["-g", geometry, output], { signal });
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* flameshot is the only backend that will not take an output path: `gui --raw`
|
||||||
|
* writes the PNG to stdout, so it is captured and written here instead.
|
||||||
|
*/
|
||||||
|
const flameshot: CaptureBackend = {
|
||||||
|
name: "flameshot",
|
||||||
|
requires: ["flameshot"],
|
||||||
|
async capture({ output, signal, run }) {
|
||||||
|
const png = (await run("flameshot", ["gui", "--raw"], { signal })).stdout;
|
||||||
|
if (png.length === 0) return;
|
||||||
|
await writeFile(output, png);
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Backends in preference order.
|
||||||
|
*
|
||||||
|
* Wayland first on Linux, because that is what the launcher's own users are on
|
||||||
|
* (the README's shortcut examples are Hyprland and niri), then the X11 tools.
|
||||||
|
* macOS needs only the built-in tool, which is why it is tried on its own.
|
||||||
|
*/
|
||||||
|
export const CAPTURE_BACKENDS: CaptureBackend[] = [
|
||||||
|
{
|
||||||
|
name: "grimblast",
|
||||||
|
requires: ["grimblast"],
|
||||||
|
async capture({ output, signal, run }) {
|
||||||
|
await run("grimblast", ["save", "area", output], { signal });
|
||||||
|
},
|
||||||
|
},
|
||||||
|
grimSlurp,
|
||||||
|
{
|
||||||
|
name: "spectacle",
|
||||||
|
requires: ["spectacle"],
|
||||||
|
async capture({ output, signal, run }) {
|
||||||
|
// -r region, -b background, -n no notification, -o output file.
|
||||||
|
await run("spectacle", ["-r", "-b", "-n", "-o", output], { signal });
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "maim",
|
||||||
|
requires: ["maim"],
|
||||||
|
async capture({ output, signal, run }) {
|
||||||
|
await run("maim", ["-s", output], { signal });
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "scrot",
|
||||||
|
requires: ["scrot"],
|
||||||
|
async capture({ output, signal, run }) {
|
||||||
|
await run("scrot", ["-s", output], { signal });
|
||||||
|
},
|
||||||
|
},
|
||||||
|
flameshot,
|
||||||
|
{
|
||||||
|
name: "screencapture",
|
||||||
|
requires: ["screencapture"],
|
||||||
|
async capture({ output, signal, run }) {
|
||||||
|
// -i interactive region, -x no shutter sound.
|
||||||
|
await run("screencapture", ["-i", "-x", output], { signal });
|
||||||
|
},
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The first backend whose tools are all installed, or `undefined`.
|
||||||
|
*
|
||||||
|
* `hasCommand` is a parameter rather than a call so the preference order can be
|
||||||
|
* tested against a fake `PATH`, which is the part most likely to break on a
|
||||||
|
* platform nobody here runs.
|
||||||
|
*/
|
||||||
|
export async function selectBackend(
|
||||||
|
hasCommand: (tool: string) => Promise<boolean>,
|
||||||
|
backends: CaptureBackend[] = CAPTURE_BACKENDS,
|
||||||
|
): Promise<CaptureBackend | undefined> {
|
||||||
|
for (const backend of backends) {
|
||||||
|
const checks = await Promise.all(backend.requires.map((tool) => hasCommand(tool)));
|
||||||
|
if (checks.every(Boolean)) return backend;
|
||||||
|
}
|
||||||
|
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Sortable, filesystem-safe, and obvious when found lying in the support directory. */
|
||||||
|
export function snipFileName(now: Date): string {
|
||||||
|
const pad = (value: number) => String(value).padStart(2, "0");
|
||||||
|
|
||||||
|
const date = `${now.getFullYear()}${pad(now.getMonth() + 1)}${pad(now.getDate())}`;
|
||||||
|
const time = `${pad(now.getHours())}${pad(now.getMinutes())}${pad(now.getSeconds())}`;
|
||||||
|
|
||||||
|
return `snip-${date}-${time}.png`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Captures a region and returns the file it wrote, or `null` when the user
|
||||||
|
* backed out.
|
||||||
|
*
|
||||||
|
* Success is decided by the file, not by an exit code: the tools disagree about
|
||||||
|
* what a cancelled selection looks like, and at least one reports success
|
||||||
|
* anyway. Failure is then triaged from what the tool said on stderr, which is
|
||||||
|
* the only channel that separates a refused permission from a cancel.
|
||||||
|
*/
|
||||||
|
export async function captureRegion(options: CaptureOptions): Promise<CaptureResult | null> {
|
||||||
|
const platform = options.platform ?? process.platform;
|
||||||
|
const hasCommand = options.hasCommand ?? commandExists;
|
||||||
|
const baseRun = options.run ?? runCommand;
|
||||||
|
|
||||||
|
const backend = await selectBackend(
|
||||||
|
hasCommand,
|
||||||
|
platform === "darwin" ? CAPTURE_BACKENDS.filter((b) => b.name === "screencapture") : CAPTURE_BACKENDS,
|
||||||
|
);
|
||||||
|
if (!backend) throw new NoCaptureToolError(platform);
|
||||||
|
|
||||||
|
await mkdir(options.outputDir, { recursive: true });
|
||||||
|
const output = join(options.outputDir, snipFileName(options.now ?? new Date()));
|
||||||
|
|
||||||
|
/** Everything the tools said, whether or not they exited cleanly. */
|
||||||
|
const diagnostics: string[] = [];
|
||||||
|
const run: RunCommand = async (tool, args, runOptions) => {
|
||||||
|
try {
|
||||||
|
const result = await baseRun(tool, args, runOptions);
|
||||||
|
if (result.stderr.trim()) diagnostics.push(result.stderr.trim());
|
||||||
|
return result;
|
||||||
|
} catch (thrown) {
|
||||||
|
const stderr = thrown instanceof CaptureToolError ? thrown.stderr : "";
|
||||||
|
if (stderr.trim()) diagnostics.push(stderr.trim());
|
||||||
|
throw thrown;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
let failure: unknown;
|
||||||
|
try {
|
||||||
|
await backend.capture({ output, signal: options.signal, run });
|
||||||
|
} catch (thrown) {
|
||||||
|
failure = thrown;
|
||||||
|
}
|
||||||
|
|
||||||
|
const written = await stat(output).catch(() => null);
|
||||||
|
if (written && written.size > 0) {
|
||||||
|
return { path: output, bytes: written.size, backend: backend.name };
|
||||||
|
}
|
||||||
|
|
||||||
|
const detail = diagnostics.join("\n");
|
||||||
|
|
||||||
|
if (SCREEN_RECORDING_DENIED.test(detail)) throw new CapturePermissionError(platform);
|
||||||
|
|
||||||
|
// A tool that explained itself is a failure worth reporting. Silence with no
|
||||||
|
// image is the user pressing Escape, which is the common case and says nothing.
|
||||||
|
if (detail) throw new CaptureToolError(backend.requires[0], detail, failure);
|
||||||
|
|
||||||
|
return null;
|
||||||
|
}
|
||||||
+190
-3
@@ -88,6 +88,35 @@ const OPENING_FENCE = /^(\s*)(`{3,}|~{3,})\s*([^\s`~]*)\s*$/;
|
|||||||
const CLOSING_FENCE = /^\s*(`{3,}|~{3,})\s*$/;
|
const CLOSING_FENCE = /^\s*(`{3,}|~{3,})\s*$/;
|
||||||
const STANDALONE_CODE_SPAN = /^\s*`([^`\n]+)`\s*[.!]?\s*$/;
|
const STANDALONE_CODE_SPAN = /^\s*`([^`\n]+)`\s*[.!]?\s*$/;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A line that is nothing but a bare http(s) URL, give or take a bullet.
|
||||||
|
*
|
||||||
|
* Vicinae only makes a URL clickable when markdown presents it as a link. Its
|
||||||
|
* parser attaches the `table` and `strikethrough` GFM extensions and nothing
|
||||||
|
* else, so the `autolink` extension is off: a bare URL renders as plain dead
|
||||||
|
* text. `MarkdownModel` turns `CMARK_NODE_LINK` into `<a href=...>`, and the
|
||||||
|
* selection controller opens that anchor on release, so `[label](url)` is the
|
||||||
|
* one form the user can actually click.
|
||||||
|
*/
|
||||||
|
const BARE_URL = /^https?:\/\/\S+$/;
|
||||||
|
/** Sentence punctuation a model appends to a URL that is not part of it. */
|
||||||
|
const TRAILING_SENTENCE_PUNCTUATION = /[.,;:!?]+$/;
|
||||||
|
/** One inline code span that holds nothing but a URL. */
|
||||||
|
const STANDALONE_URL_SPAN = /^`([^`\n]+)`[.!]?$/;
|
||||||
|
/** A bullet or numbered-list marker, kept so a list of links stays a list. */
|
||||||
|
const LIST_MARKER = /^([ \t]*(?:[-*+]|\d+[.)])[ \t]+)(.*)$/;
|
||||||
|
|
||||||
|
/** A GFM table row: it opens and closes with a pipe. */
|
||||||
|
const TABLE_ROW = /^\s*\|.*\|\s*$/;
|
||||||
|
/**
|
||||||
|
* The delimiter row under a header.
|
||||||
|
*
|
||||||
|
* The test is not how many dashes there are, because GFM wants only one per
|
||||||
|
* column. It is that the line could be nothing else: pipes, dashes, colons and
|
||||||
|
* spaces, with at least one dash to tell it apart from a row of empty cells.
|
||||||
|
*/
|
||||||
|
const TABLE_DELIMITER_ROW = /^\s*\|?[\s:|-]*-[\s:|-]*\|?\s*$/;
|
||||||
|
|
||||||
function significantLines(code: string): string[] {
|
function significantLines(code: string): string[] {
|
||||||
return code
|
return code
|
||||||
.split("\n")
|
.split("\n")
|
||||||
@@ -272,12 +301,170 @@ export function labelCodeFences(markdown: string): string {
|
|||||||
return out.join("\n");
|
return out.join("\n");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** A URL with the sentence punctuation after it removed. */
|
||||||
|
function asBareUrl(candidate: string): string | undefined {
|
||||||
|
const trimmed = candidate.trim();
|
||||||
|
if (!BARE_URL.test(trimmed)) return undefined;
|
||||||
|
return trimmed.replace(TRAILING_SENTENCE_PUNCTUATION, "");
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Wraps a URL as a markdown link, using the address itself as the label so the
|
||||||
|
* destination stays readable rather than turning into a generic word.
|
||||||
|
*
|
||||||
|
* A destination containing parentheses (plenty of Wikipedia articles) has to
|
||||||
|
* be wrapped in pointy brackets or the parser reads the first `)` as the end of
|
||||||
|
* the destination and leaves the rest as trailing prose.
|
||||||
|
*/
|
||||||
|
function asLink(url: string): string {
|
||||||
|
// Square brackets in the *label* would end the link text early, which is why
|
||||||
|
// an IPv6 literal such as `http://[::1]/` needs them escaped. The destination
|
||||||
|
// tolerates brackets, but not parentheses, which is what the pointy form is
|
||||||
|
// for.
|
||||||
|
const label = url.replace(/[[\]]/g, "\\$&");
|
||||||
|
const destination = /[()]/.test(url) ? `<${url}>` : url;
|
||||||
|
return `[${label}](${destination})`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The URLs of a fence whose every non-empty line is one, or `undefined` when
|
||||||
|
* the block is real code.
|
||||||
|
*
|
||||||
|
* Deliberately narrow: one line that is not a URL disqualifies the whole block,
|
||||||
|
* so a `curl` invocation that happens to contain an address is never touched.
|
||||||
|
*/
|
||||||
|
function urlOnlyBlock(lines: string[]): string[] | undefined {
|
||||||
|
const urls: string[] = [];
|
||||||
|
|
||||||
|
for (const line of lines) {
|
||||||
|
const trimmed = line.trim();
|
||||||
|
if (trimmed.length === 0) continue;
|
||||||
|
|
||||||
|
const url = asBareUrl(trimmed);
|
||||||
|
if (!url) return undefined;
|
||||||
|
urls.push(url);
|
||||||
|
}
|
||||||
|
|
||||||
|
return urls.length > 0 ? urls : undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Rewrites a line that is only a URL, or only a span holding one, into a link. */
|
||||||
|
function linkifyLine(line: string): string {
|
||||||
|
const marker = LIST_MARKER.exec(line);
|
||||||
|
const prefix = marker ? marker[1] : (line.match(/^[ \t]*/)?.[0] ?? "");
|
||||||
|
const rest = marker ? marker[2] : line.slice(prefix.length);
|
||||||
|
|
||||||
|
const span = STANDALONE_URL_SPAN.exec(rest.trim());
|
||||||
|
const url = asBareUrl(span ? span[1] : rest);
|
||||||
|
|
||||||
|
return url ? `${prefix}${asLink(url)}` : line;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Turns URLs the model wrote as code back into links.
|
||||||
|
*
|
||||||
|
* This is the reported bug: asked for a few GitHub projects, the model answers
|
||||||
|
* with a fenced block of bare addresses. A fence renders through
|
||||||
|
* `MdCodeBlock.qml` as a dark panel with a Copy button and no link handling at
|
||||||
|
* all, so the one thing the user wants to do with a URL is the one thing they
|
||||||
|
* cannot. The same applies to a lone `` `https://...` `` span.
|
||||||
|
*
|
||||||
|
* Three shapes are recovered, and only these:
|
||||||
|
*
|
||||||
|
* 1. A fence whose entire body is URLs.
|
||||||
|
* 2. A line that is exactly one inline span holding a URL.
|
||||||
|
* 3. A line that is exactly a URL, bulleted or not.
|
||||||
|
*
|
||||||
|
* Existing fences holding anything else are copied through untouched, so real
|
||||||
|
* code is never rewritten.
|
||||||
|
*/
|
||||||
|
export function restoreUrlLinks(markdown: string): string {
|
||||||
|
const lines = markdown.split("\n");
|
||||||
|
const out: string[] = [];
|
||||||
|
let index = 0;
|
||||||
|
|
||||||
|
while (index < lines.length) {
|
||||||
|
if (!OPENING_FENCE.test(lines[index])) {
|
||||||
|
out.push(linkifyLine(lines[index]));
|
||||||
|
index += 1;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
const body: string[] = [];
|
||||||
|
let cursor = index + 1;
|
||||||
|
while (cursor < lines.length && !CLOSING_FENCE.test(lines[cursor])) {
|
||||||
|
body.push(lines[cursor]);
|
||||||
|
cursor += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const closing = cursor < lines.length;
|
||||||
|
const urls = urlOnlyBlock(body);
|
||||||
|
|
||||||
|
if (urls) out.push(...urls.map(asLink));
|
||||||
|
else {
|
||||||
|
out.push(lines[index], ...body);
|
||||||
|
if (closing) out.push(lines[cursor]);
|
||||||
|
}
|
||||||
|
|
||||||
|
index = cursor + (closing ? 1 : 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
return out.join("\n");
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Inserts the blank line a table needs to start at all.
|
||||||
|
*
|
||||||
|
* A GFM table cannot interrupt a paragraph, so a table written straight after a
|
||||||
|
* line of prose is not a table: it renders as a paragraph of pipes and dashes,
|
||||||
|
* which is how a perfectly good answer turns into noise. The blank line looks
|
||||||
|
* unnecessary, so models leave it out.
|
||||||
|
*
|
||||||
|
* Only a real table start is touched, which is a row followed by a delimiter
|
||||||
|
* row, so prose that happens to contain a pipe is left alone. Fences are copied
|
||||||
|
* through untouched for the same reason, since a table shown as example code
|
||||||
|
* must stay exactly as written.
|
||||||
|
*/
|
||||||
|
export function separateTables(markdown: string): string {
|
||||||
|
const lines = markdown.split("\n");
|
||||||
|
const out: string[] = [];
|
||||||
|
let index = 0;
|
||||||
|
|
||||||
|
while (index < lines.length) {
|
||||||
|
if (OPENING_FENCE.test(lines[index])) {
|
||||||
|
index = copyFence(lines, out, index);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
const startsTable =
|
||||||
|
TABLE_ROW.test(lines[index]) && index + 1 < lines.length && TABLE_DELIMITER_ROW.test(lines[index + 1]);
|
||||||
|
|
||||||
|
// Only when something would otherwise swallow it. A table already at a block
|
||||||
|
// boundary, after a blank line or after a fenced block, needs nothing: the
|
||||||
|
// fence has already ended the preceding block.
|
||||||
|
const previous = out[out.length - 1];
|
||||||
|
const atBlockBoundary =
|
||||||
|
previous === undefined || previous.trim().length === 0 || CLOSING_FENCE.test(previous);
|
||||||
|
|
||||||
|
if (startsTable && !atBlockBoundary) out.push("");
|
||||||
|
|
||||||
|
out.push(lines[index]);
|
||||||
|
index += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
return out.join("\n");
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The full pre-render pass, in order.
|
* The full pre-render pass, in order.
|
||||||
*
|
*
|
||||||
* Promotion runs first so a promoted block is seen as an ordinary fence by the
|
* Tables are separated first because they change block boundaries: a blank line
|
||||||
* labeller, which is also harmless when promotion already inferred a language.
|
* has to exist before anything else looks at the line. Links are restored next
|
||||||
|
* so a recovered URL leaves no span for the promoter to fence again. Promotion
|
||||||
|
* then runs before labelling, so a promoted block is seen as an ordinary fence
|
||||||
|
* by the labeller, which is also harmless when promotion already inferred a
|
||||||
|
* language.
|
||||||
*/
|
*/
|
||||||
export function repairCodeBlocks(markdown: string): string {
|
export function repairCodeBlocks(markdown: string): string {
|
||||||
return labelCodeFences(promoteStandaloneCodeSpans(markdown));
|
return labelCodeFences(promoteStandaloneCodeSpans(restoreUrlLinks(separateTables(markdown))));
|
||||||
}
|
}
|
||||||
|
|||||||
+151
-38
@@ -1,10 +1,14 @@
|
|||||||
import { Action, ActionPanel, Icon, List } from "@vicinae/api";
|
import { Action, ActionPanel, Icon, LaunchType, List, launchCommand } from "@vicinae/api";
|
||||||
import { useCallback, useEffect, useMemo, useState } from "react";
|
import { useCallback, useEffect, useMemo, useState } from "react";
|
||||||
|
import { readAttachment } from "../attachments";
|
||||||
import { useChat } from "../hooks/useChat";
|
import { useChat } from "../hooks/useChat";
|
||||||
import { buildThreadMarkdown, type TranscriptOptions } from "../markdown";
|
import { useVisionSupport } from "../hooks/useVision";
|
||||||
|
import { buildThreadMarkdown, renderAttachments, type TranscriptOptions } from "../markdown";
|
||||||
import { getLamPreferences } from "../preferences";
|
import { getLamPreferences } from "../preferences";
|
||||||
import { deleteSession, listSessions, loadSession, summarizeSession } from "../sessions";
|
import { deleteSession, listSessions, loadSession, summarizeSession } from "../sessions";
|
||||||
import { buildThread } from "../thread";import type { Session, SessionSummary } from "../types";
|
import { resolveStartup } from "../startup";
|
||||||
|
import { buildThread } from "../thread";
|
||||||
|
import type { PendingAttachment, Session, SessionSummary } from "../types";
|
||||||
import { recencyGroup, truncate } from "../utils";
|
import { recencyGroup, truncate } from "../utils";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -44,15 +48,30 @@ import { recencyGroup, truncate } from "../utils";
|
|||||||
* selection to the first row on every model change
|
* selection to the first row on every model change
|
||||||
* (`GenericListView.selectFirstOnReset`), and streaming changes the model on
|
* (`GenericListView.selectFirstOnReset`), and streaming changes the model on
|
||||||
* every flush, so ordering by recency keeps the chat in progress near the top.
|
* every flush, so ordering by recency keeps the chat in progress near the top.
|
||||||
|
* - Capturing a region does **not** happen here. It is delegated to the `snip`
|
||||||
|
* command, because this view owns a window and the window would be in the
|
||||||
|
* shot. See {@link ChatViewProps}.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
const EMPTY_TRANSCRIPT = "**Ask Lam anything**\n\nType a question in the search bar and press Return.";
|
const EMPTY_TRANSCRIPT = "**Ask Lam anything**\n\nType a question in the search bar and press Return.";
|
||||||
|
|
||||||
|
/** Sent when a snip is attached and Return is pressed before anything is typed. */
|
||||||
|
const DEFAULT_IMAGE_PROMPT = "What is in this image?";
|
||||||
|
|
||||||
|
/** The reminder shown while a snip is attached but not yet sent. */
|
||||||
|
const PENDING_NOTE = "_Ask a question about it, or press Return to send it as it is._";
|
||||||
|
|
||||||
export interface ChatViewProps {
|
export interface ChatViewProps {
|
||||||
/** Opens a specific conversation instead of resuming the latest one. */
|
/** Opens a specific conversation instead of resuming the latest one. */
|
||||||
initialSession?: Session;
|
initialSession?: Session;
|
||||||
/** Text handed over by a deeplink or fallback command. */
|
/** Text handed over by a deeplink or fallback command. */
|
||||||
initialPrompt?: string;
|
initialPrompt?: string;
|
||||||
|
/** A conversation to reload by id, which is how the snip command hands one back. */
|
||||||
|
initialSessionId?: string;
|
||||||
|
/** A captured image to read in, which is how the snip command hands one over. */
|
||||||
|
initialAttachmentPath?: string;
|
||||||
|
/** Something the launching command needs to say, such as a failed capture. */
|
||||||
|
launchNote?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
interface SessionRow {
|
interface SessionRow {
|
||||||
@@ -62,14 +81,26 @@ interface SessionRow {
|
|||||||
isActive: boolean;
|
isActive: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
export function ChatView({ initialSession, initialPrompt }: ChatViewProps) {
|
export function ChatView({
|
||||||
|
initialSession,
|
||||||
|
initialPrompt,
|
||||||
|
initialSessionId,
|
||||||
|
initialAttachmentPath,
|
||||||
|
launchNote,
|
||||||
|
}: ChatViewProps) {
|
||||||
const preferences = useMemo(() => getLamPreferences(), []);
|
const preferences = useMemo(() => getLamPreferences(), []);
|
||||||
const chat = useChat({ preferences, initialSession });
|
const chat = useChat({ preferences, initialSession });
|
||||||
const { replaceSession } = chat;
|
const { replaceSession } = chat;
|
||||||
|
const { canSeeImages } = useVisionSupport(preferences);
|
||||||
|
|
||||||
const [input, setInput] = useState(initialPrompt ?? "");
|
const [input, setInput] = useState(initialPrompt ?? "");
|
||||||
const [summaries, setSummaries] = useState<SessionSummary[] | undefined>(undefined);
|
const [summaries, setSummaries] = useState<SessionSummary[] | undefined>(undefined);
|
||||||
const [resumed, setResumed] = useState(Boolean(initialSession));
|
// One flag, not two. See the startup effect below for why that matters.
|
||||||
|
const [startup, setStartup] = useState<"loading" | "ready">(initialSession ? "ready" : "loading");
|
||||||
|
// A captured image is held here until the user says what they want to know
|
||||||
|
// about it. Nothing is sent to the model on capture alone.
|
||||||
|
const [pending, setPending] = useState<PendingAttachment | undefined>(undefined);
|
||||||
|
const [snipNote, setSnipNote] = useState<string | undefined>(undefined);
|
||||||
|
|
||||||
const activeId = chat.session.id;
|
const activeId = chat.session.id;
|
||||||
|
|
||||||
@@ -85,35 +116,47 @@ export function ChatView({ initialSession, initialPrompt }: ChatViewProps) {
|
|||||||
}, [reload, chat.session.updatedAt]);
|
}, [reload, chat.session.updatedAt]);
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Reopening the command continues the most recent conversation, but only if it
|
* Everything that has to happen before the first render, in one place.
|
||||||
* was used recently. Past the configured idle window it starts a fresh chat
|
*
|
||||||
* instead. The previous conversation is never removed, it just drops down the
|
* This used to be two effects guarding two booleans, and a hand-off launch
|
||||||
* sidebar. Setting the preference to "Never" keeps the last chat open forever.
|
* satisfied the guard of *both* while setting neither, so `resumed` stayed
|
||||||
|
* false and the view sat on "Loading conversations…" forever. Anything that
|
||||||
|
* blocks the first render belongs in one effect with one flag, and the flag is
|
||||||
|
* cleared in a `finally` so that no branch, including a thrown one, can leave
|
||||||
|
* the view spinning.
|
||||||
*/
|
*/
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
if (initialSession) return;
|
|
||||||
|
|
||||||
let cancelled = false;
|
let cancelled = false;
|
||||||
|
|
||||||
void (async () => {
|
void (async () => {
|
||||||
try {
|
try {
|
||||||
const [latest] = await listSessions();
|
const outcome = await resolveStartup(
|
||||||
if (!latest) return;
|
{
|
||||||
|
hasInitialSession: Boolean(initialSession),
|
||||||
|
initialSessionId,
|
||||||
|
initialAttachmentPath,
|
||||||
|
resumeTimeoutMs: preferences.resumeTimeoutMs,
|
||||||
|
now: Date.now(),
|
||||||
|
},
|
||||||
|
{ loadSession, listSessions, readAttachment },
|
||||||
|
);
|
||||||
|
|
||||||
const idleMs = Date.now() - latest.updatedAt;
|
if (cancelled) return;
|
||||||
const { resumeTimeoutMs } = preferences;
|
|
||||||
if (resumeTimeoutMs !== undefined && idleMs > resumeTimeoutMs) return;
|
|
||||||
|
|
||||||
const session = await loadSession(latest.id);
|
if (outcome.session) replaceSession(outcome.session);
|
||||||
if (!cancelled && session) replaceSession(session);
|
if (outcome.attachment) setPending(outcome.attachment);
|
||||||
|
if (outcome.attachmentError) {
|
||||||
|
setSnipNote(`**Could not open the screenshot:** ${outcome.attachmentError}`);
|
||||||
|
}
|
||||||
} finally {
|
} finally {
|
||||||
if (!cancelled) setResumed(true);
|
if (!cancelled) setStartup("ready");
|
||||||
}
|
}
|
||||||
})();
|
})();
|
||||||
|
|
||||||
return () => {
|
return () => {
|
||||||
cancelled = true;
|
cancelled = true;
|
||||||
};
|
};
|
||||||
}, [initialSession, preferences, replaceSession]);
|
}, [initialSession, initialSessionId, initialAttachmentPath, preferences.resumeTimeoutMs, replaceSession]);
|
||||||
|
|
||||||
const thread = useMemo(
|
const thread = useMemo(
|
||||||
() =>
|
() =>
|
||||||
@@ -131,11 +174,31 @@ export function ChatView({ initialSession, initialPrompt }: ChatViewProps) {
|
|||||||
[preferences.showToolCalls, preferences.showReasoning],
|
[preferences.showToolCalls, preferences.showReasoning],
|
||||||
);
|
);
|
||||||
|
|
||||||
const threadMarkdown = useMemo(
|
/**
|
||||||
() =>
|
* The transcript, preceded by any notes about this view's own state.
|
||||||
thread.length === 0 ? EMPTY_TRANSCRIPT : buildThreadMarkdown(thread, transcriptOptions),
|
*
|
||||||
[thread, transcriptOptions],
|
* Notes go first because the transcript reads newest-first: the top of the
|
||||||
);
|
* panel is the text directly under the search bar, which is where the user is
|
||||||
|
* already looking.
|
||||||
|
*/
|
||||||
|
const threadMarkdown = useMemo(() => {
|
||||||
|
const notes = [
|
||||||
|
launchNote,
|
||||||
|
snipNote,
|
||||||
|
// The preview comes first so the captured image is visible before the user
|
||||||
|
// decides what to ask about it, rather than only after sending.
|
||||||
|
pending ? `${renderAttachments([pending])}\n\n${PENDING_NOTE}` : undefined,
|
||||||
|
].filter((note): note is string => Boolean(note));
|
||||||
|
|
||||||
|
// The placeholder is what tells a user that the search bar is the composer.
|
||||||
|
// When a note above already says that, printing both put the same
|
||||||
|
// instruction twice, one line under the other.
|
||||||
|
const empty = notes.length === 0 ? EMPTY_TRANSCRIPT : "";
|
||||||
|
const transcript =
|
||||||
|
thread.length === 0 ? empty : buildThreadMarkdown(thread, transcriptOptions);
|
||||||
|
|
||||||
|
return [...notes, transcript].filter((part) => part.length > 0).join("\n\n");
|
||||||
|
}, [launchNote, snipNote, pending, thread, transcriptOptions]);
|
||||||
|
|
||||||
const rows: SessionRow[] = useMemo(() => {
|
const rows: SessionRow[] = useMemo(() => {
|
||||||
const base = summaries ?? [];
|
const base = summaries ?? [];
|
||||||
@@ -151,10 +214,7 @@ export function ChatView({ initialSession, initialPrompt }: ChatViewProps) {
|
|||||||
if (base.some((summary) => summary.id === activeId)) return mapped;
|
if (base.some((summary) => summary.id === activeId)) return mapped;
|
||||||
|
|
||||||
const active = summarizeSession(chat.session);
|
const active = summarizeSession(chat.session);
|
||||||
return [
|
return [{ id: active.id, title: active.title, updatedAt: active.updatedAt, isActive: true }, ...mapped];
|
||||||
{ id: active.id, title: active.title, updatedAt: active.updatedAt, isActive: true },
|
|
||||||
...mapped,
|
|
||||||
];
|
|
||||||
}, [summaries, activeId, chat.session]);
|
}, [summaries, activeId, chat.session]);
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -194,15 +254,28 @@ export function ChatView({ initialSession, initialPrompt }: ChatViewProps) {
|
|||||||
[activeId, chat, reload],
|
[activeId, chat, reload],
|
||||||
);
|
);
|
||||||
|
|
||||||
|
/** The snip a "show me the screenshot" action would open: the pending one, else the newest. */
|
||||||
|
const latestAttachment =
|
||||||
|
pending ?? [...chat.session.turns].reverse().find((turn) => turn.attachments.length > 0)?.attachments[0];
|
||||||
|
|
||||||
const send = useCallback(
|
const send = useCallback(
|
||||||
(text: string) => {
|
(text: string) => {
|
||||||
chat.ask(text);
|
const typed = text.trim();
|
||||||
|
// With a snip attached, Return has to send even before anything is typed,
|
||||||
|
// otherwise the only way to ask about an image is to type out a question
|
||||||
|
// the image already implies.
|
||||||
|
const prompt = typed || (pending ? DEFAULT_IMAGE_PROMPT : "");
|
||||||
|
if (!prompt) return;
|
||||||
|
|
||||||
|
chat.ask(prompt, pending);
|
||||||
|
setPending(undefined);
|
||||||
|
setSnipNote(undefined);
|
||||||
setInput("");
|
setInput("");
|
||||||
},
|
},
|
||||||
[chat.ask],
|
[chat.ask, pending],
|
||||||
);
|
);
|
||||||
|
|
||||||
const hasPrompt = input.trim().length > 0;
|
const hasPrompt = input.trim().length > 0 || pending !== undefined;
|
||||||
|
|
||||||
const stopAction = <Action title="Stop Generating" icon={Icon.Stop} onAction={chat.cancel} />;
|
const stopAction = <Action title="Stop Generating" icon={Icon.Stop} onAction={chat.cancel} />;
|
||||||
const sendAction = <Action title="Send" icon={Icon.SpeechBubble} onAction={() => send(input)} />;
|
const sendAction = <Action title="Send" icon={Icon.SpeechBubble} onAction={() => send(input)} />;
|
||||||
@@ -216,10 +289,46 @@ export function ChatView({ initialSession, initialPrompt }: ChatViewProps) {
|
|||||||
/>
|
/>
|
||||||
);
|
);
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Hands the capture to the `snip` command instead of doing it here.
|
||||||
|
*
|
||||||
|
* Two reasons, both structural. The window has to be out of the way, and only
|
||||||
|
* the snip command is positioned to close it before the selector appears. And
|
||||||
|
* launching a different command is what makes the hand-off safe: Vicinae's
|
||||||
|
* `activateEntrypoint` treats a launch of the *currently active* view command
|
||||||
|
* as a toggle and closes the window instead of reopening it, so relaying
|
||||||
|
* through `snip` avoids ever asking for `chat` while `chat` is active.
|
||||||
|
*
|
||||||
|
* Absent while streaming, because the hand-off unloads this view and would
|
||||||
|
* abandon an answer mid-sentence.
|
||||||
|
*/
|
||||||
|
const snipAction =
|
||||||
|
canSeeImages && !chat.isStreaming ? (
|
||||||
|
<Action
|
||||||
|
title="Snip a Region"
|
||||||
|
icon={Icon.Screenshot}
|
||||||
|
shortcut={{ key: "s", modifiers: ["cmd", "shift"] }}
|
||||||
|
onAction={() =>
|
||||||
|
void launchCommand({
|
||||||
|
name: "snip",
|
||||||
|
type: LaunchType.UserInitiated,
|
||||||
|
context: { sessionId: chat.session.id },
|
||||||
|
})
|
||||||
|
}
|
||||||
|
/>
|
||||||
|
) : null;
|
||||||
|
|
||||||
|
// A screenshot is a file, so it opens in the system image viewer. Without this
|
||||||
|
// the only way to see what was sent would be to find it on disk by hand.
|
||||||
|
const openSnipAction = latestAttachment ? (
|
||||||
|
<Action.Open title="Open Screenshot" icon={Icon.Image} target={latestAttachment.path} />
|
||||||
|
) : null;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The primary action adapts to the search bar. With text in it, Return sends;
|
* The primary action adapts to the search bar. With text in it, Return sends;
|
||||||
* with it empty (where sending would be a no-op anyway), Return opens the
|
* with it empty (where sending would be a no-op anyway), Return opens the
|
||||||
* highlighted conversation instead.
|
* highlighted conversation instead. A pending snip counts as text, so Return
|
||||||
|
* sends it rather than navigating away.
|
||||||
*/
|
*/
|
||||||
const rowActions = (row: SessionRow) => {
|
const rowActions = (row: SessionRow) => {
|
||||||
const openAction = (
|
const openAction = (
|
||||||
@@ -231,6 +340,8 @@ export function ChatView({ initialSession, initialPrompt }: ChatViewProps) {
|
|||||||
{chat.isStreaming ? stopAction : hasPrompt ? sendAction : openAction}
|
{chat.isStreaming ? stopAction : hasPrompt ? sendAction : openAction}
|
||||||
{hasPrompt ? openAction : null}
|
{hasPrompt ? openAction : null}
|
||||||
{newChatAction}
|
{newChatAction}
|
||||||
|
{snipAction}
|
||||||
|
{openSnipAction}
|
||||||
<Action
|
<Action
|
||||||
title="Delete Conversation"
|
title="Delete Conversation"
|
||||||
icon={Icon.Trash}
|
icon={Icon.Trash}
|
||||||
@@ -244,13 +355,13 @@ export function ChatView({ initialSession, initialPrompt }: ChatViewProps) {
|
|||||||
);
|
);
|
||||||
};
|
};
|
||||||
|
|
||||||
if (!resumed || summaries === undefined) {
|
if (startup === "loading" || summaries === undefined) {
|
||||||
return <List isLoading navigationTitle="Lam" searchBarPlaceholder="Loading conversations…" />;
|
return <List isLoading navigationTitle="Lam" searchBarPlaceholder="Loading conversations…" />;
|
||||||
}
|
}
|
||||||
|
|
||||||
const placeholder = chat.status
|
const placeholder = chat.status ? `${chat.status}. Press ⏎ to stop` : "Lam is thinking. Press ⏎ to stop";
|
||||||
? `${chat.status}. Press ⏎ to stop`
|
|
||||||
: "Lam is thinking. Press ⏎ to stop";
|
const idlePlaceholder = pending ? "Ask about the screenshot…" : "Ask Lam anything…";
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<List
|
<List
|
||||||
@@ -262,11 +373,13 @@ export function ChatView({ initialSession, initialPrompt }: ChatViewProps) {
|
|||||||
// Raycast puts the conversation title in a header; Vicinae's nearest
|
// Raycast puts the conversation title in a header; Vicinae's nearest
|
||||||
// equivalent is the status-bar title, so the open chat is named there too.
|
// equivalent is the status-bar title, so the open chat is named there too.
|
||||||
navigationTitle={truncate(chat.session.title, 40)}
|
navigationTitle={truncate(chat.session.title, 40)}
|
||||||
searchBarPlaceholder={chat.isStreaming ? placeholder : "Ask Lam anything…"}
|
searchBarPlaceholder={chat.isStreaming ? placeholder : idlePlaceholder}
|
||||||
actions={
|
actions={
|
||||||
<ActionPanel>
|
<ActionPanel>
|
||||||
{chat.isStreaming ? stopAction : sendAction}
|
{chat.isStreaming ? stopAction : sendAction}
|
||||||
{newChatAction}
|
{newChatAction}
|
||||||
|
{snipAction}
|
||||||
|
{openSnipAction}
|
||||||
</ActionPanel>
|
</ActionPanel>
|
||||||
}
|
}
|
||||||
>
|
>
|
||||||
|
|||||||
@@ -24,7 +24,9 @@ export const DEFAULT_SYSTEM_PROMPT = `You are Lam, a friendly assistant in the V
|
|||||||
|
|
||||||
Sound like a person, not a document. Use contractions, talk about "you" and "I", and a short natural remark is welcome ("Sure, easy one", "Ah, that trips people up"). Performed enthusiasm and filler added to seem helpful are not.
|
Sound like a person, not a document. Use contractions, talk about "you" and "I", and a short natural remark is welcome ("Sure, easy one", "Ah, that trips people up"). Performed enthusiasm and filler added to seem helpful are not.
|
||||||
|
|
||||||
Lead with the answer, then the detail. This renders in a narrow window, so stay brief: a tight sentence or two, or a short list, then stop. Keep paragraphs short and skip large headings and wide tables.
|
Lead with the answer, then the detail. This renders in a narrow window, so stay brief: a tight sentence or two, or a short list, then stop. Keep paragraphs short.
|
||||||
|
|
||||||
|
Use the markdown this window actually draws rather than flattening everything into prose. It renders tables, bulleted and numbered lists, headings, blockquotes, bold, italics, inline code and fenced code, so pick the shape that fits. A table is the right shape for comparing three or more things across the same few attributes, so write the table itself: never hand back a table as a bulleted list of rows, and never describe a table you could simply write. Keep it to about four narrow columns so it fits, give it a header row, and leave a blank line before it. A list is for steps or for independent items. Reach for a heading only when an answer genuinely has sections, since at this size a heading usually costs a line and buys nothing.
|
||||||
|
|
||||||
Write plainly:
|
Write plainly:
|
||||||
- Never use an em dash. Use a comma, a colon, or a full stop.
|
- Never use an em dash. Use a comma, a colon, or a full stop.
|
||||||
@@ -36,6 +38,8 @@ Write plainly:
|
|||||||
|
|
||||||
Fence every command and every snippet, and always name the language: bash for shell and terminal commands, then json, sql, yaml, or the language for code (ts, js, python, rust, go). Never leave code inline in a sentence, and never open a fence without a language. Put no $ or shell prompt inside the block so it can be copied and run as-is. Ignore the formatting of earlier answers in this conversation: if one put a command in inline code, fence it anyway.
|
Fence every command and every snippet, and always name the language: bash for shell and terminal commands, then json, sql, yaml, or the language for code (ts, js, python, rust, go). Never leave code inline in a sentence, and never open a fence without a language. Put no $ or shell prompt inside the block so it can be copied and run as-is. Ignore the formatting of earlier answers in this conversation: if one put a command in inline code, fence it anyway.
|
||||||
|
|
||||||
|
A web address is a link, never code. A markdown link, written as [label](https://example.com), is clickable in this window, while a URL inside backticks or a fenced block is dead text the user cannot open. So never fence a URL, never put one in inline code, and never answer "here are the links" with a snippet. Label a link with what it is ("OpenClickly on GitHub"), and when you give several, put each on its own bullet with its own link.
|
||||||
|
|
||||||
A fenced block is never a whole answer on its own. Say in one plain sentence what the code does or when to reach for it, so the reply reads like something a person wrote rather than a bare snippet.
|
A fenced block is never a whole answer on its own. Say in one plain sentence what the code does or when to reach for it, so the reply reads like something a person wrote rather than a bare snippet.
|
||||||
|
|
||||||
Call a tool when it makes the answer more accurate. Anything time-sensitive or past your cutoff should be checked rather than guessed, and never invent tool output. If a tool fails, say so and answer with what you know.
|
Call a tool when it makes the answer more accurate. Anything time-sensitive or past your cutoff should be checked rather than guessed, and never invent tool output. If a tool fails, say so and answer with what you know.
|
||||||
|
|||||||
+14
-2
@@ -8,8 +8,17 @@ import type { ApiMessage, ChatTurn } from "./types";
|
|||||||
* messages from a previous turn are intentionally dropped: they are noise once
|
* messages from a previous turn are intentionally dropped: they are noise once
|
||||||
* the answer is written, and replaying them would re-send large page contents
|
* the answer is written, and replaying them would re-send large page contents
|
||||||
* on every subsequent request.
|
* on every subsequent request.
|
||||||
|
*
|
||||||
|
* Attached images are treated the same way, and for the same reason. They are
|
||||||
|
* the largest thing a request can carry, so only the question being asked now
|
||||||
|
* brings one. A follow-up about a screenshot needs the image attached again.
|
||||||
*/
|
*/
|
||||||
export function buildMessages(systemPrompt: string, turns: ChatTurn[], prompt: string): ApiMessage[] {
|
export function buildMessages(
|
||||||
|
systemPrompt: string,
|
||||||
|
turns: ChatTurn[],
|
||||||
|
prompt: string,
|
||||||
|
images?: string[],
|
||||||
|
): ApiMessage[] {
|
||||||
const messages: ApiMessage[] = [{ role: "system", content: systemPrompt }];
|
const messages: ApiMessage[] = [{ role: "system", content: systemPrompt }];
|
||||||
|
|
||||||
for (const turn of turns) {
|
for (const turn of turns) {
|
||||||
@@ -18,7 +27,10 @@ export function buildMessages(systemPrompt: string, turns: ChatTurn[], prompt: s
|
|||||||
messages.push({ role: "assistant", content: turn.response });
|
messages.push({ role: "assistant", content: turn.response });
|
||||||
}
|
}
|
||||||
|
|
||||||
messages.push({ role: "user", content: prompt });
|
const question: ApiMessage = { role: "user", content: prompt };
|
||||||
|
if (images && images.length > 0) question.images = images;
|
||||||
|
messages.push(question);
|
||||||
|
|
||||||
return messages;
|
return messages;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,11 +1,12 @@
|
|||||||
import { useCallback, useEffect, useRef, useState } from "react";
|
import { useCallback, useEffect, useRef, useState } from "react";
|
||||||
import { runAgent } from "../agent";
|
import { runAgent } from "../agent";
|
||||||
|
import { toStoredAttachment } from "../attachments";
|
||||||
import { buildMessages, composeSystemPrompt } from "../history";
|
import { buildMessages, composeSystemPrompt } from "../history";
|
||||||
import { OllamaError } from "../ollama";
|
import { OllamaError } from "../ollama";
|
||||||
import type { ResolvedPreferences } from "../preferences";
|
import type { ResolvedPreferences } from "../preferences";
|
||||||
import { createSession, createTurn, saveSession, withTitleFromFirstTurn } from "../sessions";
|
import { createSession, createTurn, saveSession, withTitleFromFirstTurn } from "../sessions";
|
||||||
import { toolsFor } from "../tools";
|
import { toolsFor } from "../tools";
|
||||||
import type { Session } from "../types";
|
import type { PendingAttachment, Session } from "../types";
|
||||||
import { errorMessage, isAbortError } from "../utils";
|
import { errorMessage, isAbortError } from "../utils";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -41,7 +42,7 @@ export interface UseChat {
|
|||||||
/** Short activity line such as "Searching the web", shown while a turn is in flight. */
|
/** Short activity line such as "Searching the web", shown while a turn is in flight. */
|
||||||
status?: string;
|
status?: string;
|
||||||
error?: string;
|
error?: string;
|
||||||
ask: (prompt: string) => void;
|
ask: (prompt: string, attachment?: PendingAttachment) => void;
|
||||||
cancel: () => void;
|
cancel: () => void;
|
||||||
replaceSession: (session: Session) => void;
|
replaceSession: (session: Session) => void;
|
||||||
newSession: () => void;
|
newSession: () => void;
|
||||||
@@ -119,16 +120,22 @@ export function useChat(options: UseChatOptions): UseChat {
|
|||||||
}, []);
|
}, []);
|
||||||
|
|
||||||
const ask = useCallback(
|
const ask = useCallback(
|
||||||
(rawPrompt: string) => {
|
(rawPrompt: string, attachment?: PendingAttachment) => {
|
||||||
const prompt = rawPrompt.trim();
|
const prompt = rawPrompt.trim();
|
||||||
if (!prompt || liveRef.current) return;
|
if (!prompt || liveRef.current) return;
|
||||||
|
|
||||||
const preferences = preferencesRef.current;
|
const preferences = preferencesRef.current;
|
||||||
const base = sessionRef.current;
|
const base = sessionRef.current;
|
||||||
|
|
||||||
|
// Only the path, mime type and dimensions are kept on the turn. The base64
|
||||||
|
// payload is used for this one request and then dropped, so it never
|
||||||
|
// reaches storage. The dimensions have to survive because the transcript
|
||||||
|
// lays the preview out long after the payload is gone.
|
||||||
|
const attachments = attachment ? [toStoredAttachment(attachment)] : [];
|
||||||
|
|
||||||
// The turn is inserted immediately with an empty response so the UI shows
|
// The turn is inserted immediately with an empty response so the UI shows
|
||||||
// the question while the answer streams in beneath it.
|
// the question while the answer streams in beneath it.
|
||||||
const turn = createTurn(prompt, preferences.model);
|
const turn = createTurn(prompt, preferences.model, Date.now(), attachments);
|
||||||
const started = withTitleFromFirstTurn({
|
const started = withTitleFromFirstTurn({
|
||||||
...base,
|
...base,
|
||||||
model: preferences.model,
|
model: preferences.model,
|
||||||
@@ -152,6 +159,7 @@ export function useChat(options: UseChatOptions): UseChat {
|
|||||||
composeSystemPrompt(preferences.systemPrompt, preferences.webEnabled),
|
composeSystemPrompt(preferences.systemPrompt, preferences.webEnabled),
|
||||||
base.turns,
|
base.turns,
|
||||||
prompt,
|
prompt,
|
||||||
|
attachment ? [attachment.base64] : undefined,
|
||||||
);
|
);
|
||||||
|
|
||||||
void (async () => {
|
void (async () => {
|
||||||
|
|||||||
@@ -0,0 +1,54 @@
|
|||||||
|
import { useEffect, useState } from "react";
|
||||||
|
import type { ResolvedPreferences } from "../preferences";
|
||||||
|
import { visionSupport } from "../vision";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Whether the current model accepts images, for gating the snip action.
|
||||||
|
*
|
||||||
|
* Fails closed: only a confirmed `"yes"` turns the feature on. A model that
|
||||||
|
* cannot see would otherwise accept a screenshot and answer as though it had,
|
||||||
|
* which is worse than the action not being offered.
|
||||||
|
*
|
||||||
|
* The wording for a *failed* check lives in the snip command rather than here,
|
||||||
|
* because that is the only place with somewhere to say it (a HUD). This hook
|
||||||
|
* only decides whether to show an action.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export interface VisionState {
|
||||||
|
/** True only once Ollama has confirmed the model reports the `vision` capability. */
|
||||||
|
canSeeImages: boolean;
|
||||||
|
/** True while an answer for this model is still outstanding. */
|
||||||
|
isChecking: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function useVisionSupport(preferences: ResolvedPreferences): VisionState {
|
||||||
|
const [canSeeImages, setCanSeeImages] = useState(false);
|
||||||
|
const [isChecking, setIsChecking] = useState(true);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
let cancelled = false;
|
||||||
|
const controller = new AbortController();
|
||||||
|
|
||||||
|
setIsChecking(true);
|
||||||
|
|
||||||
|
void visionSupport({
|
||||||
|
host: preferences.host,
|
||||||
|
apiKey: preferences.apiKey,
|
||||||
|
model: preferences.model,
|
||||||
|
signal: controller.signal,
|
||||||
|
})
|
||||||
|
.then((support) => {
|
||||||
|
if (!cancelled) setCanSeeImages(support === "yes");
|
||||||
|
})
|
||||||
|
.finally(() => {
|
||||||
|
if (!cancelled) setIsChecking(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
return () => {
|
||||||
|
cancelled = true;
|
||||||
|
controller.abort();
|
||||||
|
};
|
||||||
|
}, [preferences.host, preferences.apiKey, preferences.model]);
|
||||||
|
|
||||||
|
return { canSeeImages, isChecking };
|
||||||
|
}
|
||||||
+28
-3
@@ -1,5 +1,6 @@
|
|||||||
|
import { imageSource } from "./attachments";
|
||||||
import { repairCodeBlocks } from "./code";
|
import { repairCodeBlocks } from "./code";
|
||||||
import type { ChatTurn, ThreadMessage } from "./types";
|
import type { Attachment, ChatTurn, ThreadMessage } from "./types";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Markdown rendering for the detail panel.
|
* Markdown rendering for the detail panel.
|
||||||
@@ -22,6 +23,24 @@ import type { ChatTurn, ThreadMessage } from "./types";
|
|||||||
|
|
||||||
export const STREAMING_CURSOR = "▍";
|
export const STREAMING_CURSOR = "▍";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Renders attachments as markdown images.
|
||||||
|
*
|
||||||
|
* Vicinae does resolve a local file: `MdImage.qml` draws it block-level and
|
||||||
|
* centred, uses the alt text as an italic caption underneath, and caps the
|
||||||
|
* height at 200px no matter what is asked for. So this is a thumbnail rather
|
||||||
|
* than something readable, which is why the action panel also offers Open
|
||||||
|
* Screenshot for the full image.
|
||||||
|
*
|
||||||
|
* The alt text is short on purpose. Whatever is here is drawn centred under the
|
||||||
|
* image, so a sentence reads as a stray line of prose floating in the middle of
|
||||||
|
* the pane. It is also the only thing left if the file has since been deleted,
|
||||||
|
* which is why it is not empty.
|
||||||
|
*/
|
||||||
|
export function renderAttachments(attachments: Attachment[]): string {
|
||||||
|
return attachments.map((attachment) => `})`).join("\n\n");
|
||||||
|
}
|
||||||
|
|
||||||
/** Which optional sections the transcript includes. Driven by preferences. */
|
/** Which optional sections the transcript includes. Driven by preferences. */
|
||||||
export interface TranscriptOptions {
|
export interface TranscriptOptions {
|
||||||
showToolCalls: boolean;
|
showToolCalls: boolean;
|
||||||
@@ -49,7 +68,12 @@ function renderTools(message: ThreadMessage): string {
|
|||||||
|
|
||||||
function renderMessage(message: ThreadMessage, options: TranscriptOptions): string {
|
function renderMessage(message: ThreadMessage, options: TranscriptOptions): string {
|
||||||
// The question is identified by its styling alone.
|
// The question is identified by its styling alone.
|
||||||
if (message.role === "user") return quote(message.text);
|
if (message.role === "user") {
|
||||||
|
const images = renderAttachments(message.attachments);
|
||||||
|
const asked = quote(message.text);
|
||||||
|
|
||||||
|
return images ? `${asked}\n\n${images}` : asked;
|
||||||
|
}
|
||||||
|
|
||||||
// While a tool is running there is no answer text yet, so the activity line
|
// While a tool is running there is no answer text yet, so the activity line
|
||||||
// stands in and the user can see what Lam is actually doing.
|
// stands in and the user can see what Lam is actually doing.
|
||||||
@@ -138,9 +162,10 @@ export function sessionToPlainText(turns: ChatTurn[]): string {
|
|||||||
return turns
|
return turns
|
||||||
.map((turn) => {
|
.map((turn) => {
|
||||||
const question = `**You**\n\n${turn.prompt}`;
|
const question = `**You**\n\n${turn.prompt}`;
|
||||||
|
const attached = turn.attachments.length > 0 ? "\n\n_(screenshot attached)_" : "";
|
||||||
const answer = turn.response.trim() || (turn.error ? "" : "_(no response)_");
|
const answer = turn.response.trim() || (turn.error ? "" : "_(no response)_");
|
||||||
const failure = turn.error ? `\n\n**[error]** ${turn.error}` : "";
|
const failure = turn.error ? `\n\n**[error]** ${turn.error}` : "";
|
||||||
return `${question}\n\n**Lam**\n\n${answer}${failure}`;
|
return `${question}${attached}\n\n**Lam**\n\n${answer}${failure}`;
|
||||||
})
|
})
|
||||||
.join("\n\n---\n\n");
|
.join("\n\n---\n\n");
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -219,6 +219,44 @@ export async function listModels(options: ListModelsOptions): Promise<ModelInfo[
|
|||||||
.sort((a, b) => a.name.localeCompare(b.name));
|
.sort((a, b) => a.name.localeCompare(b.name));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface ModelCapabilitiesOptions {
|
||||||
|
host: string;
|
||||||
|
apiKey: string;
|
||||||
|
model: string;
|
||||||
|
signal?: AbortSignal;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The features a model reports for itself, straight from `POST /api/show`.
|
||||||
|
*
|
||||||
|
* `GET /api/tags` is not enough here: it returns names, sizes and quantization
|
||||||
|
* levels and says nothing about what the model can actually do. The capability
|
||||||
|
* list is what turns "image input" from a guess about the model's name into a
|
||||||
|
* fact, so it is worth the extra request.
|
||||||
|
*/
|
||||||
|
export async function modelCapabilities(options: ModelCapabilitiesOptions): Promise<string[]> {
|
||||||
|
const host = normalizeHost(options.host);
|
||||||
|
|
||||||
|
const response = await fetch(`${host}/api/show`, {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "Content-Type": "application/json", ...authHeaders(options.apiKey) },
|
||||||
|
body: JSON.stringify({ model: options.model }),
|
||||||
|
signal: options.signal,
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!response.ok) {
|
||||||
|
throw describeHttpError(response.status, await safeText(response), host);
|
||||||
|
}
|
||||||
|
|
||||||
|
const payload = (await response.json()) as { capabilities?: string[] };
|
||||||
|
return payload.capabilities ?? [];
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Ollama reports image input as the `vision` capability. */
|
||||||
|
export function supportsVision(capabilities: string[]): boolean {
|
||||||
|
return capabilities.includes("vision");
|
||||||
|
}
|
||||||
|
|
||||||
export interface WebSearchResult {
|
export interface WebSearchResult {
|
||||||
title: string;
|
title: string;
|
||||||
url: string;
|
url: string;
|
||||||
|
|||||||
@@ -14,6 +14,15 @@ export const ToolCallRecordSchema = z.object({
|
|||||||
isError: z.boolean(),
|
isError: z.boolean(),
|
||||||
});
|
});
|
||||||
|
|
||||||
|
export const AttachmentSchema = z.object({
|
||||||
|
path: z.string(),
|
||||||
|
mime: z.string().default("image/png"),
|
||||||
|
// Absent on sessions written before the header was read, which is fine: a
|
||||||
|
// preview without dimensions falls back to the renderer's own sizing.
|
||||||
|
width: z.number().optional(),
|
||||||
|
height: z.number().optional(),
|
||||||
|
});
|
||||||
|
|
||||||
export const ChatTurnSchema = z.object({
|
export const ChatTurnSchema = z.object({
|
||||||
id: z.string(),
|
id: z.string(),
|
||||||
prompt: z.string(),
|
prompt: z.string(),
|
||||||
@@ -23,6 +32,9 @@ export const ChatTurnSchema = z.object({
|
|||||||
createdAt: z.number(),
|
createdAt: z.number(),
|
||||||
error: z.string().optional(),
|
error: z.string().optional(),
|
||||||
toolCalls: z.array(ToolCallRecordSchema).default([]),
|
toolCalls: z.array(ToolCallRecordSchema).default([]),
|
||||||
|
// Sessions written before image input existed have no attachments key, so the
|
||||||
|
// default is what keeps them loading rather than being rejected as corrupt.
|
||||||
|
attachments: z.array(AttachmentSchema).default([]),
|
||||||
});
|
});
|
||||||
|
|
||||||
export const SessionSchema = z.object({
|
export const SessionSchema = z.object({
|
||||||
|
|||||||
+8
-2
@@ -1,7 +1,7 @@
|
|||||||
import { LocalStorage } from "@vicinae/api";
|
import { LocalStorage } from "@vicinae/api";
|
||||||
import { MAX_SESSIONS, MAX_STORED_CHARS_PER_TURN, MAX_TURNS_PER_SESSION, STORAGE_INDEX_KEY, STORAGE_SESSION_PREFIX, TITLE_MAX_LENGTH } from "./constants";
|
import { MAX_SESSIONS, MAX_STORED_CHARS_PER_TURN, MAX_TURNS_PER_SESSION, STORAGE_INDEX_KEY, STORAGE_SESSION_PREFIX, TITLE_MAX_LENGTH } from "./constants";
|
||||||
import { SessionIndexSchema, SessionSchema } from "./schemas";
|
import { SessionIndexSchema, SessionSchema } from "./schemas";
|
||||||
import type { ChatTurn, Session, SessionSummary } from "./types";
|
import type { Attachment, ChatTurn, Session, SessionSummary } from "./types";
|
||||||
import { titleFromPrompt, truncate } from "./utils";
|
import { titleFromPrompt, truncate } from "./utils";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -56,7 +56,12 @@ export function createSession(model: string, now: number = Date.now()): Session
|
|||||||
}
|
}
|
||||||
|
|
||||||
/** Creates the turn record for a prompt that is about to be sent. */
|
/** Creates the turn record for a prompt that is about to be sent. */
|
||||||
export function createTurn(prompt: string, model: string, now: number = Date.now()): ChatTurn {
|
export function createTurn(
|
||||||
|
prompt: string,
|
||||||
|
model: string,
|
||||||
|
now: number = Date.now(),
|
||||||
|
attachments: Attachment[] = [],
|
||||||
|
): ChatTurn {
|
||||||
return {
|
return {
|
||||||
id: newId(),
|
id: newId(),
|
||||||
prompt,
|
prompt,
|
||||||
@@ -65,6 +70,7 @@ export function createTurn(prompt: string, model: string, now: number = Date.now
|
|||||||
model,
|
model,
|
||||||
createdAt: now,
|
createdAt: now,
|
||||||
toolCalls: [],
|
toolCalls: [],
|
||||||
|
attachments,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,74 @@
|
|||||||
|
import { environment } from "@vicinae/api";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { CapturePermissionError, NoCaptureToolError } from "./capture";
|
||||||
|
import { errorMessage } from "./utils";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The seam between the snip command and the chat command.
|
||||||
|
*
|
||||||
|
* The capture cannot happen inside the chat view. Vicinae shows the launcher
|
||||||
|
* window when it activates a command, so a region selector started from a view
|
||||||
|
* is dragging around a panel parked in the middle of the screen. Capturing from
|
||||||
|
* a `no-view` command and handing the file over is what keeps the window out of
|
||||||
|
* the way, which means the two commands need a small agreed contract.
|
||||||
|
*
|
||||||
|
* Imports Vicinae for `environment.supportPath` only, and nothing else in the
|
||||||
|
* logic layer depends on it.
|
||||||
|
*/
|
||||||
|
|
||||||
|
/** What the snip command passes to the chat command through `launchCommand`. */
|
||||||
|
export interface SnipLaunchContext {
|
||||||
|
/** Absolute path of the captured PNG. Absent when the capture failed. */
|
||||||
|
snipPath?: string;
|
||||||
|
/** Why a capture failed, so the chat can explain itself rather than look empty. */
|
||||||
|
snipError?: string;
|
||||||
|
/** The conversation the snip was started from, restored so it is not lost. */
|
||||||
|
sessionId?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Snips live in the extension's own support directory rather than a temp
|
||||||
|
* directory: a session stores the path, and a path that survives a reboot is the
|
||||||
|
* difference between an old conversation showing its screenshot and pointing at
|
||||||
|
* nothing.
|
||||||
|
*/
|
||||||
|
export function snipDirectory(): string {
|
||||||
|
return join(environment.supportPath, "snips");
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Reads the handoff payload.
|
||||||
|
*
|
||||||
|
* A launch context is whatever the launching command put there, so every field
|
||||||
|
* is checked rather than trusted: a wrong type would otherwise reach the file
|
||||||
|
* system as a path.
|
||||||
|
*/
|
||||||
|
export function readSnipContext(launchContext: unknown): SnipLaunchContext {
|
||||||
|
if (!launchContext || typeof launchContext !== "object") return {};
|
||||||
|
|
||||||
|
const { snipPath, snipError, sessionId } = launchContext as Record<string, unknown>;
|
||||||
|
const asString = (value: unknown) => (typeof value === "string" && value.length > 0 ? value : undefined);
|
||||||
|
|
||||||
|
return {
|
||||||
|
snipPath: asString(snipPath),
|
||||||
|
snipError: asString(snipError),
|
||||||
|
sessionId: asString(sessionId),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Turns a capture failure into something worth showing.
|
||||||
|
*
|
||||||
|
* The two failures that carry a hint are the only two the user can do anything
|
||||||
|
* about, and the hints are useless if they are dropped: "no capture tool is
|
||||||
|
* installed" and "the system is blocking screen capture" need completely
|
||||||
|
* different responses from completely different places.
|
||||||
|
*/
|
||||||
|
export function describeCaptureFailure(thrown: unknown): string {
|
||||||
|
const actionable =
|
||||||
|
thrown instanceof CapturePermissionError || thrown instanceof NoCaptureToolError
|
||||||
|
? thrown.hint
|
||||||
|
: undefined;
|
||||||
|
|
||||||
|
return actionable ? `${errorMessage(thrown)} ${actionable}` : errorMessage(thrown);
|
||||||
|
}
|
||||||
@@ -0,0 +1,96 @@
|
|||||||
|
import type { PendingAttachment, Session, SessionSummary } from "./types";
|
||||||
|
import { errorMessage } from "./utils";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Everything the chat view resolves before its first render.
|
||||||
|
*
|
||||||
|
* Pulled out of the component and made pure because this is the part that keeps
|
||||||
|
* going wrong. Three things can happen on open, and all of them are
|
||||||
|
* asynchronous: a launch hands over a captured file, it may also name the
|
||||||
|
* conversation to return to, and an ordinary open resumes the most recent
|
||||||
|
* conversation if it is fresh enough. Every path has to end in a rendered chat.
|
||||||
|
*
|
||||||
|
* As a plain function it is testable without mounting anything, which matters
|
||||||
|
* because the alternative is finding out from a screenshot that a view is
|
||||||
|
* spinning forever.
|
||||||
|
*
|
||||||
|
* Imports nothing from Vicinae, so it belongs to the logic layer.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export interface StartupInput {
|
||||||
|
/** A conversation already in hand, which is how the sessions command pushes this view. */
|
||||||
|
hasInitialSession: boolean;
|
||||||
|
/** A conversation to reopen by id, handed over by the snip command. */
|
||||||
|
initialSessionId?: string;
|
||||||
|
/** A captured image to read in. */
|
||||||
|
initialAttachmentPath?: string;
|
||||||
|
/** Idle time past which an ordinary open starts fresh. Undefined means never time out. */
|
||||||
|
resumeTimeoutMs?: number;
|
||||||
|
/** Injected so the resume window is deterministic under test. */
|
||||||
|
now: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The side effects, injected so nothing here touches storage or the file system directly. */
|
||||||
|
export interface StartupPorts {
|
||||||
|
loadSession: (id: string) => Promise<Session | undefined>;
|
||||||
|
listSessions: () => Promise<SessionSummary[]>;
|
||||||
|
readAttachment: (path: string) => Promise<PendingAttachment>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface StartupResult {
|
||||||
|
/** The conversation to show, when one was restored. */
|
||||||
|
session?: Session;
|
||||||
|
/** The image that was handed over, when one could be read. */
|
||||||
|
attachment?: PendingAttachment;
|
||||||
|
/** Why a handed-over image could not be read, so the view can say so. */
|
||||||
|
attachmentError?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** True when the launch handed something over, which also means "do not resume". */
|
||||||
|
export function isHandoff(input: Pick<StartupInput, "initialSessionId" | "initialAttachmentPath">): boolean {
|
||||||
|
return Boolean(input.initialSessionId || input.initialAttachmentPath);
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function resolveStartup(input: StartupInput, ports: StartupPorts): Promise<StartupResult> {
|
||||||
|
const result: StartupResult = {};
|
||||||
|
|
||||||
|
if (input.initialSessionId) {
|
||||||
|
// A hand-off names the conversation to return to, so that one is restored
|
||||||
|
// rather than whichever chat happens to be most recent. By id matters
|
||||||
|
// because taking a snip is not a turn, so nothing was saved in between.
|
||||||
|
// A session that has since been deleted just means starting fresh.
|
||||||
|
result.session = await ports.loadSession(input.initialSessionId).catch(() => undefined);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (input.initialAttachmentPath) {
|
||||||
|
try {
|
||||||
|
result.attachment = await ports.readAttachment(input.initialAttachmentPath);
|
||||||
|
} catch (thrown) {
|
||||||
|
// A missing file leaves an empty chat with an explanation, which beats
|
||||||
|
// refusing to open at all.
|
||||||
|
result.attachmentError = errorMessage(thrown);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Nothing was handed over, so the idle-resume rule applies: reopening the
|
||||||
|
// command continues the most recent conversation, but only if it was used
|
||||||
|
// recently. The previous conversation is never removed either way, it just
|
||||||
|
// drops down the sidebar.
|
||||||
|
if (!input.hasInitialSession && !isHandoff(input)) {
|
||||||
|
const latest = await ports
|
||||||
|
.listSessions()
|
||||||
|
.then((sessions) => sessions[0])
|
||||||
|
.catch(() => undefined);
|
||||||
|
|
||||||
|
if (latest) {
|
||||||
|
const idleMs = input.now - latest.updatedAt;
|
||||||
|
const fresh = input.resumeTimeoutMs === undefined || idleMs <= input.resumeTimeoutMs;
|
||||||
|
|
||||||
|
if (fresh) {
|
||||||
|
result.session = await ports.loadSession(latest.id).catch(() => undefined);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return result;
|
||||||
|
}
|
||||||
@@ -43,6 +43,8 @@ export function buildThread(turns: ChatTurn[], live: LiveState): ThreadMessage[]
|
|||||||
// Only while there is nothing to read yet. Once tokens land, the
|
// Only while there is nothing to read yet. Once tokens land, the
|
||||||
// placeholder it feeds is gone anyway.
|
// placeholder it feeds is gone anyway.
|
||||||
status: isLive && text.length === 0 ? live.status : undefined,
|
status: isLive && text.length === 0 ? live.status : undefined,
|
||||||
|
// Only a question carries attachments, so the answer never has one.
|
||||||
|
attachments: [],
|
||||||
});
|
});
|
||||||
|
|
||||||
messages.push({
|
messages.push({
|
||||||
@@ -53,6 +55,7 @@ export function buildThread(turns: ChatTurn[], live: LiveState): ThreadMessage[]
|
|||||||
isStreaming: false,
|
isStreaming: false,
|
||||||
thinking: "",
|
thinking: "",
|
||||||
toolCalls: [],
|
toolCalls: [],
|
||||||
|
attachments: turn.attachments,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -37,8 +37,53 @@ export interface ApiMessage {
|
|||||||
tool_calls?: ToolCall[];
|
tool_calls?: ToolCall[];
|
||||||
/** Only set on `role: "tool"` messages, naming the tool that produced this content. */
|
/** Only set on `role: "tool"` messages, naming the tool that produced this content. */
|
||||||
tool_name?: string;
|
tool_name?: string;
|
||||||
|
/**
|
||||||
|
* Base64 image data, for vision models.
|
||||||
|
*
|
||||||
|
* Ollama only reads these on a `user` message (`gpt-oss` style tool roles have
|
||||||
|
* no images). They are the one genuinely large thing in a request, so they are
|
||||||
|
* attached to the question they belong to and never replayed later, the same
|
||||||
|
* way tool output is not replayed.
|
||||||
|
*/
|
||||||
|
images?: string[];
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A file the user attached to a question.
|
||||||
|
*
|
||||||
|
* Only the path is stored. A screenshot runs to megabytes, and
|
||||||
|
* `MAX_STORED_CHARS_PER_TURN` exists precisely so one turn cannot dominate
|
||||||
|
* LocalStorage, so the bytes stay on disk and are read when the turn is sent.
|
||||||
|
*/
|
||||||
|
export interface Attachment {
|
||||||
|
/** Absolute path to the image on disk. */
|
||||||
|
path: string;
|
||||||
|
/** Sent to Ollama next to the base64 payload. */
|
||||||
|
mime: string;
|
||||||
|
/**
|
||||||
|
* Pixel dimensions, read from the file's header.
|
||||||
|
*
|
||||||
|
* Kept because the transcript has to lay the preview out long after the file
|
||||||
|
* was read, and Vicinae sizes an image's container to the request rather than
|
||||||
|
* to the image, which leaves dead space unless the request has the right
|
||||||
|
* aspect ratio. Absent for a format whose header we do not parse.
|
||||||
|
*/
|
||||||
|
width?: number;
|
||||||
|
height?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* An attachment that has been read and is ready to send, held in the view while
|
||||||
|
* the user types their question.
|
||||||
|
*
|
||||||
|
* The base64 payload lives here and nowhere else. It is needed for exactly one
|
||||||
|
* request, so it is deliberately not part of `Attachment` and never reaches
|
||||||
|
* storage.
|
||||||
|
*/
|
||||||
|
export interface PendingAttachment extends Attachment {
|
||||||
|
/** Base64 of the image, with no `data:` URL prefix. Ollama wants it bare. */
|
||||||
|
base64: string;
|
||||||
|
}
|
||||||
/** What a tool actually returned during a turn, kept for display and debugging. */
|
/** What a tool actually returned during a turn, kept for display and debugging. */
|
||||||
export interface ToolCallRecord {
|
export interface ToolCallRecord {
|
||||||
name: string;
|
name: string;
|
||||||
@@ -58,6 +103,8 @@ export interface ChatTurn {
|
|||||||
/** Set when the turn failed, so a stored session can show what went wrong. */
|
/** Set when the turn failed, so a stored session can show what went wrong. */
|
||||||
error?: string;
|
error?: string;
|
||||||
toolCalls: ToolCallRecord[];
|
toolCalls: ToolCallRecord[];
|
||||||
|
/** Images the user attached, kept for the transcript. The bytes are not stored. */
|
||||||
|
attachments: Attachment[];
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface Session {
|
export interface Session {
|
||||||
@@ -104,4 +151,6 @@ export interface ThreadMessage {
|
|||||||
error?: string;
|
error?: string;
|
||||||
/** Activity line shown while this message is streaming, e.g. "Searching the web". */
|
/** Activity line shown while this message is streaming, e.g. "Searching the web". */
|
||||||
status?: string;
|
status?: string;
|
||||||
|
/** Images attached to this question. Only ever non-empty on a user message. */
|
||||||
|
attachments: Attachment[];
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,75 @@
|
|||||||
|
import { LocalStorage } from "@vicinae/api";
|
||||||
|
import { modelCapabilities, supportsVision } from "./ollama";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The cached answer to "can this model see images?".
|
||||||
|
*
|
||||||
|
* `POST /api/show` is a round trip, and the answer only changes when the user
|
||||||
|
* switches model. Without a cache the chat view would ask again on every mount
|
||||||
|
* and the snip action would flicker in and out of the action panel while the
|
||||||
|
* request was in flight.
|
||||||
|
*
|
||||||
|
* `"unknown"` is deliberately distinct from `"no"`. A failed request is not
|
||||||
|
* evidence that a model cannot see, so the caller decides how much to trust it
|
||||||
|
* and a failure is never written to the cache.
|
||||||
|
*/
|
||||||
|
|
||||||
|
/** One key per host and model, so switching model invalidates by construction. */
|
||||||
|
const VISION_CACHE_PREFIX = "lam.vision.";
|
||||||
|
|
||||||
|
export type VisionSupport = "yes" | "no" | "unknown";
|
||||||
|
|
||||||
|
export interface VisionCheck {
|
||||||
|
host: string;
|
||||||
|
apiKey: string;
|
||||||
|
model: string;
|
||||||
|
signal?: AbortSignal;
|
||||||
|
/** Ignores the cache. Used when the answer might have gone stale. */
|
||||||
|
refresh?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
function cacheKey(host: string, model: string): string {
|
||||||
|
return `${VISION_CACHE_PREFIX}${host}|${model}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function visionSupport(check: VisionCheck): Promise<VisionSupport> {
|
||||||
|
const key = cacheKey(check.host, check.model);
|
||||||
|
|
||||||
|
if (!check.refresh) {
|
||||||
|
const cached = await LocalStorage.getItem<boolean>(key).catch(() => undefined);
|
||||||
|
if (cached !== undefined) return cached ? "yes" : "no";
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const capable = supportsVision(
|
||||||
|
await modelCapabilities({
|
||||||
|
host: check.host,
|
||||||
|
apiKey: check.apiKey,
|
||||||
|
model: check.model,
|
||||||
|
signal: check.signal,
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
await LocalStorage.setItem(key, capable).catch(() => undefined);
|
||||||
|
return capable ? "yes" : "no";
|
||||||
|
} catch {
|
||||||
|
return "unknown";
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Drops every cached answer.
|
||||||
|
*
|
||||||
|
* A model can gain vision without its name changing (a re-pull, a different
|
||||||
|
* tag on the same account), so `Set Up Lam Agent` calls this rather than
|
||||||
|
* leaving the user pinned to a stale "no".
|
||||||
|
*/
|
||||||
|
export async function forgetVisionSupport(): Promise<void> {
|
||||||
|
const items = await LocalStorage.allItems().catch(() => ({}));
|
||||||
|
|
||||||
|
await Promise.all(
|
||||||
|
Object.keys(items)
|
||||||
|
.filter((key) => key.startsWith(VISION_CACHE_PREFIX))
|
||||||
|
.map((key) => LocalStorage.removeItem(key).catch(() => undefined)),
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -2,6 +2,7 @@ import { Action, ActionPanel, Icon, List, openExtensionPreferences } from "@vici
|
|||||||
import { useCallback, useEffect, useMemo, useState } from "react";
|
import { useCallback, useEffect, useMemo, useState } from "react";
|
||||||
import { describeResumeTimeout, getLamPreferences, needsSetup } from "./lib/preferences";
|
import { describeResumeTimeout, getLamPreferences, needsSetup } from "./lib/preferences";
|
||||||
import { listModels } from "./lib/ollama";
|
import { listModels } from "./lib/ollama";
|
||||||
|
import { forgetVisionSupport, visionSupport, type VisionSupport } from "./lib/vision";
|
||||||
import type { ModelInfo } from "./lib/types";
|
import type { ModelInfo } from "./lib/types";
|
||||||
import { errorMessage } from "./lib/utils";
|
import { errorMessage } from "./lib/utils";
|
||||||
|
|
||||||
@@ -36,6 +37,7 @@ export default function SetupCommand() {
|
|||||||
const [error, setError] = useState<string | undefined>(undefined);
|
const [error, setError] = useState<string | undefined>(undefined);
|
||||||
const [isLoading, setIsLoading] = useState(false);
|
const [isLoading, setIsLoading] = useState(false);
|
||||||
const [hasTested, setHasTested] = useState(false);
|
const [hasTested, setHasTested] = useState(false);
|
||||||
|
const [vision, setVision] = useState<VisionSupport>("unknown");
|
||||||
|
|
||||||
const missingKey = needsSetup(preferences);
|
const missingKey = needsSetup(preferences);
|
||||||
|
|
||||||
@@ -53,6 +55,19 @@ export default function SetupCommand() {
|
|||||||
setHasTested(true);
|
setHasTested(true);
|
||||||
setIsLoading(false);
|
setIsLoading(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A model can gain or lose image support without its name changing (a
|
||||||
|
// re-pull, a retagged build), so re-testing drops the cached answer that the
|
||||||
|
// chat view gates the snip action on rather than leaving a stale "no".
|
||||||
|
await forgetVisionSupport();
|
||||||
|
setVision(
|
||||||
|
await visionSupport({
|
||||||
|
host: preferences.host,
|
||||||
|
apiKey: preferences.apiKey,
|
||||||
|
model: preferences.model,
|
||||||
|
refresh: true,
|
||||||
|
}),
|
||||||
|
);
|
||||||
}, [preferences]);
|
}, [preferences]);
|
||||||
|
|
||||||
// Verifying on open means the user sees something real immediately rather
|
// Verifying on open means the user sees something real immediately rather
|
||||||
@@ -149,6 +164,18 @@ export default function SetupCommand() {
|
|||||||
subtitle={describeResumeTimeout(preferences.resumeTimeoutMs)}
|
subtitle={describeResumeTimeout(preferences.resumeTimeoutMs)}
|
||||||
actions={settingsActions}
|
actions={settingsActions}
|
||||||
/>
|
/>
|
||||||
|
<List.Item
|
||||||
|
icon={vision === "yes" ? Icon.Image : Icon.QuestionMarkCircle}
|
||||||
|
title="Image Input"
|
||||||
|
subtitle={
|
||||||
|
vision === "yes"
|
||||||
|
? `Supported by ${preferences.model}. Snip commands are available.`
|
||||||
|
: vision === "no"
|
||||||
|
? `${preferences.model} cannot see images. Pick a vision model to snip.`
|
||||||
|
: "Not checked yet"
|
||||||
|
}
|
||||||
|
actions={settingsActions}
|
||||||
|
/>
|
||||||
</List.Section>
|
</List.Section>
|
||||||
|
|
||||||
{error ? (
|
{error ? (
|
||||||
|
|||||||
@@ -0,0 +1,91 @@
|
|||||||
|
import { LaunchType, closeMainWindow, launchCommand, showHUD, type LaunchProps } from "@vicinae/api";
|
||||||
|
import { captureRegion } from "./lib/capture";
|
||||||
|
import { getLamPreferences } from "./lib/preferences";
|
||||||
|
import { describeCaptureFailure, readSnipContext, snipDirectory } from "./lib/snip";
|
||||||
|
import { visionSupport } from "./lib/vision";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Vicinae's `closeMainWindow` defers the actual hide to a 50ms timer
|
||||||
|
* (`closeWindow(settings, 50ms)` in `ui-service.hpp`), so a capture started the
|
||||||
|
* instant the call resolves can still find the window on screen and select a
|
||||||
|
* region around it. Waiting a little past that timer is the difference between
|
||||||
|
* "the window is closed" and "the window is usually closed".
|
||||||
|
*/
|
||||||
|
const WINDOW_SETTLE_MS = 100;
|
||||||
|
|
||||||
|
function settleWindow(): Promise<void> {
|
||||||
|
return new Promise((resolve) => setTimeout(resolve, WINDOW_SETTLE_MS));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Entrypoint for the "snip" command.
|
||||||
|
*
|
||||||
|
* `no-view`, and it has to be. Vicinae calls `showWindow()` when it activates a
|
||||||
|
* command from a closed launcher, so the window is up by the time any extension
|
||||||
|
* code runs; capturing from a view means selecting a region around a panel
|
||||||
|
* sitting in the middle of the screen. This command closes the window first,
|
||||||
|
* captures with nothing in the way, and only then hands the image to the chat,
|
||||||
|
* which is the command that brings the window back.
|
||||||
|
*
|
||||||
|
* It is also the *only* capture path. The chat's own action launches this command
|
||||||
|
* rather than capturing inline, so there is one implementation, one capability
|
||||||
|
* check, and one place where the window is dealt with.
|
||||||
|
*
|
||||||
|
* Bind it to a global shortcut for one-key "snip and ask".
|
||||||
|
*/
|
||||||
|
export default async function SnipCommand(props: LaunchProps) {
|
||||||
|
const preferences = getLamPreferences();
|
||||||
|
const context = readSnipContext(props.launchContext);
|
||||||
|
|
||||||
|
// Before anything else. Everything below is pointless if the selector is
|
||||||
|
// competing with a window for the middle of the screen.
|
||||||
|
await closeMainWindow();
|
||||||
|
await settleWindow();
|
||||||
|
|
||||||
|
// Settled before the capture, because taking a screenshot for a model that
|
||||||
|
// cannot read it throws away the selection gesture the user just performed.
|
||||||
|
// "unknown" says so in those words: a dropped check is not a verdict on the
|
||||||
|
// model, and reporting it as one would send the user shopping for another.
|
||||||
|
const vision = await visionSupport({
|
||||||
|
host: preferences.host,
|
||||||
|
apiKey: preferences.apiKey,
|
||||||
|
model: preferences.model,
|
||||||
|
});
|
||||||
|
|
||||||
|
if (vision !== "yes") {
|
||||||
|
await showHUD(
|
||||||
|
vision === "no"
|
||||||
|
? `${preferences.model} cannot see images. Pick a vision model in preferences.`
|
||||||
|
: `Could not check whether ${preferences.model} can see images. Run Set Up Lam Agent.`,
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
let snipPath: string | undefined;
|
||||||
|
let snipError: string | undefined;
|
||||||
|
|
||||||
|
try {
|
||||||
|
const capture = await captureRegion({ outputDir: snipDirectory() });
|
||||||
|
snipPath = capture?.path;
|
||||||
|
} catch (thrown) {
|
||||||
|
snipError = describeCaptureFailure(thrown);
|
||||||
|
}
|
||||||
|
|
||||||
|
// An abandoned selection says nothing at all: the user pressed Escape. But a
|
||||||
|
// snip started from an open conversation has to come back, because the window
|
||||||
|
// is now closed and the hotkey is otherwise the only way back.
|
||||||
|
if (!snipPath && !snipError && !context.sessionId) return;
|
||||||
|
|
||||||
|
// Only the fields that exist are sent. `launchCommand` types its context as a
|
||||||
|
// string map, and a key left undefined carries no information anyway.
|
||||||
|
const payload: Record<string, string> = {};
|
||||||
|
if (context.sessionId) payload.sessionId = context.sessionId;
|
||||||
|
if (snipPath) payload.snipPath = snipPath;
|
||||||
|
if (snipError) payload.snipError = snipError;
|
||||||
|
|
||||||
|
await launchCommand({
|
||||||
|
name: "chat",
|
||||||
|
type: LaunchType.UserInitiated,
|
||||||
|
context: payload,
|
||||||
|
});
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user