feat: snip a region into chat, and make links clickable #14
@@ -5,7 +5,7 @@ Agentic AI chat inside the [Vicinae](https://vicinae.com) launcher, powered by [
|
||||
Hit a global shortcut, type a question, get a streamed answer without leaving whatever you were doing.
|
||||
|
||||
<!-- prettier-ignore -->
|
||||
> **Status:** early. Chat, streaming, persistent sessions and the web tools work. Anything that *acts* on your machine (opening links, clipboard, launching apps) is deliberately not here yet. See [Roadmap](#roadmap).
|
||||
> **Status:** early. Chat, streaming, persistent sessions, the web tools and screen capture work. Anything else that *acts* on your machine (opening links, clipboard, launching apps) is deliberately not here yet. See [Roadmap](#roadmap).
|
||||
|
||||
---
|
||||
|
||||
@@ -15,16 +15,35 @@ Hit a global shortcut, type a question, get a streamed answer without leaving wh
|
||||
- **Streaming answers.** Tokens render as they arrive, in a markdown side panel.
|
||||
- **Persistent conversations.** Every chat is saved locally. The sidebar lists them by recency, and reopening the command picks up where you left off, unless it has been idle longer than you allow.
|
||||
- **Readable code.** Fenced blocks render in a dark panel with the language named in a header, syntax colouring, and a Copy button.
|
||||
- **Tools.** Lam decides when to call them: current time, location context, web search, and page fetching.
|
||||
- **Chat only.** Lam answers questions. It does not touch your files or run commands.
|
||||
- **Real markdown.** Tables, bulleted and numbered lists, headings, blockquotes, bold, italics and links all render natively, and the default prompt asks for the shape that fits the answer rather than flattening everything into prose.
|
||||
- **Tools.** Lam decides when to call them: current time, location context, web search, and page fetching. A turn that uses up its tool budget still ends in an answer, written from what it gathered, rather than a notice that it stopped.
|
||||
- **Screenshots, two ways.** Snip a region into the conversation you are already in from the action panel, or run **Snip to Lam** to start a new chat about it. The launcher window gets out of the way first, so it never ends up in the shot, and the capture appears in the transcript as a thumbnail with Open Screenshot for the full image. Neither is offered unless the current model can actually see images.
|
||||
- **Chat only.** Lam answers questions. It does not touch your files or run commands. The one exception is capture, and it only ever happens because you pressed the key for it.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Vicinae with the extension runtime enabled (Node.js 20+ on `PATH`)
|
||||
- An [Ollama](https://ollama.com) account and API key, which you can [create here](https://ollama.com/settings/keys)
|
||||
- A screenshot tool, if you want to snip. A vision model, if you want Lam to read the result
|
||||
|
||||
Ollama Cloud is the default. A local Ollama server works too: point **Ollama Host** at `http://localhost:11434`. Note that the web tools are a cloud feature and are not available from a local server.
|
||||
|
||||
### Snipping needs a capture tool
|
||||
|
||||
Vicinae has no screenshot API, so the extension shells out. It auto-detects the first of these that is installed, in this order:
|
||||
|
||||
| Platform | Tools, in preference order |
|
||||
| --- | --- |
|
||||
| Linux (Wayland) | `grimblast`, then `grim` + `slurp`, then `spectacle` |
|
||||
| Linux (X11) | `maim` + `slop`, then `scrot`, then `flameshot` |
|
||||
| macOS | `screencapture`, which ships with the system |
|
||||
|
||||
If none is found, the chat says so and names what to install. Nothing to configure: the detection happens per platform at capture time.
|
||||
|
||||
**On macOS, Vicinae itself needs Screen Recording permission.** macOS attributes a capture request to the app responsible for launching the process that made it, which is Vicinae rather than the extension, so the grant goes on Vicinae: **System Settings, Privacy & Security, Screen & System Audio Recording**, then quit and reopen Vicinae (a TCC change does not apply to a running app).
|
||||
|
||||
Without it, `screencapture` prints `could not create image from rect` and writes nothing. That is awkward, because a cancelled selection also writes nothing, and `screencapture -i` has been reported to exit `0` when the permission is missing. So the file alone cannot tell the two apart. `capture.ts` therefore reads stderr as well: that specific message becomes a permission error naming the setting to change, and only silence with no image is treated as a cancel.
|
||||
|
||||
## Install
|
||||
|
||||
```sh
|
||||
@@ -57,6 +76,7 @@ To change the model later, either edit the **Model** preference directly, or run
|
||||
| `↩` | Send the message in the search bar |
|
||||
| `↩` with an empty bar | Open the highlighted conversation |
|
||||
| `⌘ N` | New chat |
|
||||
| `⌘ ⇧ S` | Snip a region and attach it to this chat (only with a vision model) |
|
||||
| `⌘ ⇧ ⌫` | Delete the highlighted conversation (also in **Chat Sessions**) |
|
||||
|
||||
Prompts are single line. An earlier `⌘ ↩` multi-line composer was removed. See [Architecture](#architecture) for why a `Form.TextArea` cannot be laid out cleanly in a launcher. The `Tab` shortcut went with it: the left column *is* the conversation switcher now, so an accessory dropdown would only duplicate it.
|
||||
@@ -67,14 +87,17 @@ Assign these in **Vicinae Settings → Extensions → Lam Agent**, or bind them
|
||||
|
||||
```sh
|
||||
vicinae 'vicinae://launch/lam-agent/chat?toggle=true'
|
||||
vicinae 'vicinae://launch/lam-agent/snip'
|
||||
```
|
||||
|
||||
```ini
|
||||
# Hyprland
|
||||
bind = $mainAlt, Space, exec, vicinae 'vicinae://launch/lam-agent/chat?toggle=true'
|
||||
bind = $mainAlt, S, exec, vicinae 'vicinae://launch/lam-agent/snip'
|
||||
|
||||
# niri
|
||||
Mod+Shift+Space repeat=false { spawn "vicinae" "vicinae://launch/lam-agent/chat?toggle=true"; }
|
||||
Mod+Shift+S repeat=false { spawn "vicinae" "vicinae://launch/lam-agent/snip"; }
|
||||
```
|
||||
|
||||
`fallbackText` pre-fills the prompt, which makes a handy "ask about this" command:
|
||||
@@ -88,8 +111,9 @@ vicinae 'vicinae://launch/lam-agent/chat?fallbackText=summarise%20this'
|
||||
| Command | Purpose |
|
||||
| --- | --- |
|
||||
| **Chat with Lam** | The chat surface. Bind this to your global shortcut. |
|
||||
| **Snip to Lam** | Capture a region and start a new chat about it. A `no-view` command, so the launcher window closes before the selector appears. Bind this to a second shortcut for one-key "snip and ask". |
|
||||
| **Chat Sessions** | Browse, reopen, and delete past conversations. |
|
||||
| **Set Up Lam Agent** | Verify the connection and list available models. |
|
||||
| **Set Up Lam Agent** | Verify the connection, list available models, and check whether the selected model can see images. |
|
||||
|
||||
## Configuration
|
||||
|
||||
@@ -98,6 +122,8 @@ vicinae 'vicinae://launch/lam-agent/chat?fallbackText=summarise%20this'
|
||||
| **Ollama API Key** | None | Required. Stored by Vicinae as a password preference. |
|
||||
| **Model** | None | Required. Any model your key can reach. |
|
||||
| **Ollama Host** | `https://ollama.com` | Set to `http://localhost:11434` for a local server. |
|
||||
| **Location** | None | The place Lam should treat as yours, for example `Cebu City, Philippines`. Optional. The only exact location available, and the only one that stops anything being inferred. |
|
||||
| **IP Geolocation** | off | Off by default. When on, and only when **Location** is empty, your IP is sent to ipwho.is to resolve a rough area. |
|
||||
| **System Prompt** | Lam's default | Replaces the built-in prompt entirely. The default asks for plain prose: no em dashes, no generated-text tells, no restating the question. |
|
||||
| **Web Tools** | on | Enables `web_search` and `web_fetch`. |
|
||||
| **Reasoning Effort** | Off | `low`/`medium`/`high` for models that support thinking. |
|
||||
@@ -107,37 +133,60 @@ vicinae 'vicinae://launch/lam-agent/chat?fallbackText=summarise%20this'
|
||||
|
||||
## Tools
|
||||
|
||||
Lam calls these on its own when they would improve an answer. All four are read-only.
|
||||
Lam calls these on its own when they would improve an answer. All four are read-only, and all four are the entire list of things Lam is allowed to do without being asked twice. Screen capture is deliberately not among them.
|
||||
|
||||
| Tool | What it does |
|
||||
| --- | --- |
|
||||
| `get_datetime` | Current local date, time, weekday, timezone, and UTC offset. |
|
||||
| `get_location` | Timezone, UTC offset, locale, region, and city. |
|
||||
| `get_location` | The place you configured, or a rough city from your IP if you enabled that, plus timezone, UTC offset, locale and region. |
|
||||
| `web_search` | Searches the web; returns titles, URLs, and snippets. |
|
||||
| `web_fetch` | Fetches one page and returns its content as markdown. |
|
||||
|
||||
`get_location` is derived from your device timezone rather than GPS or an IP lookup, so no third-party geolocation service sees your address. Treat it as "which region am I in", not "where am I".
|
||||
`get_location` reports the **Location** you set, if you set one, because that is the only answer here that is actually about where you are. Everything else is inferred from your device, and the inference is coarser than it looks.
|
||||
|
||||
Web search results and fetched pages can be large. Lam truncates them before they reach the model, and gives up after 5 rounds of tool calls rather than looping.
|
||||
**A timezone is not a city.** This is the trap the tool used to fall into: it reported `City: Manila`, taken from the last segment of the IANA zone name. That name is the database's representative place, not yours, and a single zone is often a whole country. `Asia/Manila` is the Philippines, `Asia/Kolkata` is India, `Europe/Oslo` is Norway, `Australia/Brisbane` is Queensland. Reporting the zone's namesake as your city was wrong for anyone outside it, and confidently so, because a model asked about the weather will use it without hesitating. The zone's label is still reported, as `Timezone area`, and named for what it is.
|
||||
|
||||
The **region** comes from your locale, which is a declaration rather than an inference: a locale of `en-PH` says the Philippines because that is what you set. It costs nothing and it cannot be wrong the way a zone name can.
|
||||
|
||||
If you turn on **IP Geolocation**, and only then, `get_location` resolves a rough area from your IP address via [ipwho.is](https://ipwho.is). That is a third party learning approximately where you are, so it is off by default, it is skipped entirely whenever you have set a Location (the better answer is already there, and there is no reason to spend your privacy on a worse one), and the tool output names the provider and calls the result a rough guess rather than a city.
|
||||
|
||||
**It is coarser than "city-level" suggests, and that is worth knowing before trusting it.** An IP address identifies the point where the traffic left the ISP, not the person. Measured from one machine on Globe Telecom, an IPv4 lookup resolved to Cebu City and an IPv6 lookup moments later resolved to Talisay in Calabarzon, roughly 500km apart. If you want a place you can rely on, set **Location** and turn this off.
|
||||
|
||||
The answer is cached for 6 hours rather than looked up per question, because an area does not change between questions and every lookup hands over another copy of your address. `Set Up Lam Agent` clears that cache, and the capability cache, when you re-test.
|
||||
|
||||
This extension does not use GPS. That would mean CoreLocation on macOS, which has no command-line interface, or GeoClue on Linux, whose usual provider resolves nearby Wi-Fi networks through a Mozilla service anyway. Neither is worth it for "is it late where you are".
|
||||
|
||||
**The provider has to be one that answers an unauthenticated caller.** The first choice was ipapi.co, which no longer does: a request with no User-Agent now returns `HTTP 429` with `"Please sign up for a paid plan"`, and adding a browser User-Agent gets an `HTTP 403` Cloudflare bot challenge instead. ipwho.is answers without a key and signals failures with `success: false` on an `HTTP 200`, so the body is checked as well as the status. `GEO_PROVIDER` is a single constant, and swapping it means changing the parser beside it.
|
||||
|
||||
Web search results and fetched pages can be large. Lam truncates them before they reach the model.
|
||||
|
||||
**A turn that runs out of tool calls still ends in an answer.** The loop stops after 8 rounds, and the first version of it stopped by throwing away every preamble the model had written and returning a notice about the loop: asked for the fish in every Valheim biome, it researched for five rounds and then said nothing useful at all. The budget is now spent on a *closing round* with the tools taken away and an instruction to answer from what was gathered and to name what it could not confirm. So the worst case is a partial answer that says what is missing, rather than an app-generated message about its own control flow. If the closing round produces nothing either, the last preamble is used, and only then a notice. The closing round is why the cap is 8 rather than 5: the extra round is not more searching, it is the answer.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
src/
|
||||
chat.tsx # "chat" command entrypoint, thin wrapper
|
||||
snip.tsx # "snip" command entrypoint, no-view: captures then hands over
|
||||
sessions.tsx # browse, reopen, delete conversations
|
||||
setup.tsx # connection check + live model list
|
||||
lib/
|
||||
ollama.ts # the only file that talks HTTP
|
||||
agent.ts # the tool-calling loop
|
||||
tools.ts # tool schemas + dispatch
|
||||
capture.ts # screen capture backends, shelled out
|
||||
attachments.ts # image files -> base64 payloads
|
||||
vision.ts # cached "can this model see?" answer
|
||||
sessions.ts # LocalStorage persistence
|
||||
history.ts # turns -> API messages
|
||||
location.ts # time/place, derived locally
|
||||
thread.ts # turns -> flat message list
|
||||
markdown.ts # messages -> detail-panel markdown
|
||||
code.ts # code fences and links, repaired
|
||||
preferences.ts # preferences, normalised
|
||||
hooks/useChat.ts # the chat state machine
|
||||
hooks/useVision.ts # capability check, for gating the snip action
|
||||
snip.ts # the snip -> chat handoff contract
|
||||
components/ChatView.tsx # the chat surface
|
||||
```
|
||||
|
||||
@@ -157,16 +206,42 @@ A few decisions worth knowing about before you change things:
|
||||
|
||||
**The transcript reads newest message first.** The search bar sits at the top of the window, so the most recent message, an answer once one has been written, is the first line under it rather than below the fold. Within a single exchange that means the answer appears above the question it responds to. `buildThread` produces that order, and `buildThreadMarkdown` regroups messages back into exchanges so the horizontal rule separates one exchange from the next instead of dividing every message.
|
||||
|
||||
**The transcript is text-only, and nothing in it is bold except an error.** The answer is what gets read, so it is plain body text carrying no label; the question is a blockquote, which the renderer draws dimmed, italic, and with a left rule. A `**You**` / `**Lam**` header on each exchange gave both speakers identical weight and made the answer sit behind chrome. Tool calls collapse to one italic line (_Tools: `web_search`_) rather than a bold header plus a list, so they stay below the answer too. Raycast solves the same problem with alignment, but markdown offers no alignment control.
|
||||
**The transcript is text plus the attached images, and nothing in it is bold except an error.** The answer is what gets read, so it is plain body text carrying no label; the question is a blockquote, which the renderer draws dimmed, italic, and with a left rule. A `**You**` / `**Lam**` header on each exchange gave both speakers identical weight and made the answer sit behind chrome. Tool calls collapse to one italic line (_Tools: `web_search`_) rather than a bold header plus a list, so they stay below the answer too. Raycast solves the same problem with alignment, but markdown offers no alignment control.
|
||||
|
||||
**An attached screenshot is rendered, not described.** This is a correction: the transcript used to say `_Screenshot attached_` and leave it at that, on the belief that Vicinae could not resolve a local image. It can. `imageProviderUrl` in `markdown-model.cpp` routes `file://` (and a scheme-less string) through `ImageURL::local`, and `MdImage.qml` draws it block-level and centred with the alt text as an italic caption underneath.
|
||||
|
||||
**The preview is asked for as an exact box, which is why the PNG header is parsed.** `MdImage.qml` sizes its container to the *request* rather than to the image: with no hint it is the full pane width and 200px tall, and the image is aspect-fitted inside that, so a wide capture sits in a letterbox with dead space above and below. Measured on real captures here, a 4:1 crop wasted about 106px of a 200px container and a 6.7:1 crop wasted about 140px. `previewBox` therefore computes a box from the image's real dimensions, scales it to fit 400x200, and emits it as the `raycast-width` and `raycast-height` hints the renderer reads, so the container matches the image and nothing is wasted. The scale is capped at 1 so a small capture is not blown up.
|
||||
|
||||
The rendered height stays capped at 200px whatever is asked for (`Math.min(_rawH, 200, maxImageHeight)`), so this is a thumbnail rather than something readable, which is why the action panel still offers Open Screenshot for the full image. The alt text is short for the same reason it is not empty: whatever is there is drawn centred under the image, so a sentence reads as a stray line of prose, and it is the only thing left if the capture file has since been deleted. A preview of an unsent attachment is shown the same way, so a snip is visible before the question about it exists.
|
||||
|
||||
**Code fences are labelled so Vicinae can colour them.** `MdCodeBlock.qml` already provides the dark panel, a header naming the language, and a Copy button, so none of that is built here. The language string is what turns on highlighting: `syntax::highlight` resolves it with `definitionForName(language)` and then `definitionForFileName("file." + language)`. So `bash` and `ts` colour correctly, while an unrecognised string such as `terminal` still renders a label but produces no highlighting at all, so a wrong label is worse than none. `code.ts` therefore labels bare fences conservatively, only when a signal is clear, and the default prompt asks for a language on every fence.
|
||||
|
||||
**A URL is only clickable as a markdown link, so `code.ts` takes them back out of code.** Vicinae's parser attaches the `table` and `strikethrough` GFM extensions and nothing else, so the `autolink` extension is off: a bare URL is plain dead text. Clickability comes from `MarkdownModel` turning `CMARK_NODE_LINK` into an `<a href>`, which the selection controller opens on release. A URL the model fenced is therefore the worst possible presentation, a dark panel with a Copy button and no way to open the address. `restoreUrlLinks` recovers three narrow shapes (a fence whose entire body is URLs, a lone `` `https://...` `` span, and a line that is only a URL) into `[url](url)` links, and the default prompt now says never to fence a URL in the first place. Anything that is not one of those three shapes is copied through, so a `curl` invocation containing an address is never rewritten.
|
||||
|
||||
**A table needs a blank line in front of it, and the prompt alone does not guarantee one.** Vicinae renders tables properly (`MdTable.qml`, block type 5), but GFM tables cannot interrupt a paragraph, so a table written directly under a line of prose is not a table at all: it renders as a paragraph of pipes and dashes. The blank line looks unnecessary, so models leave it out. `separateTables` adds it, and only in front of a genuine table start, which is a row followed by a delimiter row. Prose containing a pipe is left alone, a table already at a block boundary is left alone, and a fenced example is never touched.
|
||||
|
||||
**The prompt used to forbid tables, which is worth not repeating.** It asked for brevity in a narrow window and then said "skip large headings and wide tables", and a model follows that instruction by describing the table it was about to write instead of writing it: a bulleted list of rows, with the column names in inline code. Brevity is about length, not about which markdown is allowed, so the guidance now names what renders and asks for the shape that fits, with a cap of about four narrow columns because the detail pane really is narrow.
|
||||
|
||||
**Streamed deltas are buffered in refs and flushed to state on a timer** (`FLUSH_INTERVAL_MS`). Re-rendering the transcript per token makes the launcher visibly janky.
|
||||
|
||||
**The logic layer has no Vicinae imports.** `agent.ts`, `ollama.ts`, `tools.ts` and friends are plain TypeScript, which is why they unit test without a runtime. A test enforces this.
|
||||
|
||||
**Sessions are capped twice over**, once per turn and once per session, because `LocalStorage` has no documented ceiling.
|
||||
|
||||
**Screen capture is owned, not borrowed.** There is no screenshot call anywhere in `@vicinae/api`, and `Clipboard` moves text, html, urls and a file path and nothing binary, so a bitmap can never be read back out of the clipboard either. The only way to get a region of the screen is to shell out, which is fair game because extensions run under Node. The useful consequence is that the extension picks the output path and reads the file it wrote, rather than guessing what some other application left on the clipboard.
|
||||
|
||||
Deciding whether a capture *worked* takes two signals, not one. The image landing on disk is what makes it a success, because the backends disagree about exit codes and at least one reports success on a cancelled selection. But the file cannot distinguish a cancel from a refused permission, since both leave nothing behind and macOS can exit `0` on the refusal, so the refusal is recognised by its stderr message instead. Silence with no image is the user pressing Escape; a complaint with no image is a failure the user is told about. Snips go in the extension's support directory rather than a temp directory, because a stored conversation keeps the path and a path that survives a reboot is the difference between an old chat showing its screenshot and pointing at nothing.
|
||||
|
||||
**Capture runs in a `no-view` command, because the chat view owns a window.** This is the part that is easy to get wrong. `activateEntrypoint` calls `showWindow()` when it activates a command from a closed launcher, so the window is already on screen before any extension code runs, and a region selector started from a view is dragging a rectangle around a panel parked in the middle of the screen. So `snip` is declared `no-view`, calls `closeMainWindow()` as its first statement, captures with nothing in the way, and only then brings the window back by launching `chat` with the file. That command is also the single capture path: the chat's own action launches `snip` rather than capturing inline, so there is one implementation and one place that deals with the window.
|
||||
|
||||
The hand-off goes through `snip` specifically rather than straight to `chat` on purpose. `activateEntrypoint` treats a launch of the command that is *already active* as a toggle and closes the window instead of reopening it (`toggleIfAlreadyActive` defaults to true, and a launch cannot override it). Relaying through a different, no-view command means never asking for `chat` while `chat` is active, so the toggle branch is unreachable. The chat's own snip action is also hidden while a turn is streaming, because the hand-off unloads the view and would abandon an answer mid-sentence.
|
||||
|
||||
**Image support is a fact from the API, not a guess from the model name, and it is cached.** `GET /api/tags` returns names, sizes and quantization levels and says nothing about capability, so the check is a `POST /api/show` that reads the `vision` flag. That is a round trip per model, so the answer is persisted under a key built from the host and model name, which means switching model invalidates it by construction. The state is a tri-state rather than a boolean: a failed request is `"unknown"`, never `"no"`, because telling someone their model cannot see when the network merely dropped is a false claim about their setup that sends them shopping for a model they do not need. The snip command settles this *before* the selector appears, so a model that cannot read images never costs the user a selection gesture, and the two outcomes get different HUD wording. The in-chat action still fails closed: only a confirmed `"yes"` offers it.
|
||||
|
||||
**The capture is never a tool.** It stays in the action panel and on a keybinding, so it only ever runs because a person asked for it. A model that could decide to screenshot the screen is a different risk profile from one that answers questions, which is the same line the read-only tools already sit behind.
|
||||
|
||||
**An attached image is sent once and never replayed.** It is the largest thing a request can carry, so `buildMessages` attaches it to the question it belongs to and drops it afterwards, exactly as it already drops tool output. The cost is that a follow-up about a screenshot needs the image attached again; the alternative is re-sending every screenshot in the conversation on every turn. Only the path and mime type go into storage: `MAX_STORED_CHARS_PER_TURN` exists so one turn cannot dominate `LocalStorage`, and base64 would blow past it on a single 1080p capture.
|
||||
|
||||
## Development
|
||||
|
||||
```sh
|
||||
@@ -183,7 +258,8 @@ npm test
|
||||
- **Actions.** Opening links, clipboard access, launching apps. These need an approval step first. An assistant that can silently act on your machine is a different risk profile from one that answers questions, and this release does not pretend otherwise.
|
||||
- **`@` mentions** for files and clipboard content.
|
||||
- **Per-conversation model** overrides.
|
||||
- **Image input**, which Ollama's chat API already supports.
|
||||
- **Annotations on a snip.** The capture backends are there, but drawing on the result before sending it needs a second tool and a wait for the editor to save.
|
||||
- **Follow-ups that keep the image.** Replaying an attachment is a token cost that has to be made opt-in first.
|
||||
|
||||
## License
|
||||
|
||||
|
||||
@@ -15,6 +15,13 @@
|
||||
"description": "Ask Lam anything. Answers stream in as they are generated, your session history is kept, and Lam can search the web when it needs current information.",
|
||||
"mode": "view"
|
||||
},
|
||||
{
|
||||
"name": "snip",
|
||||
"title": "Snip to Lam",
|
||||
"subtitle": "Lam Agent",
|
||||
"description": "Capture a region of the screen and start a new chat about it. The launcher window closes first so it cannot end up in the shot. Offered only when the model in preferences supports image input.",
|
||||
"mode": "no-view"
|
||||
},
|
||||
{
|
||||
"name": "sessions",
|
||||
"title": "Chat Sessions",
|
||||
@@ -58,6 +65,22 @@
|
||||
"type": "textfield",
|
||||
"required": false
|
||||
},
|
||||
{
|
||||
"name": "location",
|
||||
"title": "Location",
|
||||
"description": "The place Lam should treat as yours, for example Cebu City, Philippines. Optional. Without it Lam falls back to your device timezone and locale, which are accurate to a region and not to a city, so anything that depends on exactly where you are (weather, opening hours) is worth filling in.",
|
||||
"type": "textfield",
|
||||
"required": false
|
||||
},
|
||||
{
|
||||
"name": "ipGeolocation",
|
||||
"title": "IP Geolocation",
|
||||
"label": "Look up my city from my IP address",
|
||||
"description": "Off by default. When this is on, and the Location preference above is empty, Lam sends your IP address to ipwho.is to work out roughly where you are. That is a third party learning approximately where you are, which is why it is opt-in, and why setting Location makes it unnecessary. It is coarser than a city: an IP address locates the point where traffic left your ISP, which can be hundreds of kilometres away.",
|
||||
"type": "checkbox",
|
||||
"required": false,
|
||||
"default": false
|
||||
},
|
||||
{
|
||||
"name": "webSearchEnabled",
|
||||
"title": "Web Tools",
|
||||
|
||||
@@ -46,3 +46,26 @@ export function __setPreferenceValues(values: Record<string, unknown>): void {
|
||||
export function getPreferenceValues<T = Record<string, unknown>>(): T {
|
||||
return preferenceOverride as T;
|
||||
}
|
||||
|
||||
/**
|
||||
* The runtime `environment`. Only `supportPath` is exercised, by the snip
|
||||
* directory, and a stable fake path keeps those assertions independent of where
|
||||
* the test happens to run.
|
||||
*/
|
||||
export const environment = {
|
||||
supportPath: "/tmp/lam-agent-support",
|
||||
assetsPath: "/tmp/lam-agent-assets",
|
||||
extensionName: "lam-agent",
|
||||
ownerOrAuthorName: "adrianbonpin",
|
||||
commandName: "chat",
|
||||
commandMode: "view" as const,
|
||||
isDevelopment: true,
|
||||
appearance: "dark" as const,
|
||||
theme: "dark" as const,
|
||||
textSize: "medium" as const,
|
||||
launchType: "userInitiated" as const,
|
||||
raycastVersion: "0.0.0",
|
||||
vicinaeVersion: { tag: "test", commit: "test" },
|
||||
isRaycast: false,
|
||||
canAccess: () => true,
|
||||
};
|
||||
|
||||
+103
-5
@@ -2,6 +2,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import path from "node:path";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { pickToolCalls, runAgent } from "../lib/agent";
|
||||
import { MAX_TOOL_ITERATIONS } from "../lib/constants";
|
||||
import { OllamaError } from "../lib/ollama";
|
||||
import { toolsFor } from "../lib/tools";
|
||||
import type { ApiMessage, ToolCall } from "../lib/types";
|
||||
@@ -136,15 +137,102 @@ describe("runAgent", () => {
|
||||
|
||||
it("stops after the iteration cap instead of looping forever", async () => {
|
||||
// Every response asks for another tool call, so only the cap can end this.
|
||||
for (let i = 0; i < 10; i++) {
|
||||
for (let i = 0; i < MAX_TOOL_ITERATIONS + 2; i++) {
|
||||
fetchMock.mockResolvedValueOnce(ndjsonResponse([toolCall("get_datetime")]));
|
||||
}
|
||||
|
||||
const result = await runAgent(baseOptions([{ role: "user", content: "hi" }]));
|
||||
|
||||
expect(result.content).toContain("Stopped after");
|
||||
expect(result.toolCalls).toHaveLength(5);
|
||||
expect(fetchMock).toHaveBeenCalledTimes(5);
|
||||
expect(result.toolCalls).toHaveLength(MAX_TOOL_ITERATIONS);
|
||||
// Every tool round, plus one closing round with the tools taken away.
|
||||
expect(fetchMock).toHaveBeenCalledTimes(MAX_TOOL_ITERATIONS + 1);
|
||||
// The closing round produced no text either, so the notice is the last resort.
|
||||
expect(result.content).toContain("Ran out of tool calls");
|
||||
});
|
||||
|
||||
it("gives the model a closing round so a spent budget still ends in an answer", async () => {
|
||||
// The reported failure: a broad research question used up the budget and the
|
||||
// turn ended with a notice about the loop instead of an answer.
|
||||
for (let i = 0; i < MAX_TOOL_ITERATIONS; i++) {
|
||||
fetchMock.mockResolvedValueOnce(ndjsonResponse([toolCall("web_search")]));
|
||||
}
|
||||
fetchMock.mockResolvedValueOnce(
|
||||
ndjsonResponse(answer("Meadows have Perch, at 2 fish each. I could not confirm the others.")),
|
||||
);
|
||||
|
||||
const result = await runAgent(baseOptions([{ role: "user", content: "what fish are in valheim" }]));
|
||||
|
||||
expect(result.content).toContain("Meadows have Perch");
|
||||
expect(result.content).not.toContain("Ran out of tool calls");
|
||||
});
|
||||
|
||||
it("takes the tools away for the closing round, so it writes rather than searches again", async () => {
|
||||
for (let i = 0; i < MAX_TOOL_ITERATIONS; i++) {
|
||||
fetchMock.mockResolvedValueOnce(ndjsonResponse([toolCall("web_search")]));
|
||||
}
|
||||
fetchMock.mockResolvedValueOnce(ndjsonResponse(answer("Here is what I found.")));
|
||||
|
||||
await runAgent(baseOptions([{ role: "user", content: "hi" }]));
|
||||
|
||||
const first = JSON.parse(fetchMock.mock.calls[0][1].body as string) as { tools?: unknown };
|
||||
const last = JSON.parse(fetchMock.mock.calls[MAX_TOOL_ITERATIONS][1].body as string) as {
|
||||
tools?: unknown;
|
||||
};
|
||||
|
||||
expect(first.tools).toBeDefined();
|
||||
expect(last.tools).toBeUndefined();
|
||||
});
|
||||
|
||||
it("asks the closing round to name what it could not confirm", async () => {
|
||||
for (let i = 0; i < MAX_TOOL_ITERATIONS; i++) {
|
||||
fetchMock.mockResolvedValueOnce(ndjsonResponse([toolCall("web_search")]));
|
||||
}
|
||||
fetchMock.mockResolvedValueOnce(ndjsonResponse(answer("ok")));
|
||||
|
||||
await runAgent(baseOptions([{ role: "user", content: "hi" }]));
|
||||
|
||||
const last = JSON.parse(fetchMock.mock.calls[MAX_TOOL_ITERATIONS][1].body as string) as {
|
||||
messages: ApiMessage[];
|
||||
};
|
||||
const nudge = last.messages[last.messages.length - 1];
|
||||
|
||||
expect(nudge.role).toBe("user");
|
||||
expect(nudge.content).toContain("no more tools are available");
|
||||
expect(nudge.content).toContain("could not confirm");
|
||||
});
|
||||
|
||||
it("falls back to the last preamble when the closing round comes back empty", async () => {
|
||||
// A preamble is not normally the answer, but a turn that gathered things
|
||||
// should still say more than a notice about its own loop.
|
||||
const preamble = {
|
||||
message: {
|
||||
role: "assistant",
|
||||
content: "Let me look that up.",
|
||||
tool_calls: [{ function: { name: "web_search", arguments: {} } }],
|
||||
},
|
||||
done: true,
|
||||
};
|
||||
|
||||
for (let i = 0; i < MAX_TOOL_ITERATIONS; i++) {
|
||||
fetchMock.mockResolvedValueOnce(ndjsonResponse([preamble]));
|
||||
}
|
||||
fetchMock.mockResolvedValueOnce(ndjsonResponse([{ message: { role: "assistant", content: "" }, done: true }]));
|
||||
|
||||
const result = await runAgent(baseOptions([{ role: "user", content: "hi" }]));
|
||||
|
||||
expect(result.content).toBe("Let me look that up.");
|
||||
});
|
||||
|
||||
it("still reports the tools it used when the budget runs out", async () => {
|
||||
for (let i = 0; i < MAX_TOOL_ITERATIONS; i++) {
|
||||
fetchMock.mockResolvedValueOnce(ndjsonResponse([toolCall("get_datetime")]));
|
||||
}
|
||||
fetchMock.mockResolvedValueOnce(ndjsonResponse(answer("done")));
|
||||
|
||||
const result = await runAgent(baseOptions([{ role: "user", content: "hi" }]));
|
||||
|
||||
expect(result.toolCalls).toHaveLength(MAX_TOOL_ITERATIONS);
|
||||
expect(result.toolCalls.every((call) => call.name === "get_datetime")).toBe(true);
|
||||
});
|
||||
|
||||
it("surfaces an actionable hint when the API key is rejected", async () => {
|
||||
@@ -165,7 +253,17 @@ describe("module wiring", () => {
|
||||
// These modules must stay renderer-free: they are pure logic, they are what
|
||||
// the unit tests cover, and nothing in them should need a Vicinae runtime.
|
||||
// (`sessions.ts` is deliberately excluded: it legitimately needs LocalStorage.)
|
||||
for (const file of ["agent.ts", "ollama.ts", "tools.ts", "location.ts", "history.ts", "markdown.ts"]) {
|
||||
for (const file of [
|
||||
"agent.ts",
|
||||
"ollama.ts",
|
||||
"tools.ts",
|
||||
"location.ts",
|
||||
"history.ts",
|
||||
"markdown.ts",
|
||||
"capture.ts",
|
||||
"attachments.ts",
|
||||
"startup.ts",
|
||||
]) {
|
||||
const source = readFileSync(path.resolve(__dirname, `../lib/${file}`), "utf8");
|
||||
expect(source, `${file} should not import @vicinae/api`).not.toMatch(/from\s+"@vicinae\/api"/);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,177 @@
|
||||
import { mkdtemp, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { imageSource, mimeForPath, pngSize, previewBox, readAttachment } from "../lib/attachments";
|
||||
|
||||
const tempDirs: string[] = [];
|
||||
|
||||
/** A 24-byte PNG header, which is all `pngSize` reads. */
|
||||
function pngHeader(width: number, height: number): Buffer {
|
||||
const bytes = Buffer.alloc(24);
|
||||
Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]).copy(bytes, 0);
|
||||
bytes.writeUInt32BE(13, 8);
|
||||
bytes.write("IHDR", 12, "ascii");
|
||||
bytes.writeUInt32BE(width, 16);
|
||||
bytes.writeUInt32BE(height, 20);
|
||||
return bytes;
|
||||
}
|
||||
|
||||
async function tempDir(): Promise<string> {
|
||||
const dir = await mkdtemp(join(tmpdir(), "lam-attach-"));
|
||||
tempDirs.push(dir);
|
||||
return dir;
|
||||
}
|
||||
|
||||
afterEach(async () => {
|
||||
await Promise.all(tempDirs.splice(0).map((dir) => rm(dir, { recursive: true, force: true })));
|
||||
});
|
||||
|
||||
describe("mimeForPath", () => {
|
||||
it("names the image types Ollama accepts", () => {
|
||||
expect(mimeForPath("/tmp/snip.png")).toBe("image/png");
|
||||
expect(mimeForPath("/tmp/photo.JPG")).toBe("image/jpeg");
|
||||
expect(mimeForPath("/tmp/photo.jpeg")).toBe("image/jpeg");
|
||||
expect(mimeForPath("/tmp/shot.webp")).toBe("image/webp");
|
||||
});
|
||||
|
||||
it("does not guess a type for anything else", () => {
|
||||
expect(mimeForPath("/tmp/notes.txt")).toBe("application/octet-stream");
|
||||
expect(mimeForPath("/tmp/noextension")).toBe("application/octet-stream");
|
||||
});
|
||||
});
|
||||
|
||||
describe("pngSize", () => {
|
||||
it("reads the dimensions out of the IHDR chunk", () => {
|
||||
expect(pngSize(pngHeader(1600, 400))).toEqual({ width: 1600, height: 400 });
|
||||
});
|
||||
|
||||
it("refuses anything that is not a PNG", () => {
|
||||
// Guessing at another format would put a wrong aspect ratio into the layout.
|
||||
expect(pngSize(Buffer.from("not a png at all, just some text"))).toBeUndefined();
|
||||
expect(pngSize(Buffer.alloc(0))).toBeUndefined();
|
||||
});
|
||||
|
||||
it("refuses a header that is present but truncated", () => {
|
||||
expect(pngSize(pngHeader(10, 10).subarray(0, 20))).toBeUndefined();
|
||||
});
|
||||
|
||||
it("refuses degenerate dimensions", () => {
|
||||
expect(pngSize(pngHeader(0, 100))).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("previewBox", () => {
|
||||
it("fits a wide capture to the width limit and keeps the aspect ratio", () => {
|
||||
// This is the case that produced the dead space: 4:1 at full pane width was
|
||||
// letterboxed into a 200px-tall container it barely used.
|
||||
expect(previewBox(1600, 400)).toEqual({ width: 400, height: 100 });
|
||||
});
|
||||
|
||||
it("fits a 16:9 capture to the height limit instead", () => {
|
||||
expect(previewBox(1920, 1080)).toEqual({ width: 356, height: 200 });
|
||||
});
|
||||
|
||||
it("fits a portrait capture by its height", () => {
|
||||
expect(previewBox(400, 1600)).toEqual({ width: 50, height: 200 });
|
||||
});
|
||||
|
||||
it("does not blow a small capture up into a blurry one", () => {
|
||||
expect(previewBox(100, 50)).toEqual({ width: 100, height: 50 });
|
||||
});
|
||||
|
||||
it("has nothing to say when the size is unknown", () => {
|
||||
expect(previewBox(undefined, undefined)).toBeUndefined();
|
||||
expect(previewBox(1600, undefined)).toBeUndefined();
|
||||
expect(previewBox(1600, 0)).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("imageSource", () => {
|
||||
it("produces a file URL, which is the only scheme the renderer resolves for a local file", () => {
|
||||
expect(imageSource({ path: "/tmp/snip.png" })).toBe("file:///tmp/snip.png");
|
||||
});
|
||||
|
||||
it("escapes a path with a space, which the user's own screenshot directory has", () => {
|
||||
expect(imageSource({ path: "/Users/x/Pictures/Screenshot 2026-10-04.png" })).toBe(
|
||||
"file:///Users/x/Pictures/Screenshot%202026-10-04.png",
|
||||
);
|
||||
});
|
||||
|
||||
it("escapes non-ascii", () => {
|
||||
expect(imageSource({ path: "/tmp/caf\u00e9.png" })).toBe("file:///tmp/caf%C3%A9.png");
|
||||
});
|
||||
|
||||
it("keeps a bare path out of the markdown, because that resolves to nothing", () => {
|
||||
expect(imageSource({ path: "/tmp/snip.png" })).not.toBe("/tmp/snip.png");
|
||||
});
|
||||
|
||||
it("carries the size hints the renderer reads", () => {
|
||||
// `parseImageSize` looks for these names, and resolves the path from the URL
|
||||
// before the query, so the hints never reach the file name.
|
||||
expect(imageSource({ path: "/tmp/snip.png", width: 1600, height: 400 })).toBe(
|
||||
"file:///tmp/snip.png?raycast-width=400&raycast-height=100",
|
||||
);
|
||||
});
|
||||
|
||||
it("omits the hints when the dimensions are unknown", () => {
|
||||
expect(imageSource({ path: "/tmp/snip.png", width: undefined, height: undefined })).toBe(
|
||||
"file:///tmp/snip.png",
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("readAttachment", () => {
|
||||
it("reads the bytes as bare base64, with no data URL prefix", async () => {
|
||||
const dir = await tempDir();
|
||||
const path = join(dir, "snip.png");
|
||||
await writeFile(path, Buffer.from([0x89, 0x50, 0x4e, 0x47]));
|
||||
|
||||
const attachment = await readAttachment(path);
|
||||
|
||||
// Ollama wants the payload bare. A `data:image/png;base64,` prefix makes it
|
||||
// reject the image, and the mistake is invisible until a request comes back
|
||||
// with an error that says nothing about encoding.
|
||||
expect(attachment.base64).toBe("iVBORw==");
|
||||
expect(attachment.base64).not.toContain("data:");
|
||||
expect(attachment.mime).toBe("image/png");
|
||||
expect(attachment.path).toBe(path);
|
||||
});
|
||||
|
||||
it("round-trips a payload larger than one base64 padding boundary", async () => {
|
||||
const dir = await tempDir();
|
||||
const path = join(dir, "snip.png");
|
||||
const bytes = Buffer.alloc(300, 7);
|
||||
await writeFile(path, bytes);
|
||||
|
||||
const attachment = await readAttachment(path);
|
||||
|
||||
expect(Buffer.from(attachment.base64, "base64")).toEqual(bytes);
|
||||
});
|
||||
|
||||
it("keeps the dimensions, which the transcript needs long after this returns", async () => {
|
||||
const dir = await tempDir();
|
||||
const path = join(dir, "snip.png");
|
||||
await writeFile(path, pngHeader(1600, 400));
|
||||
|
||||
const attachment = await readAttachment(path);
|
||||
|
||||
expect(attachment.width).toBe(1600);
|
||||
expect(attachment.height).toBe(400);
|
||||
});
|
||||
|
||||
it("omits the dimensions for an image it cannot measure", async () => {
|
||||
const dir = await tempDir();
|
||||
const path = join(dir, "snip.png");
|
||||
await writeFile(path, Buffer.from([0x89, 0x50, 0x4e, 0x47]));
|
||||
|
||||
const attachment = await readAttachment(path);
|
||||
|
||||
expect(attachment.width).toBeUndefined();
|
||||
expect(attachment.height).toBeUndefined();
|
||||
});
|
||||
|
||||
it("fails loudly when the file is gone", async () => {
|
||||
await expect(readAttachment("/nonexistent/snip.png")).rejects.toThrow();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,329 @@
|
||||
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
CAPTURE_BACKENDS,
|
||||
CapturePermissionError,
|
||||
CaptureToolError,
|
||||
NoCaptureToolError,
|
||||
captureRegion,
|
||||
selectBackend,
|
||||
snipFileName,
|
||||
type CaptureBackend,
|
||||
} from "../lib/capture";
|
||||
|
||||
/** A `PATH` containing exactly the named tools. */
|
||||
function pathWith(...tools: string[]) {
|
||||
const present = new Set(tools);
|
||||
return async (tool: string) => present.has(tool);
|
||||
}
|
||||
|
||||
function backendNamed(name: string): CaptureBackend {
|
||||
const backend = CAPTURE_BACKENDS.find((candidate) => candidate.name === name);
|
||||
if (!backend) throw new Error(`no backend named ${name}`);
|
||||
return backend;
|
||||
}
|
||||
|
||||
/** A tool that ran, said nothing, and produced no image. */
|
||||
function silent(): { stdout: Buffer; stderr: string } {
|
||||
return { stdout: Buffer.alloc(0), stderr: "" };
|
||||
}
|
||||
|
||||
const tempDirs: string[] = [];
|
||||
|
||||
async function tempDir(): Promise<string> {
|
||||
const dir = await mkdtemp(join(tmpdir(), "lam-snip-"));
|
||||
tempDirs.push(dir);
|
||||
return dir;
|
||||
}
|
||||
|
||||
afterEach(async () => {
|
||||
await Promise.all(tempDirs.splice(0).map((dir) => rm(dir, { recursive: true, force: true })));
|
||||
});
|
||||
|
||||
describe("snipFileName", () => {
|
||||
it("is sortable and filesystem-safe", () => {
|
||||
expect(snipFileName(new Date(2026, 9, 4, 14, 59, 26))).toBe("snip-20261004-145926.png");
|
||||
});
|
||||
|
||||
it("zero-pads every field so names sort as text", () => {
|
||||
expect(snipFileName(new Date(2026, 0, 1, 0, 0, 0))).toBe("snip-20260101-000000.png");
|
||||
});
|
||||
});
|
||||
|
||||
describe("selectBackend", () => {
|
||||
it("returns nothing when no tool is installed", async () => {
|
||||
expect(await selectBackend(pathWith())).toBeUndefined();
|
||||
});
|
||||
|
||||
it("prefers grimblast on a Hyprland machine", async () => {
|
||||
const backend = await selectBackend(pathWith("grimblast", "grim", "slurp", "maim"));
|
||||
expect(backend?.name).toBe("grimblast");
|
||||
});
|
||||
|
||||
it("falls back to grim + slurp when grimblast is absent", async () => {
|
||||
const backend = await selectBackend(pathWith("grim", "slurp", "maim"));
|
||||
expect(backend?.name).toBe("grim + slurp");
|
||||
});
|
||||
|
||||
it("needs both halves of a pair, not just one", async () => {
|
||||
// grim without slurp cannot select a region, so it must not be chosen.
|
||||
const backend = await selectBackend(pathWith("grim", "maim"));
|
||||
expect(backend?.name).toBe("maim");
|
||||
});
|
||||
|
||||
it("falls back through the X11 tools", async () => {
|
||||
expect((await selectBackend(pathWith("maim")))?.name).toBe("maim");
|
||||
expect((await selectBackend(pathWith("scrot")))?.name).toBe("scrot");
|
||||
expect((await selectBackend(pathWith("flameshot")))?.name).toBe("flameshot");
|
||||
});
|
||||
|
||||
it("recognises the macOS built-in tool", async () => {
|
||||
expect((await selectBackend(pathWith("screencapture")))?.name).toBe("screencapture");
|
||||
});
|
||||
|
||||
it("can be given a shorter candidate list, which is how macOS is restricted", async () => {
|
||||
const onlyScreencapture = CAPTURE_BACKENDS.filter((b) => b.name === "screencapture");
|
||||
const backend = await selectBackend(pathWith("screencapture", "maim"), onlyScreencapture);
|
||||
|
||||
expect(backend?.name).toBe("screencapture");
|
||||
});
|
||||
});
|
||||
|
||||
describe("backend commands", () => {
|
||||
it("runs slurp before grim and passes the geometry through", async () => {
|
||||
const calls: Array<{ tool: string; args: string[] }> = [];
|
||||
const run = vi.fn(async (tool: string, args: string[]) => {
|
||||
calls.push({ tool, args });
|
||||
return tool === "slurp" ? { stdout: Buffer.from("120,340 800x600\n"), stderr: "" } : silent();
|
||||
});
|
||||
|
||||
await backendNamed("grim + slurp").capture({ output: "/tmp/snip.png", run });
|
||||
|
||||
expect(calls).toEqual([
|
||||
{ tool: "slurp", args: [] },
|
||||
{ tool: "grim", args: ["-g", "120,340 800x600", "/tmp/snip.png"] },
|
||||
]);
|
||||
});
|
||||
|
||||
it("does not shell out to grim when the selection was cancelled", async () => {
|
||||
const run = vi.fn(async () => silent());
|
||||
|
||||
await backendNamed("grim + slurp").capture({ output: "/tmp/snip.png", run });
|
||||
|
||||
expect(run).toHaveBeenCalledTimes(1);
|
||||
expect(run).toHaveBeenCalledWith("slurp", [], expect.anything());
|
||||
});
|
||||
|
||||
it("keeps a shell pipeline out of grim by using argv", () => {
|
||||
// `grim -g "$(slurp)"` is the documented one-liner, but a shell here would
|
||||
// be a needless injection surface and would not survive a path with spaces.
|
||||
expect(backendNamed("grim + slurp").requires).toEqual(["grim", "slurp"]);
|
||||
});
|
||||
|
||||
it("writes flameshot's stdout, because it refuses an output path", async () => {
|
||||
const dir = await tempDir();
|
||||
const output = join(dir, "snip.png");
|
||||
const png = Buffer.from([0x89, 0x50, 0x4e, 0x47]);
|
||||
|
||||
await backendNamed("flameshot").capture({ output, run: async () => ({ stdout: png, stderr: "" }) });
|
||||
|
||||
expect(await readFile(output)).toEqual(png);
|
||||
});
|
||||
|
||||
it("writes nothing when flameshot returned no image", async () => {
|
||||
const dir = await tempDir();
|
||||
const output = join(dir, "snip.png");
|
||||
|
||||
await backendNamed("flameshot").capture({ output, run: async () => silent() });
|
||||
|
||||
await expect(readFile(output)).rejects.toThrow();
|
||||
});
|
||||
|
||||
it("asks screencapture for an interactive region and no sound", async () => {
|
||||
const run = vi.fn(async () => silent());
|
||||
|
||||
await backendNamed("screencapture").capture({ output: "/tmp/snip.png", run });
|
||||
|
||||
expect(run).toHaveBeenCalledWith("screencapture", ["-i", "-x", "/tmp/snip.png"], expect.anything());
|
||||
});
|
||||
});
|
||||
|
||||
describe("captureRegion", () => {
|
||||
/** A `run` that writes a real file, which is what the success path checks for. */
|
||||
function writingRun(bytes = Buffer.from([0x89, 0x50])) {
|
||||
return async (_tool: string, args: string[]) => {
|
||||
await writeFile(args[args.length - 1], bytes);
|
||||
return silent();
|
||||
};
|
||||
}
|
||||
|
||||
it("writes into the directory it was given and reports the path", async () => {
|
||||
const dir = await tempDir();
|
||||
const output = join(dir, "snip-20261004-145926.png");
|
||||
|
||||
const result = await captureRegion({
|
||||
outputDir: dir,
|
||||
platform: "darwin",
|
||||
now: new Date(2026, 9, 4, 14, 59, 26),
|
||||
hasCommand: pathWith("screencapture"),
|
||||
run: writingRun(Buffer.from([0x89, 0x50])),
|
||||
});
|
||||
|
||||
expect(result).toEqual({ path: output, bytes: 2, backend: "screencapture" });
|
||||
});
|
||||
|
||||
it("creates the output directory on first use", async () => {
|
||||
const dir = await tempDir();
|
||||
const nested = join(dir, "snips");
|
||||
const now = new Date(2026, 9, 4, 14, 59, 26);
|
||||
|
||||
await captureRegion({
|
||||
outputDir: nested,
|
||||
platform: "darwin",
|
||||
now,
|
||||
hasCommand: pathWith("screencapture"),
|
||||
run: writingRun(Buffer.from([0x89])),
|
||||
});
|
||||
|
||||
expect((await readFile(join(nested, snipFileName(now)))).length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it("returns null when the user backed out and said nothing", async () => {
|
||||
const dir = await tempDir();
|
||||
|
||||
const result = await captureRegion({
|
||||
outputDir: dir,
|
||||
platform: "darwin",
|
||||
hasCommand: pathWith("screencapture"),
|
||||
run: async () => silent(),
|
||||
});
|
||||
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
|
||||
it("treats an empty file as a cancel rather than an attachment", async () => {
|
||||
// At least one tool exits 0 on cancel, so exit codes cannot be trusted. A
|
||||
// zero-byte PNG would be sent to the model as a broken image.
|
||||
const dir = await tempDir();
|
||||
|
||||
const result = await captureRegion({
|
||||
outputDir: dir,
|
||||
platform: "darwin",
|
||||
hasCommand: pathWith("screencapture"),
|
||||
run: writingRun(Buffer.alloc(0)),
|
||||
});
|
||||
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
|
||||
it("still succeeds when the tool also complained on stderr", async () => {
|
||||
// Some tools warn on stderr and write the image anyway. The file is the
|
||||
// source of truth, so a warning must not turn a good capture into an error.
|
||||
const dir = await tempDir();
|
||||
|
||||
const result = await captureRegion({
|
||||
outputDir: dir,
|
||||
platform: "darwin",
|
||||
hasCommand: pathWith("screencapture"),
|
||||
run: async (_tool, args) => {
|
||||
await writeFile(args[args.length - 1], Buffer.from([0x89]));
|
||||
return { stdout: Buffer.alloc(0), stderr: "warning: something\n" };
|
||||
},
|
||||
});
|
||||
|
||||
expect(result?.bytes).toBe(1);
|
||||
});
|
||||
|
||||
describe("macOS Screen Recording permission", () => {
|
||||
// The message is the only difference between a refused permission and a
|
||||
// cancelled selection: both leave no file behind, and screencapture has been
|
||||
// reported to exit 0 when the permission is missing.
|
||||
const denial = "could not create image from rect\n";
|
||||
|
||||
it("reports a permission problem instead of a cancel when the tool exits non-zero", async () => {
|
||||
const dir = await tempDir();
|
||||
|
||||
await expect(
|
||||
captureRegion({
|
||||
outputDir: dir,
|
||||
platform: "darwin",
|
||||
hasCommand: pathWith("screencapture"),
|
||||
run: async () => {
|
||||
throw new CaptureToolError("screencapture", denial);
|
||||
},
|
||||
}),
|
||||
).rejects.toThrow(CapturePermissionError);
|
||||
});
|
||||
|
||||
it("reports a permission problem even when the tool exits 0", async () => {
|
||||
// The documented exit-0 shape. Without reading stderr this is
|
||||
// indistinguishable from Escape, which is how a silently useless feature
|
||||
// gets shipped.
|
||||
const dir = await tempDir();
|
||||
|
||||
await expect(
|
||||
captureRegion({
|
||||
outputDir: dir,
|
||||
platform: "darwin",
|
||||
hasCommand: pathWith("screencapture"),
|
||||
run: async () => ({ stdout: Buffer.alloc(0), stderr: denial }),
|
||||
}),
|
||||
).rejects.toThrow(CapturePermissionError);
|
||||
});
|
||||
|
||||
it("names Vicinae rather than the extension, because that is who holds the grant", async () => {
|
||||
const dir = await tempDir();
|
||||
const attempt = () =>
|
||||
captureRegion({
|
||||
outputDir: dir,
|
||||
platform: "darwin",
|
||||
hasCommand: pathWith("screencapture"),
|
||||
run: async () => {
|
||||
throw new CaptureToolError("screencapture", denial);
|
||||
},
|
||||
});
|
||||
|
||||
await expect(attempt()).rejects.toMatchObject({
|
||||
name: "CapturePermissionError",
|
||||
hint: expect.stringContaining("Vicinae"),
|
||||
});
|
||||
await expect(attempt()).rejects.toMatchObject({
|
||||
hint: expect.stringContaining("Screen Recording"),
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
it("reports any other explanation rather than swallowing it", async () => {
|
||||
const dir = await tempDir();
|
||||
|
||||
await expect(
|
||||
captureRegion({
|
||||
outputDir: dir,
|
||||
platform: "linux",
|
||||
hasCommand: pathWith("maim"),
|
||||
run: async () => {
|
||||
throw new CaptureToolError("maim", "maim: cannot open display\n");
|
||||
},
|
||||
}),
|
||||
).rejects.toThrow(CaptureToolError);
|
||||
});
|
||||
|
||||
it("explains what to install when nothing is available", async () => {
|
||||
const dir = await tempDir();
|
||||
|
||||
await expect(
|
||||
captureRegion({ outputDir: dir, platform: "linux", hasCommand: pathWith() }),
|
||||
).rejects.toThrow(NoCaptureToolError);
|
||||
});
|
||||
|
||||
it("does not invent a backend on macOS", async () => {
|
||||
const dir = await tempDir();
|
||||
|
||||
// maim is available in this fake PATH but is meaningless on darwin.
|
||||
await expect(
|
||||
captureRegion({ outputDir: dir, platform: "darwin", hasCommand: pathWith("maim") }),
|
||||
).rejects.toThrow(NoCaptureToolError);
|
||||
});
|
||||
});
|
||||
@@ -4,6 +4,8 @@ import {
|
||||
labelCodeFences,
|
||||
promoteStandaloneCodeSpans,
|
||||
repairCodeBlocks,
|
||||
restoreUrlLinks,
|
||||
separateTables,
|
||||
} from "../lib/code";
|
||||
|
||||
describe("inferLanguage", () => {
|
||||
@@ -132,11 +134,207 @@ describe("promoteStandaloneCodeSpans", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("restoreUrlLinks", () => {
|
||||
it("unwraps a fence that is nothing but URLs", () => {
|
||||
// The reported case: "here are the links" arriving as a code block, so the
|
||||
// one thing a URL is for, being clicked, does not work.
|
||||
const input = [
|
||||
"```",
|
||||
"https://github.com/jasonkneen/openclicky",
|
||||
"https://github.com/unn-Known1/clickyX",
|
||||
"```",
|
||||
].join("\n");
|
||||
|
||||
expect(restoreUrlLinks(input)).toBe(
|
||||
[
|
||||
"[https://github.com/jasonkneen/openclicky](https://github.com/jasonkneen/openclicky)",
|
||||
"[https://github.com/unn-Known1/clickyX](https://github.com/unn-Known1/clickyX)",
|
||||
].join("\n"),
|
||||
);
|
||||
});
|
||||
|
||||
it("unwraps a URL fence even when the model labelled it", () => {
|
||||
expect(restoreUrlLinks("```bash\nhttps://example.com\n```")).toBe(
|
||||
"[https://example.com](https://example.com)",
|
||||
);
|
||||
});
|
||||
|
||||
it("links a bare URL standing alone on a line", () => {
|
||||
expect(restoreUrlLinks("https://example.com")).toBe("[https://example.com](https://example.com)");
|
||||
});
|
||||
|
||||
it("links a lone inline span holding a URL", () => {
|
||||
expect(restoreUrlLinks("`https://example.com`")).toBe(
|
||||
"[https://example.com](https://example.com)",
|
||||
);
|
||||
});
|
||||
|
||||
it("keeps a bulleted list of links a bulleted list", () => {
|
||||
const input = "- https://a.example\n- https://b.example";
|
||||
|
||||
expect(restoreUrlLinks(input)).toBe(
|
||||
"- [https://a.example](https://a.example)\n- [https://b.example](https://b.example)",
|
||||
);
|
||||
});
|
||||
|
||||
it("keeps an ordered list marker", () => {
|
||||
expect(restoreUrlLinks("1. https://example.com")).toBe(
|
||||
"1. [https://example.com](https://example.com)",
|
||||
);
|
||||
});
|
||||
|
||||
it("drops sentence punctuation that is not part of the address", () => {
|
||||
expect(restoreUrlLinks("See this:\n\nhttps://example.com.")).toContain(
|
||||
"[https://example.com](https://example.com)",
|
||||
);
|
||||
});
|
||||
|
||||
it("brackets a destination that contains parentheses", () => {
|
||||
const url = "https://en.wikipedia.org/wiki/Foo_(bar)";
|
||||
expect(restoreUrlLinks(url)).toBe(`[${url}](<${url}>)`);
|
||||
});
|
||||
|
||||
it("escapes brackets in the label, which an IPv6 literal contains", () => {
|
||||
// Unescaped, the `]` in the label ends the link text early and the rest of
|
||||
// the address is left as prose outside the link.
|
||||
expect(restoreUrlLinks("http://[::1]:8080/")).toBe(
|
||||
"[http://\\[::1\\]:8080/](http://[::1]:8080/)",
|
||||
);
|
||||
});
|
||||
|
||||
it("never touches a fence holding real code", () => {
|
||||
const input = "```bash\ncurl https://example.com | jq .\n```";
|
||||
expect(restoreUrlLinks(input)).toBe(input);
|
||||
});
|
||||
|
||||
it("never touches a fence when one line is not a URL", () => {
|
||||
const input = "```\nhttps://example.com\nnot a url\n```";
|
||||
expect(restoreUrlLinks(input)).toBe(input);
|
||||
});
|
||||
|
||||
it("leaves a URL embedded in a sentence alone", () => {
|
||||
const input = "The docs at https://example.com cover it.";
|
||||
expect(restoreUrlLinks(input)).toBe(input);
|
||||
});
|
||||
|
||||
it("leaves a markdown link the model already wrote alone", () => {
|
||||
const input = "[OpenClickly](https://github.com/jasonkneen/openclicky)";
|
||||
expect(restoreUrlLinks(input)).toBe(input);
|
||||
});
|
||||
|
||||
it("preserves a fence that is never closed and is real code", () => {
|
||||
expect(restoreUrlLinks("```\ngit log")).toBe("```\ngit log");
|
||||
});
|
||||
|
||||
it("is idempotent", () => {
|
||||
const once = restoreUrlLinks("```\nhttps://example.com\n```");
|
||||
expect(restoreUrlLinks(once)).toBe(once);
|
||||
});
|
||||
|
||||
it("is a no-op on ordinary prose", () => {
|
||||
const input = "There are no addresses in this sentence.";
|
||||
expect(restoreUrlLinks(input)).toBe(input);
|
||||
});
|
||||
});
|
||||
|
||||
describe("separateTables", () => {
|
||||
/** The table from the answer that prompted this, verbatim. */
|
||||
const table = [
|
||||
"| snip | source | aspect | dead space before | box after |",
|
||||
"| --- | --- | --- | --- | --- |",
|
||||
"| `…153223` | 626×184 | 3.40 | **~82px** | 400×118 |",
|
||||
].join("\n");
|
||||
|
||||
it("gives a table the blank line it needs to be a table", () => {
|
||||
// GFM tables cannot interrupt a paragraph, so glued to the line above they
|
||||
// render as a paragraph of pipes and dashes.
|
||||
const repaired = separateTables(`Here are the numbers:\n${table}`);
|
||||
|
||||
expect(repaired).toBe(`Here are the numbers:\n\n${table}`);
|
||||
});
|
||||
|
||||
it("leaves a table alone when it already has one", () => {
|
||||
const already = `Here are the numbers:\n\n${table}`;
|
||||
expect(separateTables(already)).toBe(already);
|
||||
});
|
||||
|
||||
it("leaves a table alone when it opens the answer", () => {
|
||||
expect(separateTables(table)).toBe(table);
|
||||
});
|
||||
|
||||
it("leaves a table alone when it follows a fence", () => {
|
||||
const input = `\`\`\`bash\npwd\n\`\`\`\n${table}`;
|
||||
expect(separateTables(input)).toBe(input);
|
||||
});
|
||||
|
||||
it("does not touch a table inside a fence", () => {
|
||||
// A table shown as example code has to stay exactly as written, blank line
|
||||
// or not.
|
||||
const fenced = `\`\`\`markdown\n${table}\n\`\`\``;
|
||||
expect(separateTables(fenced)).toBe(fenced);
|
||||
});
|
||||
|
||||
it("leaves prose containing a pipe alone", () => {
|
||||
const input = "Use the pipe character:\nls | grep foo";
|
||||
expect(separateTables(input)).toBe(input);
|
||||
});
|
||||
|
||||
it("does not insert a blank line before a data row", () => {
|
||||
// Only the header starts a table. A blank line inside one would split it.
|
||||
const input = `text\n${table}\n| a | b | c | d | e |`;
|
||||
expect(separateTables(input)).toBe(`text\n\n${table}\n| a | b | c | d | e |`);
|
||||
});
|
||||
|
||||
it("recognises a one-dash delimiter, which GFM allows", () => {
|
||||
const minimal = "| a | b |\n| - | - |\n| 1 | 2 |";
|
||||
expect(separateTables(`text\n${minimal}`)).toBe(`text\n\n${minimal}`);
|
||||
});
|
||||
|
||||
it("recognises alignment colons", () => {
|
||||
const aligned = "| a | b |\n|:---|:---:|\n| 1 | 2 |";
|
||||
expect(separateTables(`text\n${aligned}`)).toBe(`text\n\n${aligned}`);
|
||||
});
|
||||
|
||||
it("is idempotent", () => {
|
||||
const once = separateTables(`text\n${table}`);
|
||||
expect(separateTables(once)).toBe(once);
|
||||
});
|
||||
});
|
||||
|
||||
describe("repairCodeBlocks", () => {
|
||||
it("promotes a lone command and labels it in one pass", () => {
|
||||
expect(repairCodeBlocks("`pwd`")).toBe("```bash\npwd\n```");
|
||||
});
|
||||
|
||||
it("leaves a table exactly as the model wrote it", () => {
|
||||
// The whole repair pass runs over every answer, and a table has to survive
|
||||
// it untouched: pipes are not fences, not links, and not lone spans.
|
||||
const table = [
|
||||
"| snip | source | aspect | box after |",
|
||||
"| --- | --- | --- | --- |",
|
||||
"| `…153223` | 626×184 | 3.40 | 400×118 |",
|
||||
"| [docs](https://example.com) | `pwd` | https://example.com | 400×60 |",
|
||||
].join("\n");
|
||||
|
||||
expect(repairCodeBlocks(table)).toBe(table);
|
||||
});
|
||||
|
||||
it("does not fence a lone URL", () => {
|
||||
// Restoring links runs before promotion, so a recovered URL never becomes a
|
||||
// fence again. Fencing it here was the bug the user saw.
|
||||
const repaired = repairCodeBlocks("`https://example.com`");
|
||||
|
||||
expect(repaired).not.toContain("```");
|
||||
expect(repaired).toBe("[https://example.com](https://example.com)");
|
||||
});
|
||||
|
||||
it("unwraps a fenced list of links and leaves no fence behind", () => {
|
||||
const repaired = repairCodeBlocks("```\nhttps://a.example\nhttps://b.example\n```");
|
||||
|
||||
expect(repaired).not.toContain("```");
|
||||
expect(repaired).toContain("[https://a.example](https://a.example)");
|
||||
});
|
||||
|
||||
it("still labels a bare multi-line fence", () => {
|
||||
expect(repairCodeBlocks("```\ngit status\n```")).toBe("```bash\ngit status\n```");
|
||||
});
|
||||
|
||||
@@ -0,0 +1,218 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { __resetLocalStorage } from "../__mocks__/@vicinae/api";
|
||||
import { GEO_TTL_MS, buildPlace, forgetGeoPlace, ipPlace } from "../lib/geo";
|
||||
|
||||
/**
|
||||
* The only code in the extension that sends anything about the user to a third
|
||||
* party, so the parts worth pinning down are that it caches (rather than leaking
|
||||
* the address on every question), that it never throws, and that a failure is
|
||||
* reported instead of being served as a coarser answer.
|
||||
*/
|
||||
|
||||
const fetchMock = vi.fn();
|
||||
|
||||
function answer(payload: unknown, status = 200): Response {
|
||||
return new Response(JSON.stringify(payload), {
|
||||
status,
|
||||
headers: { "Content-Type": "application/json" },
|
||||
});
|
||||
}
|
||||
|
||||
/** A real ipwho.is payload, trimmed. Note the nested timezone and `country`. */
|
||||
const CEBU = {
|
||||
success: true,
|
||||
city: "Cebu City",
|
||||
region: "Central Visayas",
|
||||
country: "Philippines",
|
||||
latitude: 10.3167,
|
||||
longitude: 123.8907,
|
||||
timezone: { id: "Asia/Manila", utc: "+08:00" },
|
||||
};
|
||||
|
||||
beforeEach(() => {
|
||||
__resetLocalStorage();
|
||||
fetchMock.mockReset();
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
|
||||
describe("buildPlace", () => {
|
||||
it("joins the parts that exist", () => {
|
||||
expect(buildPlace({ city: "Cebu City", region: "Central Visayas", country: "Philippines" })).toBe(
|
||||
"Cebu City, Central Visayas, Philippines",
|
||||
);
|
||||
});
|
||||
|
||||
it("drops repeats, which providers return for city-states", () => {
|
||||
// "Singapore, Singapore, Singapore" reads like a bug rather than a place.
|
||||
expect(buildPlace({ city: "Singapore", region: "Singapore", country: "Singapore" })).toBe("Singapore");
|
||||
});
|
||||
|
||||
it("ignores repeats that differ only in case", () => {
|
||||
expect(buildPlace({ city: "Manila", region: "manila" })).toBe("Manila");
|
||||
});
|
||||
|
||||
it("drops empty and whitespace parts", () => {
|
||||
expect(buildPlace({ city: " ", region: "", country: "Philippines" })).toBe("Philippines");
|
||||
});
|
||||
|
||||
it("returns nothing when there is nothing", () => {
|
||||
expect(buildPlace({})).toBe("");
|
||||
});
|
||||
});
|
||||
|
||||
describe("ipPlace", () => {
|
||||
it("resolves a place from a provider answer", async () => {
|
||||
fetchMock.mockResolvedValueOnce(answer(CEBU));
|
||||
|
||||
const result = await ipPlace({ now: 1_000 });
|
||||
|
||||
expect(result.status).toBe("ok");
|
||||
if (result.status !== "ok") return;
|
||||
|
||||
expect(result.place.place).toBe("Cebu City, Central Visayas, Philippines");
|
||||
expect(result.place.latitude).toBe(10.3167);
|
||||
expect(result.place.timezone).toBe("Asia/Manila");
|
||||
expect(result.place.source).toBe("ipwho.is");
|
||||
});
|
||||
|
||||
it("names itself to the provider rather than arriving anonymously", async () => {
|
||||
fetchMock.mockResolvedValueOnce(answer(CEBU));
|
||||
await ipPlace({ now: 1_000 });
|
||||
|
||||
const [url, init] = fetchMock.mock.calls[0];
|
||||
expect(url).toBe("https://ipwho.is/");
|
||||
expect(init.headers.Accept).toBe("application/json");
|
||||
expect(init.headers["User-Agent"]).toContain("lam-agent");
|
||||
});
|
||||
|
||||
it("reads a timezone handed back as a plain string too", async () => {
|
||||
// Providers differ here: ipwho.is nests it, others return a bare zone name.
|
||||
fetchMock.mockResolvedValueOnce(answer({ ...CEBU, timezone: "Asia/Manila" }));
|
||||
|
||||
const result = await ipPlace({ now: 1_000 });
|
||||
|
||||
expect(result.status === "ok" && result.place.timezone).toBe("Asia/Manila");
|
||||
});
|
||||
|
||||
it("serves a repeat from the cache instead of re-sending the address", async () => {
|
||||
fetchMock.mockResolvedValueOnce(answer(CEBU));
|
||||
|
||||
await ipPlace({ now: 1_000 });
|
||||
const second = await ipPlace({ now: 2_000 });
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1);
|
||||
expect(second.status).toBe("ok");
|
||||
});
|
||||
|
||||
it("re-fetches once the cached place is stale", async () => {
|
||||
fetchMock.mockResolvedValueOnce(answer(CEBU));
|
||||
await ipPlace({ now: 1_000 });
|
||||
|
||||
fetchMock.mockResolvedValueOnce(answer({ ...CEBU, city: "Davao City" }));
|
||||
const later = await ipPlace({ now: 1_000 + GEO_TTL_MS + 1 });
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(2);
|
||||
expect(later.status === "ok" && later.place.city).toBe("Davao City");
|
||||
});
|
||||
|
||||
it("treats exactly the TTL as still fresh", async () => {
|
||||
fetchMock.mockResolvedValueOnce(answer(CEBU));
|
||||
await ipPlace({ now: 1_000 });
|
||||
|
||||
await ipPlace({ now: 1_000 + GEO_TTL_MS });
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("ignores the cache when asked to refresh", async () => {
|
||||
fetchMock.mockResolvedValueOnce(answer(CEBU));
|
||||
await ipPlace({ now: 1_000 });
|
||||
|
||||
fetchMock.mockResolvedValueOnce(answer({ ...CEBU, city: "Davao City" }));
|
||||
const refreshed = await ipPlace({ now: 1_001, refresh: true });
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(2);
|
||||
expect(refreshed.status === "ok" && refreshed.place.city).toBe("Davao City");
|
||||
});
|
||||
|
||||
it("reports a rate limit by status rather than pretending to know the place", async () => {
|
||||
fetchMock.mockResolvedValueOnce(new Response("slow down", { status: 429 }));
|
||||
|
||||
const result = await ipPlace({ now: 1_000 });
|
||||
|
||||
expect(result).toEqual({ status: "failed", message: "ipwho.is answered HTTP 429" });
|
||||
});
|
||||
|
||||
it("reports a provider that refuses in the body while answering 200", async () => {
|
||||
// ipwho.is signals failure with success: false rather than a status code,
|
||||
// so checking the status alone would read a refusal as a success.
|
||||
fetchMock.mockResolvedValueOnce(answer({ success: false, message: "Reserved range" }));
|
||||
|
||||
const result = await ipPlace({ now: 1_000 });
|
||||
|
||||
expect(result).toEqual({ status: "failed", message: "Reserved range" });
|
||||
});
|
||||
|
||||
it("reports a body that is not JSON", async () => {
|
||||
fetchMock.mockResolvedValueOnce(new Response("<html>nope</html>", { status: 200 }));
|
||||
|
||||
const result = await ipPlace({ now: 1_000 });
|
||||
|
||||
expect(result.status).toBe("failed");
|
||||
});
|
||||
|
||||
it("reports an answer with no usable place", async () => {
|
||||
fetchMock.mockResolvedValueOnce(answer({ success: true, city: null, region: null, country: null }));
|
||||
|
||||
const result = await ipPlace({ now: 1_000 });
|
||||
|
||||
expect(result).toEqual({ status: "failed", message: "ipwho.is returned no usable place" });
|
||||
});
|
||||
|
||||
it("survives the network being down without throwing", async () => {
|
||||
fetchMock.mockRejectedValueOnce(new Error("offline"));
|
||||
|
||||
const result = await ipPlace({ now: 1_000 });
|
||||
|
||||
expect(result.status).toBe("failed");
|
||||
});
|
||||
|
||||
it("does not cache a failure", async () => {
|
||||
fetchMock.mockRejectedValueOnce(new Error("offline"));
|
||||
await ipPlace({ now: 1_000 });
|
||||
|
||||
fetchMock.mockResolvedValueOnce(answer(CEBU));
|
||||
const second = await ipPlace({ now: 1_001 });
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(2);
|
||||
expect(second.status).toBe("ok");
|
||||
});
|
||||
|
||||
it("ignores a corrupt cache entry rather than failing", async () => {
|
||||
const { LocalStorage } = await import("../__mocks__/@vicinae/api");
|
||||
await LocalStorage.setItem("lam.geo", "{not json");
|
||||
|
||||
fetchMock.mockResolvedValueOnce(answer(CEBU));
|
||||
|
||||
await expect(ipPlace({ now: 1_000 })).resolves.toMatchObject({ status: "ok" });
|
||||
});
|
||||
});
|
||||
|
||||
describe("forgetGeoPlace", () => {
|
||||
it("drops the cached place so a move is not pinned for hours", async () => {
|
||||
fetchMock.mockResolvedValueOnce(answer(CEBU));
|
||||
await ipPlace({ now: 1_000 });
|
||||
|
||||
await forgetGeoPlace();
|
||||
|
||||
fetchMock.mockResolvedValueOnce(answer({ ...CEBU, city: "Davao City" }));
|
||||
const after = await ipPlace({ now: 1_001 });
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(2);
|
||||
expect(after.status === "ok" && after.place.city).toBe("Davao City");
|
||||
});
|
||||
});
|
||||
@@ -4,7 +4,7 @@ import { buildMessages, composeSystemPrompt } from "../lib/history";
|
||||
import type { ChatTurn } from "../lib/types";
|
||||
|
||||
function turn(prompt: string, response: string): ChatTurn {
|
||||
return { id: prompt, prompt, response, thinking: "", model: "m", createdAt: 0, toolCalls: [] };
|
||||
return { id: prompt, prompt, response, thinking: "", model: "m", createdAt: 0, toolCalls: [], attachments: [] };
|
||||
}
|
||||
|
||||
describe("buildMessages", () => {
|
||||
@@ -66,3 +66,37 @@ describe("composeSystemPrompt", () => {
|
||||
expect(composeSystemPrompt("Be terse.", false)).not.toContain(RECENCY_HINT);
|
||||
});
|
||||
});
|
||||
|
||||
describe("buildMessages images", () => {
|
||||
it("attaches images to the question being asked", () => {
|
||||
const messages = buildMessages("SYSTEM", [], "what is this?", ["BASE64"]);
|
||||
|
||||
expect(messages[messages.length - 1]).toEqual({
|
||||
role: "user",
|
||||
content: "what is this?",
|
||||
images: ["BASE64"],
|
||||
});
|
||||
});
|
||||
|
||||
it("omits the images key entirely when there is no attachment", () => {
|
||||
// The key has to be absent, not an empty array: Ollama treats `images: []`
|
||||
// differently from no images at all on some models.
|
||||
const messages = buildMessages("SYSTEM", [], "hello");
|
||||
|
||||
expect(messages[messages.length - 1]).toEqual({ role: "user", content: "hello" });
|
||||
});
|
||||
|
||||
it("attaches several images to one question", () => {
|
||||
const messages = buildMessages("SYSTEM", [], "compare these", ["A", "B"]);
|
||||
|
||||
expect(messages[messages.length - 1].images).toEqual(["A", "B"]);
|
||||
});
|
||||
|
||||
it("does not replay images from earlier turns", () => {
|
||||
// A screenshot is the largest thing a request can carry, and re-sending it
|
||||
// with every later question is exactly what dropping tool output avoids.
|
||||
const messages = buildMessages("SYSTEM", [turn("earlier", "answer")], "follow up");
|
||||
|
||||
expect(messages.some((message) => message.images !== undefined)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,21 +1,53 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { describeTimezone, getLocationContext, utcOffsetLabel, weekdayName } from "../lib/location";
|
||||
import {
|
||||
describeTimezoneArea,
|
||||
getLocationContext,
|
||||
localeRegion,
|
||||
utcOffsetLabel,
|
||||
weekdayName,
|
||||
} from "../lib/location";
|
||||
|
||||
describe("describeTimezone", () => {
|
||||
it("splits a simple zone into region and city", () => {
|
||||
expect(describeTimezone("Asia/Manila")).toEqual({ region: "Asia", city: "Manila" });
|
||||
describe("describeTimezoneArea", () => {
|
||||
it("keeps the zone's own label", () => {
|
||||
expect(describeTimezoneArea("Asia/Manila")).toEqual({ area: "Manila" });
|
||||
});
|
||||
|
||||
it("handles zones with more than two segments", () => {
|
||||
expect(describeTimezone("America/Argentina/Salta")).toEqual({ region: "America", city: "Salta" });
|
||||
expect(describeTimezoneArea("America/Argentina/Salta")).toEqual({ area: "Salta" });
|
||||
});
|
||||
|
||||
it("underscores become spaces", () => {
|
||||
expect(describeTimezone("America/New_York")).toEqual({ region: "America", city: "New York" });
|
||||
it("turns underscores into spaces", () => {
|
||||
expect(describeTimezoneArea("America/New_York")).toEqual({ area: "New York" });
|
||||
});
|
||||
|
||||
it("drops the continent grouping, which answers a question nobody asked", () => {
|
||||
// "Asia" beside a locale region of "Philippines" reads as two answers to
|
||||
// the same question.
|
||||
expect(describeTimezoneArea("Asia/Manila")).not.toHaveProperty("region");
|
||||
});
|
||||
|
||||
it("returns nothing for a bare zone like UTC", () => {
|
||||
expect(describeTimezone("UTC")).toEqual({});
|
||||
expect(describeTimezoneArea("UTC")).toEqual({});
|
||||
});
|
||||
});
|
||||
|
||||
describe("localeRegion", () => {
|
||||
it("reads the region the user configured, rather than inferring one", () => {
|
||||
expect(localeRegion("en-PH")).toEqual({ code: "PH", name: "Philippines" });
|
||||
});
|
||||
|
||||
it("handles other regions", () => {
|
||||
expect(localeRegion("de-DE")?.name).toBe("Germany");
|
||||
expect(localeRegion("en-US")?.code).toBe("US");
|
||||
});
|
||||
|
||||
it("has nothing to say for a locale with no region", () => {
|
||||
expect(localeRegion("en")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("does not throw on a malformed locale", () => {
|
||||
expect(localeRegion("not a locale")).toBeUndefined();
|
||||
expect(localeRegion("")).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -54,10 +86,11 @@ describe("getLocationContext", () => {
|
||||
const context = getLocationContext(new Date("2026-01-02T03:04:05Z"), "Asia/Manila");
|
||||
|
||||
expect(context.timezone).toBe("Asia/Manila");
|
||||
expect(context.city).toBe("Manila");
|
||||
expect(context.region).toBe("Asia");
|
||||
expect(context.utcOffset).toContain("+08:00");
|
||||
expect(context.locale.length).toBeGreaterThan(0);
|
||||
// The zone's label is reported as such, never as the user's city.
|
||||
expect(context.timezoneArea).toBe("Manila");
|
||||
expect(context).not.toHaveProperty("city");
|
||||
});
|
||||
|
||||
it("still returns a usable context for an unknown zone", () => {
|
||||
@@ -65,4 +98,15 @@ describe("getLocationContext", () => {
|
||||
expect(context.timezone).toBe("Not/AReal_Zone");
|
||||
expect(context.utcOffset).toBe("UTC");
|
||||
});
|
||||
|
||||
it("carries the configured place through verbatim", () => {
|
||||
const context = getLocationContext(new Date(), "Asia/Manila", " Cebu City, Philippines ");
|
||||
expect(context.place).toBe("Cebu City, Philippines");
|
||||
});
|
||||
|
||||
it("treats an empty or whitespace place as unset", () => {
|
||||
expect(getLocationContext(new Date(), "Asia/Manila", "").place).toBeUndefined();
|
||||
expect(getLocationContext(new Date(), "Asia/Manila", " ").place).toBeUndefined();
|
||||
expect(getLocationContext(new Date(), "Asia/Manila").place).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -15,6 +15,7 @@ function message(overrides: Partial<ThreadMessage> & Pick<ThreadMessage, "role">
|
||||
isStreaming: false,
|
||||
thinking: "",
|
||||
toolCalls: [],
|
||||
attachments: [],
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
@@ -74,9 +75,9 @@ describe("buildThreadMarkdown", () => {
|
||||
expect(markdown.indexOf("second question")).toBeLessThan(markdown.indexOf("first answer"));
|
||||
});
|
||||
|
||||
it("renders no images, because markdown cannot resolve them", () => {
|
||||
// Images render block-level and centred at up to 200px (`MdImage.qml`), and
|
||||
// every candidate icon source fails. Keep the transcript text-only.
|
||||
it("renders no images for a message that has no attachment", () => {
|
||||
// Images only ever come from an attachment this extension took itself. A URL
|
||||
// the model invented is not resolvable and renders as an empty block.
|
||||
const markdown = buildThreadMarkdown([
|
||||
message({ role: "user", text: "q" }),
|
||||
message({ role: "assistant", text: "a" }),
|
||||
@@ -146,6 +147,34 @@ describe("buildThreadMarkdown", () => {
|
||||
expect(markdown).not.toContain("```bash");
|
||||
});
|
||||
|
||||
it("turns a fenced list of URLs into links, because a fence is not clickable", () => {
|
||||
// The reported case: asked for alternatives, the model answered with a code
|
||||
// block of bare GitHub addresses. `MdCodeBlock.qml` draws that as a dark
|
||||
// panel with a Copy button and no link handling, so the address could not be
|
||||
// opened. Only `CMARK_NODE_LINK` becomes an `<a href>` the selection
|
||||
// controller can activate.
|
||||
const markdown = buildThreadMarkdown([
|
||||
message({
|
||||
role: "assistant",
|
||||
text: "GitHub links:\n\n```\nhttps://github.com/jasonkneen/openclicky\nhttps://github.com/unn-Known1/clickyX\n```",
|
||||
}),
|
||||
]);
|
||||
|
||||
expect(markdown).not.toContain("```");
|
||||
expect(markdown).toContain(
|
||||
"[https://github.com/jasonkneen/openclicky](https://github.com/jasonkneen/openclicky)",
|
||||
);
|
||||
});
|
||||
|
||||
it("links a lone URL the model put in inline code", () => {
|
||||
const markdown = buildThreadMarkdown([
|
||||
message({ role: "assistant", text: "`https://example.com`" }),
|
||||
]);
|
||||
|
||||
expect(markdown).not.toContain("```");
|
||||
expect(markdown).toContain("[https://example.com](https://example.com)");
|
||||
});
|
||||
|
||||
it("lists tools on one quiet line, flagging failures", () => {
|
||||
const markdown = buildThreadMarkdown([
|
||||
message({
|
||||
@@ -212,7 +241,17 @@ describe("buildThreadMarkdown", () => {
|
||||
|
||||
describe("sessionToPlainText", () => {
|
||||
function turn(prompt: string, response: string, error?: string): ChatTurn {
|
||||
return { id: prompt, prompt, response, thinking: "", model: "m", createdAt: 0, toolCalls: [], error };
|
||||
return {
|
||||
id: prompt,
|
||||
prompt,
|
||||
response,
|
||||
thinking: "",
|
||||
model: "m",
|
||||
createdAt: 0,
|
||||
toolCalls: [],
|
||||
attachments: [],
|
||||
error,
|
||||
};
|
||||
}
|
||||
|
||||
it("returns an empty string for an empty conversation", () => {
|
||||
@@ -247,3 +286,69 @@ describe("sessionToPlainText", () => {
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("attachments in the transcript", () => {
|
||||
const screenshot = { path: "/tmp/snip.png", mime: "image/png" };
|
||||
|
||||
it("shows the attachment instead of just naming it", () => {
|
||||
// Vicinae resolves a local file through `ImageURL::local`, so the capture is
|
||||
// rendered rather than described. `MdImage.qml` caps the height at 200px, so
|
||||
// it is a thumbnail, and the alt text is what is left if the file is gone.
|
||||
const markdown = buildThreadMarkdown([
|
||||
message({ role: "user", text: "what is this?", attachments: [screenshot] }),
|
||||
]);
|
||||
|
||||
expect(markdown).toContain("");
|
||||
});
|
||||
|
||||
it("keeps the question above its image", () => {
|
||||
const markdown = buildThreadMarkdown([
|
||||
message({ role: "user", text: "what is this?", attachments: [screenshot] }),
|
||||
]);
|
||||
|
||||
expect(markdown.indexOf("> what is this?")).toBeLessThan(markdown.indexOf("!["));
|
||||
});
|
||||
|
||||
it("renders every attachment on a turn", () => {
|
||||
const markdown = buildThreadMarkdown([
|
||||
message({
|
||||
role: "user",
|
||||
text: "compare",
|
||||
attachments: [screenshot, { path: "/tmp/second.png", mime: "image/png" }],
|
||||
}),
|
||||
]);
|
||||
|
||||
expect(markdown).toContain("file:///tmp/snip.png");
|
||||
expect(markdown).toContain("file:///tmp/second.png");
|
||||
});
|
||||
|
||||
it("never puts an image on an answer, which has no attachments", () => {
|
||||
const markdown = buildThreadMarkdown([
|
||||
message({ role: "assistant", text: "a cat", attachments: [screenshot] }),
|
||||
]);
|
||||
|
||||
expect(markdown).not.toContain("![");
|
||||
});
|
||||
|
||||
it("says nothing about attachments when there are none", () => {
|
||||
const markdown = buildThreadMarkdown([message({ role: "user", text: "q" })]);
|
||||
|
||||
expect(markdown).not.toContain("Screenshot");
|
||||
});
|
||||
|
||||
it("notes the screenshot when a conversation is copied as plain text", () => {
|
||||
const question = {
|
||||
id: "t1",
|
||||
prompt: "what is this?",
|
||||
response: "a cat",
|
||||
thinking: "",
|
||||
model: "m",
|
||||
createdAt: 0,
|
||||
toolCalls: [],
|
||||
attachments: [screenshot],
|
||||
};
|
||||
const text = sessionToPlainText([question]);
|
||||
|
||||
expect(text).toContain("_(screenshot attached)_");
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { normalizeHost, parseNdjsonStream } from "../lib/ollama";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { modelCapabilities, normalizeHost, parseNdjsonStream, supportsVision } from "../lib/ollama";
|
||||
|
||||
/**
|
||||
* The NDJSON reader is the part of the client most likely to break in a way
|
||||
@@ -84,3 +84,70 @@ describe("parseNdjsonStream", () => {
|
||||
expect(await collect<unknown>([])).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("supportsVision", () => {
|
||||
it("reads the capability Ollama reports for image input", () => {
|
||||
expect(supportsVision(["completion", "tools", "vision"])).toBe(true);
|
||||
});
|
||||
|
||||
it("is false for a text-only model", () => {
|
||||
expect(supportsVision(["completion", "tools"])).toBe(false);
|
||||
expect(supportsVision([])).toBe(false);
|
||||
});
|
||||
|
||||
it("does not match a capability that merely contains the word", () => {
|
||||
expect(supportsVision(["visions"])).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("modelCapabilities", () => {
|
||||
const fetchMock = vi.fn();
|
||||
|
||||
beforeEach(() => {
|
||||
fetchMock.mockReset();
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
|
||||
it("posts to /api/show and returns the capability list", async () => {
|
||||
fetchMock.mockResolvedValueOnce(
|
||||
new Response(JSON.stringify({ capabilities: ["completion", "vision"] }), { status: 200 }),
|
||||
);
|
||||
|
||||
await expect(
|
||||
modelCapabilities({ host: "https://ollama.com", apiKey: "k", model: "gemma4" }),
|
||||
).resolves.toEqual(["completion", "vision"]);
|
||||
|
||||
const [url, init] = fetchMock.mock.calls[0];
|
||||
expect(url).toBe("https://ollama.com/api/show");
|
||||
expect(init.headers.Authorization).toBe("Bearer k");
|
||||
expect(JSON.parse(init.body)).toEqual({ model: "gemma4" });
|
||||
});
|
||||
|
||||
it("returns nothing when a model reports no capabilities", async () => {
|
||||
fetchMock.mockResolvedValueOnce(new Response(JSON.stringify({}), { status: 200 }));
|
||||
|
||||
await expect(
|
||||
modelCapabilities({ host: "https://ollama.com", apiKey: "k", model: "gemma4" }),
|
||||
).resolves.toEqual([]);
|
||||
});
|
||||
|
||||
it("turns a 401 into the key hint rather than a bare status code", async () => {
|
||||
fetchMock.mockResolvedValueOnce(new Response("unauthorized", { status: 401 }));
|
||||
|
||||
await expect(
|
||||
modelCapabilities({ host: "https://ollama.com", apiKey: "bad", model: "gemma4" }),
|
||||
).rejects.toMatchObject({ status: 401, hint: expect.stringContaining("ollama.com/settings/keys") });
|
||||
});
|
||||
|
||||
it("normalises the host so the path does not double up a slash", async () => {
|
||||
fetchMock.mockResolvedValueOnce(new Response(JSON.stringify({ capabilities: [] }), { status: 200 }));
|
||||
|
||||
await modelCapabilities({ host: "http://localhost:11434/", apiKey: "", model: "gemma4" });
|
||||
|
||||
expect(fetchMock.mock.calls[0][0]).toBe("http://localhost:11434/api/show");
|
||||
});
|
||||
});
|
||||
|
||||
@@ -53,6 +53,26 @@ describe("resolvePreferences", () => {
|
||||
expect(resolved.showReasoning).toBe(true);
|
||||
expect(resolved.think).toBeUndefined();
|
||||
expect(resolved.resumeTimeoutMs).toBe(HOUR);
|
||||
// An empty string rather than undefined, so callers never branch on it.
|
||||
expect(resolved.place).toBe("");
|
||||
});
|
||||
|
||||
it("leaves IP geolocation off unless it was switched on", () => {
|
||||
// The one setting that sends something about the user to a third party is
|
||||
// opted into, never defaulted into.
|
||||
expect(resolvePreferences({}).ipGeolocation).toBe(false);
|
||||
expect(resolvePreferences({ ipGeolocation: true }).ipGeolocation).toBe(true);
|
||||
expect(resolvePreferences({ ipGeolocation: false }).ipGeolocation).toBe(false);
|
||||
});
|
||||
|
||||
it("keeps the configured location, trimmed", () => {
|
||||
expect(resolvePreferences({ location: " Cebu City, Philippines " }).place).toBe(
|
||||
"Cebu City, Philippines",
|
||||
);
|
||||
});
|
||||
|
||||
it("treats a whitespace-only location as unset", () => {
|
||||
expect(resolvePreferences({ location: " " }).place).toBe("");
|
||||
});
|
||||
|
||||
it("honours the display toggles being switched off", () => {
|
||||
|
||||
@@ -24,6 +24,7 @@ function turn(index: number, response = `answer ${index}`): ChatTurn {
|
||||
model: "test-model",
|
||||
createdAt: 1_000 + index,
|
||||
toolCalls: [],
|
||||
attachments: [],
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { readSnipContext, snipDirectory } from "../lib/snip";
|
||||
|
||||
/**
|
||||
* The snip handoff crosses a command boundary as a serialized launch context, so
|
||||
* every field arrives as unknown. This is the boundary where a wrong type would
|
||||
* otherwise reach the file system as a path.
|
||||
*/
|
||||
|
||||
describe("readSnipContext", () => {
|
||||
it("reads the fields the snip command writes", () => {
|
||||
expect(
|
||||
readSnipContext({
|
||||
snipPath: "/tmp/snip.png",
|
||||
snipError: "no tool",
|
||||
sessionId: "abc123",
|
||||
}),
|
||||
).toEqual({ snipPath: "/tmp/snip.png", snipError: "no tool", sessionId: "abc123" });
|
||||
});
|
||||
|
||||
it("returns nothing for a launch with no context at all", () => {
|
||||
expect(readSnipContext(undefined)).toEqual({});
|
||||
expect(readSnipContext(null)).toEqual({});
|
||||
});
|
||||
|
||||
it("ignores a context that is not an object", () => {
|
||||
expect(readSnipContext("snip.png")).toEqual({});
|
||||
expect(readSnipContext(42)).toEqual({});
|
||||
expect(readSnipContext(["/tmp/snip.png"])).toEqual({});
|
||||
});
|
||||
|
||||
it("drops a value that is not a string rather than passing it on", () => {
|
||||
// A number or object here would reach `readAttachment` as a path.
|
||||
expect(readSnipContext({ snipPath: 42, sessionId: { id: "x" } })).toEqual({});
|
||||
});
|
||||
|
||||
it("treats an empty string as absent", () => {
|
||||
expect(readSnipContext({ snipPath: "", sessionId: "" })).toEqual({});
|
||||
});
|
||||
|
||||
it("ignores unknown keys, so a stray field cannot widen the contract", () => {
|
||||
expect(readSnipContext({ snipPath: "/tmp/snip.png", somethingElse: "/etc/passwd" })).toEqual({
|
||||
snipPath: "/tmp/snip.png",
|
||||
});
|
||||
});
|
||||
|
||||
it("reads a payload that only names a conversation, which is a cancelled snip", () => {
|
||||
expect(readSnipContext({ sessionId: "abc123" })).toEqual({ sessionId: "abc123" });
|
||||
});
|
||||
});
|
||||
|
||||
describe("snipDirectory", () => {
|
||||
it("keeps snips under the extension's support directory, not a temp directory", () => {
|
||||
// A stored conversation holds the path, so it has to survive a reboot.
|
||||
expect(snipDirectory()).toBe("/tmp/lam-agent-support/snips");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,203 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { isHandoff, resolveStartup, type StartupPorts } from "../lib/startup";
|
||||
import type { PendingAttachment, Session, SessionSummary } from "../lib/types";
|
||||
|
||||
/**
|
||||
* The chat view's startup is the part that kept breaking: three asynchronous
|
||||
* paths (a pushed session, a handed-over capture, an idle resume) that all have
|
||||
* to end in a rendered chat. Two of those bugs shipped, so the decision tree is
|
||||
* tested here rather than discovered from a screenshot of a spinner.
|
||||
*/
|
||||
|
||||
const NOW = 1_000_000;
|
||||
const HOUR = 3_600_000;
|
||||
|
||||
function session(id: string, updatedAt: number): Session {
|
||||
return { id, title: id, model: "m", createdAt: updatedAt, updatedAt, turns: [] };
|
||||
}
|
||||
|
||||
function summary(id: string, updatedAt: number): SessionSummary {
|
||||
return { id, title: id, model: "m", createdAt: updatedAt, updatedAt, turnCount: 0, preview: "" };
|
||||
}
|
||||
|
||||
function attachment(path: string): PendingAttachment {
|
||||
return { path, mime: "image/png", base64: "AAAA" };
|
||||
}
|
||||
|
||||
/** Ports that record what was asked for, so "did not resume" is checkable. */
|
||||
function ports(overrides: Partial<StartupPorts> = {}) {
|
||||
const loadSession = vi.fn(async (id: string) => session(id, NOW));
|
||||
const listSessions = vi.fn(async () => [summary("latest", NOW)]);
|
||||
const readAttachment = vi.fn(async (path: string) => attachment(path));
|
||||
|
||||
return {
|
||||
loadSession,
|
||||
listSessions,
|
||||
readAttachment,
|
||||
...overrides,
|
||||
} satisfies StartupPorts;
|
||||
}
|
||||
|
||||
function input(overrides: Partial<Parameters<typeof resolveStartup>[0]> = {}) {
|
||||
return {
|
||||
hasInitialSession: false,
|
||||
now: NOW,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe("isHandoff", () => {
|
||||
it("is true when a file was handed over", () => {
|
||||
expect(isHandoff({ initialAttachmentPath: "/tmp/snip.png" })).toBe(true);
|
||||
});
|
||||
|
||||
it("is true when a conversation was named", () => {
|
||||
expect(isHandoff({ initialSessionId: "abc" })).toBe(true);
|
||||
});
|
||||
|
||||
it("is false on an ordinary open", () => {
|
||||
expect(isHandoff({})).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("resolveStartup", () => {
|
||||
describe("a snip hand-off", () => {
|
||||
it("reads the attachment and restores the named conversation", async () => {
|
||||
const p = ports();
|
||||
|
||||
const result = await resolveStartup(
|
||||
input({ initialSessionId: "abc", initialAttachmentPath: "/tmp/snip.png" }),
|
||||
p,
|
||||
);
|
||||
|
||||
expect(result.session?.id).toBe("abc");
|
||||
expect(result.attachment?.path).toBe("/tmp/snip.png");
|
||||
});
|
||||
|
||||
it("does not fall through to the idle resume", async () => {
|
||||
// The bug this file exists for: a hand-off has to finish startup even
|
||||
// though it took the resume path off the table. Guarding the resume with
|
||||
// "and there was no hand-off" while forgetting to clear the readiness flag
|
||||
// left the view on its loading row forever.
|
||||
const p = ports();
|
||||
|
||||
await resolveStartup(input({ initialAttachmentPath: "/tmp/snip.png" }), p);
|
||||
|
||||
expect(p.listSessions).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("does not attach the most recent conversation's session", async () => {
|
||||
const p = ports();
|
||||
|
||||
const result = await resolveStartup(input({ initialAttachmentPath: "/tmp/snip.png" }), p);
|
||||
|
||||
expect(result.session).toBeUndefined();
|
||||
});
|
||||
|
||||
it("still resolves when the named conversation has been deleted", async () => {
|
||||
// A snip started from a conversation someone has since deleted should mean
|
||||
// a fresh chat with the image, not a spinner or a crash.
|
||||
const p = ports({ loadSession: vi.fn(async () => undefined) });
|
||||
|
||||
const result = await resolveStartup(
|
||||
input({ initialSessionId: "gone", initialAttachmentPath: "/tmp/snip.png" }),
|
||||
p,
|
||||
);
|
||||
|
||||
expect(result.session).toBeUndefined();
|
||||
expect(result.attachment?.path).toBe("/tmp/snip.png");
|
||||
});
|
||||
|
||||
it("reports an unreadable attachment without refusing to open", async () => {
|
||||
const p = ports({
|
||||
readAttachment: vi.fn(async () => {
|
||||
throw new Error("ENOENT: no such file");
|
||||
}),
|
||||
});
|
||||
|
||||
const result = await resolveStartup(input({ initialAttachmentPath: "/tmp/gone.png" }), p);
|
||||
|
||||
expect(result.attachment).toBeUndefined();
|
||||
expect(result.attachmentError).toContain("ENOENT");
|
||||
});
|
||||
|
||||
it("survives a storage failure while loading the named conversation", async () => {
|
||||
const p = ports({
|
||||
loadSession: vi.fn(async () => {
|
||||
throw new Error("storage exploded");
|
||||
}),
|
||||
});
|
||||
|
||||
const result = await resolveStartup(input({ initialSessionId: "abc" }), p);
|
||||
|
||||
expect(result.session).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("an ordinary open", () => {
|
||||
it("resumes the most recent conversation when it is fresh", async () => {
|
||||
const p = ports({ listSessions: vi.fn(async () => [summary("latest", NOW - 60_000)]) });
|
||||
|
||||
const result = await resolveStartup(input({ resumeTimeoutMs: HOUR }), p);
|
||||
|
||||
expect(result.session?.id).toBe("latest");
|
||||
expect(p.loadSession).toHaveBeenCalledWith("latest");
|
||||
});
|
||||
|
||||
it("starts fresh when the newest conversation is older than the window", async () => {
|
||||
const p = ports({ listSessions: vi.fn(async () => [summary("latest", NOW - 2 * HOUR)]) });
|
||||
|
||||
const result = await resolveStartup(input({ resumeTimeoutMs: HOUR }), p);
|
||||
|
||||
expect(result.session).toBeUndefined();
|
||||
expect(p.loadSession).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("treats exactly the window as fresh", async () => {
|
||||
const p = ports({ listSessions: vi.fn(async () => [summary("latest", NOW - HOUR)]) });
|
||||
|
||||
const result = await resolveStartup(input({ resumeTimeoutMs: HOUR }), p);
|
||||
|
||||
expect(result.session?.id).toBe("latest");
|
||||
});
|
||||
|
||||
it("always resumes when the preference is 'never time out'", async () => {
|
||||
const p = ports({ listSessions: vi.fn(async () => [summary("latest", NOW - 400 * HOUR)]) });
|
||||
|
||||
const result = await resolveStartup(input({ resumeTimeoutMs: undefined }), p);
|
||||
|
||||
expect(result.session?.id).toBe("latest");
|
||||
});
|
||||
|
||||
it("starts fresh when there are no conversations at all", async () => {
|
||||
const p = ports({ listSessions: vi.fn(async () => []) });
|
||||
|
||||
const result = await resolveStartup(input({ resumeTimeoutMs: HOUR }), p);
|
||||
|
||||
expect(result).toEqual({});
|
||||
expect(p.loadSession).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("starts fresh rather than failing when the index cannot be read", async () => {
|
||||
const p = ports({
|
||||
listSessions: vi.fn(async () => {
|
||||
throw new Error("corrupt index");
|
||||
}),
|
||||
});
|
||||
|
||||
const result = await resolveStartup(input({ resumeTimeoutMs: HOUR }), p);
|
||||
|
||||
expect(result).toEqual({});
|
||||
});
|
||||
|
||||
it("does not resume when a session was pushed in by hand", async () => {
|
||||
// The sessions command already knows which conversation to show.
|
||||
const p = ports();
|
||||
|
||||
await resolveStartup(input({ hasInitialSession: true }), p);
|
||||
|
||||
expect(p.listSessions).not.toHaveBeenCalled();
|
||||
expect(p.loadSession).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -13,6 +13,7 @@ function turn(id: string, prompt: string, response: string, overrides: Partial<C
|
||||
model: "m",
|
||||
createdAt: 100,
|
||||
toolCalls: [],
|
||||
attachments: [],
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
+128
-2
@@ -225,13 +225,139 @@ describe("local tools", () => {
|
||||
expect(result.content).toContain("Current date and time:");
|
||||
});
|
||||
|
||||
it("describes location as a region rather than an address", async () => {
|
||||
it("describes location as a region rather than an address when nothing is configured", async () => {
|
||||
const result = await executeTool("get_location", {}, context);
|
||||
|
||||
expect(result.isError).toBe(false);
|
||||
expect(result.content).toContain("Timezone:");
|
||||
expect(result.content).toContain("Locale:");
|
||||
expect(result.content).toContain("not a precise address");
|
||||
expect(result.content).toContain("no Location is set in preferences");
|
||||
expect(result.content).toContain("Ask which city they are in");
|
||||
});
|
||||
|
||||
it("never claims a city derived from the timezone name", async () => {
|
||||
// `Asia/Manila` is one zone covering the whole Philippines, so reporting
|
||||
// "City: Manila" was wrong for anyone in Cebu and confidently so.
|
||||
const result = await executeTool("get_location", {}, context);
|
||||
|
||||
expect(result.content).not.toMatch(/^City:/m);
|
||||
expect(result.content).toContain("The timezone area is the timezone's own name, not the user's city.");
|
||||
});
|
||||
|
||||
it("puts the configured place first and says to prefer it", async () => {
|
||||
const result = await executeTool("get_location", {}, {
|
||||
...context,
|
||||
place: "Cebu City, Philippines",
|
||||
});
|
||||
|
||||
expect(result.content).toContain("Place: Cebu City, Philippines");
|
||||
expect(result.content).toContain("prefer it");
|
||||
expect(result.content).not.toContain("no Location is set");
|
||||
expect(result.content.indexOf("Place:")).toBeLessThan(result.content.indexOf("Timezone:"));
|
||||
});
|
||||
|
||||
it("still reports the timezone alongside a configured place, for local time questions", async () => {
|
||||
const result = await executeTool("get_location", {}, { ...context, place: "Cebu City" });
|
||||
|
||||
expect(result.content).toContain("Timezone:");
|
||||
expect(result.content).toContain("UTC offset:");
|
||||
});
|
||||
});
|
||||
|
||||
describe("get_location with IP geolocation", () => {
|
||||
const resolved = {
|
||||
status: "ok" as const,
|
||||
place: {
|
||||
place: "Cebu City, Central Visayas, Philippines",
|
||||
city: "Cebu City",
|
||||
region: "Central Visayas",
|
||||
country: "Philippines",
|
||||
timezone: "Asia/Manila",
|
||||
source: "ipwho.is",
|
||||
fetchedAt: 1_000,
|
||||
},
|
||||
};
|
||||
|
||||
it("reports the resolved place and says how approximate it is", async () => {
|
||||
const ipPlace = vi.fn(async () => resolved);
|
||||
|
||||
const result = await executeTool("get_location", {}, { ...context, ipPlace });
|
||||
|
||||
expect(result.isError).toBe(false);
|
||||
expect(result.content).toContain("Place: Cebu City, Central Visayas, Philippines");
|
||||
expect(result.content).toContain("rough guess");
|
||||
expect(result.content).toContain("ipwho.is");
|
||||
});
|
||||
|
||||
it("does not look the address up at all when the user has said where they are", async () => {
|
||||
// The preference is both more accurate and free, so sending the IP anyway
|
||||
// would be a privacy cost for nothing.
|
||||
const ipPlace = vi.fn(async () => resolved);
|
||||
|
||||
const result = await executeTool("get_location", {}, {
|
||||
...context,
|
||||
place: "Cebu City, Philippines",
|
||||
ipPlace,
|
||||
});
|
||||
|
||||
expect(ipPlace).not.toHaveBeenCalled();
|
||||
expect(result.content).toContain("Place: Cebu City, Philippines");
|
||||
expect(result.content).not.toContain("ipwho.is");
|
||||
});
|
||||
|
||||
it("surfaces a failed lookup instead of quietly serving the region answer", async () => {
|
||||
const ipPlace = vi.fn(async () => ({ status: "failed" as const, message: "ipwho.is answered HTTP 429" }));
|
||||
|
||||
const result = await executeTool("get_location", {}, { ...context, ipPlace });
|
||||
|
||||
expect(result.content).toContain("IP geolocation is switched on but the lookup failed");
|
||||
expect(result.content).toContain("HTTP 429");
|
||||
// The region answer is still there, so the tool stays useful.
|
||||
expect(result.content).toContain("Timezone:");
|
||||
});
|
||||
|
||||
it("does not blame an unset preference when the lookup itself failed", async () => {
|
||||
// These are different problems and send the user to different places. The
|
||||
// original wording said "no Location is set in preferences" for both, which
|
||||
// is exactly what made a broken lookup look like a missing setting.
|
||||
const ipPlace = vi.fn(async () => ({ status: "failed" as const, message: "ipwho.is answered HTTP 429" }));
|
||||
|
||||
const result = await executeTool("get_location", {}, { ...context, ipPlace });
|
||||
|
||||
expect(result.content).not.toContain("no Location is set in preferences");
|
||||
});
|
||||
|
||||
it("tells the model to pass a failed lookup on, since it is the only channel to the user", async () => {
|
||||
const ipPlace = vi.fn(async () => ({ status: "failed" as const, message: "offline" }));
|
||||
|
||||
const result = await executeTool("get_location", {}, { ...context, ipPlace });
|
||||
|
||||
expect(result.content).toContain("Tell the user");
|
||||
expect(result.content).toContain("Setting Location in preferences avoids the lookup");
|
||||
});
|
||||
|
||||
it("flags an IP timezone that disagrees with the clock, which usually means a VPN", async () => {
|
||||
const ipPlace = vi.fn(async () => ({
|
||||
status: "ok" as const,
|
||||
place: { ...resolved.place, timezone: "America/New_York" },
|
||||
}));
|
||||
|
||||
const result = await executeTool("get_location", {}, { ...context, ipPlace });
|
||||
|
||||
expect(result.content).toContain("differs from the device timezone");
|
||||
});
|
||||
|
||||
it("says nothing about a mismatch when the timezones agree", async () => {
|
||||
const ipPlace = vi.fn(async () => resolved);
|
||||
const result = await executeTool("get_location", {}, { ...context, ipPlace });
|
||||
|
||||
// The test host's own zone varies, so this asserts the absence of the note
|
||||
// only for the case where the provider echoed the same value.
|
||||
if (result.content.includes("IP-reported timezone")) {
|
||||
expect(result.content).toContain("differs");
|
||||
} else {
|
||||
expect(result.content).not.toContain("IP-reported timezone");
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { __resetLocalStorage } from "../__mocks__/@vicinae/api";
|
||||
import { forgetVisionSupport, visionSupport } from "../lib/vision";
|
||||
|
||||
/**
|
||||
* The capability check decides whether a snip feature is offered at all, so the
|
||||
* cases that matter are the ones where the answer is not a clean yes or no: a
|
||||
* dead network must not be mistaken for "this model cannot see".
|
||||
*/
|
||||
|
||||
const fetchMock = vi.fn();
|
||||
|
||||
function showResponse(capabilities: string[]): Response {
|
||||
return new Response(JSON.stringify({ capabilities }), {
|
||||
status: 200,
|
||||
headers: { "Content-Type": "application/json" },
|
||||
});
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
__resetLocalStorage();
|
||||
fetchMock.mockReset();
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
});
|
||||
|
||||
const base = { host: "https://ollama.com", apiKey: "k", model: "gemma4" };
|
||||
|
||||
describe("visionSupport", () => {
|
||||
it("reports yes when Ollama lists the vision capability", async () => {
|
||||
fetchMock.mockResolvedValueOnce(showResponse(["completion", "vision"]));
|
||||
await expect(visionSupport(base)).resolves.toBe("yes");
|
||||
});
|
||||
|
||||
it("reports no for a text-only model", async () => {
|
||||
fetchMock.mockResolvedValueOnce(showResponse(["completion", "tools"]));
|
||||
await expect(visionSupport(base)).resolves.toBe("no");
|
||||
});
|
||||
|
||||
it("asks /api/show for the model", async () => {
|
||||
fetchMock.mockResolvedValueOnce(showResponse(["vision"]));
|
||||
await visionSupport(base);
|
||||
|
||||
const [url, init] = fetchMock.mock.calls[0];
|
||||
expect(url).toBe("https://ollama.com/api/show");
|
||||
expect(JSON.parse(init.body)).toEqual({ model: "gemma4" });
|
||||
});
|
||||
|
||||
it("serves a repeat question from the cache", async () => {
|
||||
fetchMock.mockResolvedValueOnce(showResponse(["vision"]));
|
||||
await visionSupport(base);
|
||||
await expect(visionSupport(base)).resolves.toBe("yes");
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("caches a negative too, so a text model is not re-checked on every mount", async () => {
|
||||
fetchMock.mockResolvedValueOnce(showResponse(["tools"]));
|
||||
await visionSupport(base);
|
||||
await expect(visionSupport(base)).resolves.toBe("no");
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("keys the cache per model, so switching model re-checks", async () => {
|
||||
fetchMock.mockResolvedValueOnce(showResponse(["vision"]));
|
||||
await visionSupport(base);
|
||||
|
||||
fetchMock.mockResolvedValueOnce(showResponse(["tools"]));
|
||||
await expect(visionSupport({ ...base, model: "gpt-oss:120b" })).resolves.toBe("no");
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("re-checks when explicitly asked to, even with a cache entry", async () => {
|
||||
fetchMock.mockResolvedValueOnce(showResponse(["tools"]));
|
||||
await visionSupport(base);
|
||||
|
||||
fetchMock.mockResolvedValueOnce(showResponse(["vision"]));
|
||||
await expect(visionSupport({ ...base, refresh: true })).resolves.toBe("yes");
|
||||
});
|
||||
|
||||
it("reports unknown rather than no when the request fails", async () => {
|
||||
// A dead network is not evidence that a model cannot see, and treating it as
|
||||
// such would hide the feature for a reason the user cannot see.
|
||||
fetchMock.mockRejectedValueOnce(new Error("offline"));
|
||||
await expect(visionSupport(base)).resolves.toBe("unknown");
|
||||
});
|
||||
|
||||
it("does not cache a failure", async () => {
|
||||
fetchMock.mockRejectedValueOnce(new Error("offline"));
|
||||
await visionSupport(base);
|
||||
|
||||
fetchMock.mockResolvedValueOnce(showResponse(["vision"]));
|
||||
await expect(visionSupport(base)).resolves.toBe("yes");
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("reports a rejected key as unknown rather than crashing", async () => {
|
||||
fetchMock.mockResolvedValueOnce(new Response("unauthorized", { status: 401 }));
|
||||
await expect(visionSupport(base)).resolves.toBe("unknown");
|
||||
});
|
||||
});
|
||||
|
||||
describe("forgetVisionSupport", () => {
|
||||
it("drops every cached answer, so a re-pulled model is not pinned to no", async () => {
|
||||
fetchMock.mockResolvedValueOnce(showResponse(["vision"]));
|
||||
await visionSupport(base);
|
||||
|
||||
await forgetVisionSupport();
|
||||
|
||||
fetchMock.mockResolvedValueOnce(showResponse(["vision"]));
|
||||
await visionSupport(base);
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
});
|
||||
+14
-1
@@ -1,5 +1,6 @@
|
||||
import { type LaunchProps } from "@vicinae/api";
|
||||
import { ChatView } from "./lib/components/ChatView";
|
||||
import { readSnipContext } from "./lib/snip";
|
||||
|
||||
/**
|
||||
* Entrypoint for the "chat" command.
|
||||
@@ -11,7 +12,19 @@ import { ChatView } from "./lib/components/ChatView";
|
||||
* launcher-and-start-typing behaviour. `fallbackText` is populated when the
|
||||
* command is opened through a deeplink carrying text, so
|
||||
* `vicinae://launch/lam-agent/chat?fallbackText=...` arrives pre-filled.
|
||||
*
|
||||
* It is also the command the `snip` command returns through, carrying a captured
|
||||
* file and, when the snip came from an open conversation, that conversation's id.
|
||||
*/
|
||||
export default function ChatCommand(props: LaunchProps) {
|
||||
return <ChatView initialPrompt={props.fallbackText} />;
|
||||
const snip = readSnipContext(props.launchContext);
|
||||
|
||||
return (
|
||||
<ChatView
|
||||
initialPrompt={props.fallbackText}
|
||||
initialSessionId={snip.sessionId}
|
||||
initialAttachmentPath={snip.snipPath}
|
||||
launchNote={snip.snipError ? `**Could not take a screenshot:** ${snip.snipError}` : undefined}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
+73
-6
@@ -59,10 +59,36 @@ function normalizeToolCall(call: ToolCall): { name: string; arguments: Record<st
|
||||
return { name, arguments: args as Record<string, unknown> };
|
||||
}
|
||||
|
||||
/**
|
||||
* The last word, when the tool budget runs out.
|
||||
*
|
||||
* The loop used to end by throwing away every preamble the model had written and
|
||||
* returning a notice about itself, so a long research question produced no answer
|
||||
* at all. One closing round, with the tools taken away, turns that into an answer
|
||||
* built from whatever was already gathered, with the gaps named. The instruction
|
||||
* asks for the gaps because an answer that knows what it is missing is worth more
|
||||
* than a confident half of one.
|
||||
*/
|
||||
const CLOSING_INSTRUCTION =
|
||||
"You have reached the limit on tool calls for this turn, so no more tools are available. Answer now using what you have already gathered. If something is missing, say plainly which parts you could not confirm.";
|
||||
|
||||
/** Only reachable if the closing round comes back empty too. */
|
||||
function exhaustedNotice(rounds: number): string {
|
||||
return `_Ran out of tool calls after ${rounds} rounds and could not write an answer. Try asking something narrower._`;
|
||||
}
|
||||
|
||||
export async function runAgent(options: AgentOptions): Promise<AgentResult> {
|
||||
const conversation: ApiMessage[] = [...options.messages];
|
||||
const toolCalls: ToolCallRecord[] = [];
|
||||
let thinking = "";
|
||||
/**
|
||||
* The most recent preamble the model wrote.
|
||||
*
|
||||
* Not normally shown, because a preamble is what it says before finding out
|
||||
* anything. Kept anyway, so that a turn which runs out of budget still has
|
||||
* something to fall back on if the closing round produces nothing.
|
||||
*/
|
||||
let lastPreamble = "";
|
||||
|
||||
for (let iteration = 0; iteration < MAX_TOOL_ITERATIONS; iteration++) {
|
||||
let content = "";
|
||||
@@ -91,20 +117,25 @@ export async function runAgent(options: AgentOptions): Promise<AgentResult> {
|
||||
|
||||
thinking += streamedThinking;
|
||||
|
||||
// No tool calls means this was the model's answer. Earlier iterations only
|
||||
// produced a preamble, which is intentionally discarded rather than shown
|
||||
// as a half-answer.
|
||||
// No tool calls means this was the model's answer, rather than something it
|
||||
// said on the way to finding out.
|
||||
if (!pending || pending.length === 0) {
|
||||
return { content, thinking, toolCalls };
|
||||
}
|
||||
|
||||
if (content.trim()) lastPreamble = content;
|
||||
|
||||
conversation.push({ role: "assistant", content, tool_calls: pending });
|
||||
|
||||
for (const rawCall of pending) {
|
||||
const { name, arguments: args } = normalizeToolCall(rawCall);
|
||||
|
||||
if (!name) {
|
||||
conversation.push({ role: "tool", content: "Malformed tool call: missing function name.", tool_name: "unknown" });
|
||||
conversation.push({
|
||||
role: "tool",
|
||||
content: "Malformed tool call: missing function name.",
|
||||
tool_name: "unknown",
|
||||
});
|
||||
toolCalls.push({ name: "unknown", arguments: {}, result: "Malformed tool call.", isError: true });
|
||||
continue;
|
||||
}
|
||||
@@ -114,11 +145,17 @@ export async function runAgent(options: AgentOptions): Promise<AgentResult> {
|
||||
const result = await executeTool(name, args, {
|
||||
host: options.toolContext.host,
|
||||
apiKey: options.toolContext.apiKey,
|
||||
place: options.toolContext.place,
|
||||
now: options.toolContext.now,
|
||||
signal: options.signal,
|
||||
});
|
||||
|
||||
const record: ToolCallRecord = { name, arguments: args, result: result.content, isError: result.isError };
|
||||
const record: ToolCallRecord = {
|
||||
name,
|
||||
arguments: args,
|
||||
result: result.content,
|
||||
isError: result.isError,
|
||||
};
|
||||
toolCalls.push(record);
|
||||
options.events?.onToolCall?.(record);
|
||||
|
||||
@@ -126,8 +163,38 @@ export async function runAgent(options: AgentOptions): Promise<AgentResult> {
|
||||
}
|
||||
}
|
||||
|
||||
// Budget spent, so the turn gets one closing round with the tools taken away.
|
||||
// Ending on an answer drawn from what was gathered beats ending on a notice
|
||||
// about the loop itself, and the model is the only one that can say what it
|
||||
// did and did not manage to confirm.
|
||||
options.events?.onStatus?.("Writing the answer");
|
||||
|
||||
conversation.push({ role: "user", content: CLOSING_INSTRUCTION });
|
||||
|
||||
let closing = "";
|
||||
|
||||
for await (const chunk of streamChat({
|
||||
host: options.host,
|
||||
apiKey: options.apiKey,
|
||||
model: options.model,
|
||||
messages: conversation,
|
||||
// Deliberately no tools: the answer has to be written from what is already here.
|
||||
tools: undefined,
|
||||
think: options.think,
|
||||
signal: options.signal,
|
||||
})) {
|
||||
if (chunk.thinking) {
|
||||
thinking += chunk.thinking;
|
||||
options.events?.onThinkingDelta?.(chunk.thinking);
|
||||
}
|
||||
if (chunk.content) {
|
||||
closing += chunk.content;
|
||||
options.events?.onContentDelta?.(chunk.content);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
content: `_Stopped after ${MAX_TOOL_ITERATIONS} rounds of tool calls without a final answer._`,
|
||||
content: closing.trim() || lastPreamble.trim() || exhaustedNotice(MAX_TOOL_ITERATIONS),
|
||||
thinking,
|
||||
toolCalls,
|
||||
};
|
||||
|
||||
@@ -0,0 +1,137 @@
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { extname } from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import type { Attachment, PendingAttachment } from "./types";
|
||||
|
||||
/**
|
||||
* Reading attachments off disk, and describing them to the renderer.
|
||||
*
|
||||
* Kept separate from `capture.ts` because the two answer different questions:
|
||||
* capture is about getting an image *onto* the disk, this is about getting it
|
||||
* back off again and telling Vicinae how to draw it.
|
||||
*
|
||||
* Imports no Vicinae module, so it unit tests with no runtime.
|
||||
*/
|
||||
|
||||
/** Extensions Ollama accepts. A snip is always PNG; the rest are here for a future picker. */
|
||||
const MIME_TYPES: Record<string, string> = {
|
||||
".png": "image/png",
|
||||
".jpg": "image/jpeg",
|
||||
".jpeg": "image/jpeg",
|
||||
".webp": "image/webp",
|
||||
".gif": "image/gif",
|
||||
};
|
||||
|
||||
/**
|
||||
* The box a preview is fitted into.
|
||||
*
|
||||
* The height is not a preference: `MdImage.qml` clamps rendered height to 200px
|
||||
* whatever is asked for (`Math.min(_rawH, 200, maxImageHeight)`), so a taller
|
||||
* request is silently ignored. The width is here to keep a very wide capture
|
||||
* from being scaled up past the detail pane it sits in.
|
||||
*/
|
||||
const PREVIEW_MAX_WIDTH = 400;
|
||||
const PREVIEW_MAX_HEIGHT = 200;
|
||||
|
||||
export function mimeForPath(path: string): string {
|
||||
return MIME_TYPES[extname(path).toLowerCase()] ?? "application/octet-stream";
|
||||
}
|
||||
|
||||
/** The eight bytes every PNG starts with. */
|
||||
const PNG_SIGNATURE = [0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a];
|
||||
|
||||
/**
|
||||
* Pixel dimensions from a PNG's `IHDR` chunk, or `undefined` for anything else.
|
||||
*
|
||||
* Only PNG is handled, because that is the only thing a snip produces and
|
||||
* guessing at other formats would be a lie. The header is 24 bytes, so this
|
||||
* costs nothing on top of the read that is already happening.
|
||||
*/
|
||||
export function pngSize(bytes: Buffer): { width: number; height: number } | undefined {
|
||||
if (bytes.length < 24) return undefined;
|
||||
if (!PNG_SIGNATURE.every((byte, index) => bytes[index] === byte)) return undefined;
|
||||
if (bytes.toString("ascii", 12, 16) !== "IHDR") return undefined;
|
||||
|
||||
const width = bytes.readUInt32BE(16);
|
||||
const height = bytes.readUInt32BE(20);
|
||||
|
||||
return width > 0 && height > 0 ? { width, height } : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* The exact box to ask the renderer for, or `undefined` when the size is unknown.
|
||||
*
|
||||
* This is the whole point of reading the header. With no hint, `MdImage.qml`
|
||||
* sizes its container to the pane width and 200px tall and then aspect-fits the
|
||||
* image inside it, so a wide screenshot sits in a letterbox with dead space
|
||||
* above and below it. Asking for a box that already matches the image's aspect
|
||||
* ratio leaves none.
|
||||
*
|
||||
* The scale is capped at 1 so a small capture is not blown up into a blurry one.
|
||||
*/
|
||||
export function previewBox(
|
||||
width: number | undefined,
|
||||
height: number | undefined,
|
||||
): { width: number; height: number } | undefined {
|
||||
if (!width || !height || width <= 0 || height <= 0) return undefined;
|
||||
|
||||
const scale = Math.min(1, PREVIEW_MAX_WIDTH / width, PREVIEW_MAX_HEIGHT / height);
|
||||
|
||||
return {
|
||||
width: Math.max(1, Math.round(width * scale)),
|
||||
height: Math.max(1, Math.round(height * scale)),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* A markdown image source for a local file.
|
||||
*
|
||||
* A `file://` URL rather than a bare path, because Vicinae's markdown renderer
|
||||
* only resolves a scheme it recognises: `imageProviderUrl` in
|
||||
* `markdown-model.cpp` routes `file://`, and a scheme-less string, through
|
||||
* `ImageURL::local`, and anything it does not recognise it returns as empty.
|
||||
* `pathToFileURL` does the escaping, so a capture under a directory with a space
|
||||
* in its name still resolves.
|
||||
*
|
||||
* The query is read from the URL before it is resolved to a path, so the size
|
||||
* hints travel here without ever reaching the file name. They are spelled the
|
||||
* way Raycast spelled them, which is what the parser looks for.
|
||||
*/
|
||||
export function imageSource(attachment: Pick<Attachment, "path" | "width" | "height">): string {
|
||||
const url = pathToFileURL(attachment.path);
|
||||
const box = previewBox(attachment.width, attachment.height);
|
||||
|
||||
if (box) {
|
||||
url.searchParams.set("raycast-width", String(box.width));
|
||||
url.searchParams.set("raycast-height", String(box.height));
|
||||
}
|
||||
|
||||
return url.href;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads an image into the shape a request needs.
|
||||
*
|
||||
* Ollama wants base64 with no `data:` URL prefix, which is why the payload is
|
||||
* built here rather than handed straight to the renderer. The dimensions are
|
||||
* kept for the same reason the path is: the transcript needs them long after
|
||||
* this call is over, and the preview layout depends on them.
|
||||
*/
|
||||
export async function readAttachment(path: string): Promise<PendingAttachment> {
|
||||
const bytes = await readFile(path);
|
||||
const size = pngSize(bytes);
|
||||
|
||||
return {
|
||||
path,
|
||||
mime: mimeForPath(path),
|
||||
base64: bytes.toString("base64"),
|
||||
width: size?.width,
|
||||
height: size?.height,
|
||||
};
|
||||
}
|
||||
|
||||
/** The persistable half of a pending attachment: everything except the base64 payload. */
|
||||
export function toStoredAttachment(attachment: PendingAttachment): Attachment {
|
||||
const { path, mime, width, height } = attachment;
|
||||
return { path, mime, width, height };
|
||||
}
|
||||
@@ -0,0 +1,346 @@
|
||||
import { execFile } from "node:child_process";
|
||||
import { mkdir, stat, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
|
||||
/**
|
||||
* Screen capture for "snip to chat".
|
||||
*
|
||||
* Vicinae has no screenshot API. There is no capture call anywhere in
|
||||
* `@vicinae/api`, and `Clipboard` moves text, html, urls and a file path and
|
||||
* nothing binary, so a bitmap can never be read back *out* of the clipboard
|
||||
* either. The only way to get a region of the screen is to shell out.
|
||||
*
|
||||
* That is acceptable here because Vicinae extensions run under Node ("all
|
||||
* standard Node.js APIs are available"), which is why `node:child_process` is
|
||||
* not a workaround so much as the supported route. The important consequence is
|
||||
* that the capture is *owned* rather than borrowed: a tool writes a PNG to a
|
||||
* path this module picks, and that path is what the chat stores. Nothing has to
|
||||
* guess what some other application left on the clipboard.
|
||||
*
|
||||
* Every function here takes its environment as an argument and imports no
|
||||
* Vicinae module, so the selection logic unit tests with no runtime and no
|
||||
* installed tools.
|
||||
*/
|
||||
|
||||
/** A PNG piped through stdout (flameshot) is easily several megabytes. */
|
||||
const MAX_CAPTURE_BYTES = 64 * 1024 * 1024;
|
||||
|
||||
/** Long enough to pick a region unhurried, short enough that a wedged tool cannot hang the view forever. */
|
||||
const CAPTURE_TIMEOUT_MS = 5 * 60 * 1000;
|
||||
|
||||
/**
|
||||
* What macOS says when Screen Recording permission is missing.
|
||||
*
|
||||
* This one pattern is why the module reads stderr at all. The message is the
|
||||
* *only* difference between "the system refused to let us look" and "the user
|
||||
* pressed Escape": both produce no file, and `screencapture -i` has been
|
||||
* reported to exit `0` in the refusal case. Treating the refusal as a cancel
|
||||
* leaves the user with a feature that silently does nothing.
|
||||
*/
|
||||
const SCREEN_RECORDING_DENIED = /could not create image from (rect|display|window)/i;
|
||||
|
||||
export interface RunResult {
|
||||
stdout: Buffer;
|
||||
stderr: string;
|
||||
}
|
||||
|
||||
/** Runs a command. Injected so tests never spawn anything. */
|
||||
export type RunCommand = (
|
||||
tool: string,
|
||||
args: string[],
|
||||
options?: { signal?: AbortSignal },
|
||||
) => Promise<RunResult>;
|
||||
|
||||
export interface CaptureBackend {
|
||||
/** Shown to the user when a capture fails, so they know what to install. */
|
||||
name: string;
|
||||
/** Every one of these has to resolve on `PATH` for the backend to be usable. */
|
||||
requires: string[];
|
||||
/** Produces the image at `output`, or resolves without one when the user cancels. */
|
||||
capture: (context: CaptureContext) => Promise<void>;
|
||||
}
|
||||
|
||||
export interface CaptureContext {
|
||||
output: string;
|
||||
signal?: AbortSignal;
|
||||
run: RunCommand;
|
||||
}
|
||||
|
||||
export interface CaptureResult {
|
||||
/** Absolute path of the image that was written. */
|
||||
path: string;
|
||||
bytes: number;
|
||||
/** The tool that produced it, for the transcript and for error messages. */
|
||||
backend: string;
|
||||
}
|
||||
|
||||
export interface CaptureOptions {
|
||||
/** Where images are kept. The caller supplies `environment.supportPath`, which keeps this module Vicinae-free. */
|
||||
outputDir: string;
|
||||
platform?: NodeJS.Platform;
|
||||
signal?: AbortSignal;
|
||||
now?: Date;
|
||||
/** Overrides for tests. */
|
||||
hasCommand?: (tool: string) => Promise<boolean>;
|
||||
run?: RunCommand;
|
||||
}
|
||||
|
||||
/** Raised when the machine has no capture tool at all, with something actionable to do about it. */
|
||||
export class NoCaptureToolError extends Error {
|
||||
readonly hint: string;
|
||||
|
||||
constructor(platform: NodeJS.Platform) {
|
||||
super("No screen capture tool is installed.");
|
||||
this.name = "NoCaptureToolError";
|
||||
this.hint =
|
||||
platform === "darwin"
|
||||
? "macOS ships screencapture, so this should not happen. Check that it is on PATH."
|
||||
: "Install one of: grim + slurp (Wayland), grimblast (Hyprland), spectacle, maim + slop (X11), scrot, or flameshot.";
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Raised when the system refuses to hand over the screen.
|
||||
*
|
||||
* Separate from a generic tool failure because it is the one capture error the
|
||||
* user can actually fix, and because on macOS the fix is not obvious: the
|
||||
* permission belongs to the *launcher*, since TCC attributes the request to the
|
||||
* app responsible for spawning the extension runtime rather than to the
|
||||
* extension itself.
|
||||
*/
|
||||
export class CapturePermissionError extends Error {
|
||||
readonly hint: string;
|
||||
|
||||
constructor(platform: NodeJS.Platform) {
|
||||
super("The system denied screen capture.");
|
||||
this.name = "CapturePermissionError";
|
||||
this.hint =
|
||||
platform === "darwin"
|
||||
? "Give Vicinae Screen Recording permission in System Settings under Privacy & Security, then quit Vicinae and open it again. The permission belongs to Vicinae rather than to the extension, because macOS attributes the request to the app that launched the extension runtime."
|
||||
: "Give the launcher permission to record the screen, then restart it.";
|
||||
}
|
||||
}
|
||||
|
||||
/** A capture tool that ran and failed for a reason it could explain. */
|
||||
export class CaptureToolError extends Error {
|
||||
readonly tool: string;
|
||||
readonly stderr: string;
|
||||
|
||||
constructor(tool: string, stderr: string, cause?: unknown) {
|
||||
const detail = stderr.trim().split("\n")[0] ?? "";
|
||||
super(`${tool} failed.${detail ? ` ${detail}` : ""}`, cause === undefined ? undefined : { cause });
|
||||
this.name = "CaptureToolError";
|
||||
this.tool = tool;
|
||||
this.stderr = stderr;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* `command -v` rather than spawning the tool itself: several of these block or
|
||||
* open a window when run bare, and `--version` is not reliably supported.
|
||||
*
|
||||
* The tool names are literals from the table below and never come from the
|
||||
* user, so there is nothing to escape.
|
||||
*/
|
||||
async function commandExists(tool: string): Promise<boolean> {
|
||||
return new Promise((resolve) => {
|
||||
execFile("sh", ["-c", `command -v ${tool}`], { timeout: 5_000 }, (error) => resolve(!error));
|
||||
});
|
||||
}
|
||||
|
||||
function runCommand(
|
||||
tool: string,
|
||||
args: string[],
|
||||
options: { signal?: AbortSignal } = {},
|
||||
): Promise<RunResult> {
|
||||
return new Promise((resolve, reject) => {
|
||||
execFile(
|
||||
tool,
|
||||
args,
|
||||
{
|
||||
signal: options.signal,
|
||||
timeout: CAPTURE_TIMEOUT_MS,
|
||||
maxBuffer: MAX_CAPTURE_BYTES,
|
||||
encoding: "buffer",
|
||||
},
|
||||
(error, stdout, stderr) => {
|
||||
const text = stderr?.toString("utf8") ?? "";
|
||||
|
||||
if (error) {
|
||||
reject(new CaptureToolError(tool, text, error));
|
||||
return;
|
||||
}
|
||||
|
||||
resolve({ stdout: stdout as Buffer, stderr: text });
|
||||
},
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* `grim -g "$(slurp)"`, expressed as two argv calls rather than a shell pipeline.
|
||||
*
|
||||
* slurp prints the chosen geometry and exits non-zero when the selection is
|
||||
* cancelled, so an empty or absent geometry is the cancel signal.
|
||||
*/
|
||||
const grimSlurp: CaptureBackend = {
|
||||
name: "grim + slurp",
|
||||
requires: ["grim", "slurp"],
|
||||
async capture({ output, signal, run }) {
|
||||
const geometry = (await run("slurp", [], { signal })).stdout.toString("utf8").trim();
|
||||
if (!geometry) return;
|
||||
await run("grim", ["-g", geometry, output], { signal });
|
||||
},
|
||||
};
|
||||
|
||||
/**
|
||||
* flameshot is the only backend that will not take an output path: `gui --raw`
|
||||
* writes the PNG to stdout, so it is captured and written here instead.
|
||||
*/
|
||||
const flameshot: CaptureBackend = {
|
||||
name: "flameshot",
|
||||
requires: ["flameshot"],
|
||||
async capture({ output, signal, run }) {
|
||||
const png = (await run("flameshot", ["gui", "--raw"], { signal })).stdout;
|
||||
if (png.length === 0) return;
|
||||
await writeFile(output, png);
|
||||
},
|
||||
};
|
||||
|
||||
/**
|
||||
* Backends in preference order.
|
||||
*
|
||||
* Wayland first on Linux, because that is what the launcher's own users are on
|
||||
* (the README's shortcut examples are Hyprland and niri), then the X11 tools.
|
||||
* macOS needs only the built-in tool, which is why it is tried on its own.
|
||||
*/
|
||||
export const CAPTURE_BACKENDS: CaptureBackend[] = [
|
||||
{
|
||||
name: "grimblast",
|
||||
requires: ["grimblast"],
|
||||
async capture({ output, signal, run }) {
|
||||
await run("grimblast", ["save", "area", output], { signal });
|
||||
},
|
||||
},
|
||||
grimSlurp,
|
||||
{
|
||||
name: "spectacle",
|
||||
requires: ["spectacle"],
|
||||
async capture({ output, signal, run }) {
|
||||
// -r region, -b background, -n no notification, -o output file.
|
||||
await run("spectacle", ["-r", "-b", "-n", "-o", output], { signal });
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "maim",
|
||||
requires: ["maim"],
|
||||
async capture({ output, signal, run }) {
|
||||
await run("maim", ["-s", output], { signal });
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "scrot",
|
||||
requires: ["scrot"],
|
||||
async capture({ output, signal, run }) {
|
||||
await run("scrot", ["-s", output], { signal });
|
||||
},
|
||||
},
|
||||
flameshot,
|
||||
{
|
||||
name: "screencapture",
|
||||
requires: ["screencapture"],
|
||||
async capture({ output, signal, run }) {
|
||||
// -i interactive region, -x no shutter sound.
|
||||
await run("screencapture", ["-i", "-x", output], { signal });
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
/**
|
||||
* The first backend whose tools are all installed, or `undefined`.
|
||||
*
|
||||
* `hasCommand` is a parameter rather than a call so the preference order can be
|
||||
* tested against a fake `PATH`, which is the part most likely to break on a
|
||||
* platform nobody here runs.
|
||||
*/
|
||||
export async function selectBackend(
|
||||
hasCommand: (tool: string) => Promise<boolean>,
|
||||
backends: CaptureBackend[] = CAPTURE_BACKENDS,
|
||||
): Promise<CaptureBackend | undefined> {
|
||||
for (const backend of backends) {
|
||||
const checks = await Promise.all(backend.requires.map((tool) => hasCommand(tool)));
|
||||
if (checks.every(Boolean)) return backend;
|
||||
}
|
||||
|
||||
return undefined;
|
||||
}
|
||||
|
||||
/** Sortable, filesystem-safe, and obvious when found lying in the support directory. */
|
||||
export function snipFileName(now: Date): string {
|
||||
const pad = (value: number) => String(value).padStart(2, "0");
|
||||
|
||||
const date = `${now.getFullYear()}${pad(now.getMonth() + 1)}${pad(now.getDate())}`;
|
||||
const time = `${pad(now.getHours())}${pad(now.getMinutes())}${pad(now.getSeconds())}`;
|
||||
|
||||
return `snip-${date}-${time}.png`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Captures a region and returns the file it wrote, or `null` when the user
|
||||
* backed out.
|
||||
*
|
||||
* Success is decided by the file, not by an exit code: the tools disagree about
|
||||
* what a cancelled selection looks like, and at least one reports success
|
||||
* anyway. Failure is then triaged from what the tool said on stderr, which is
|
||||
* the only channel that separates a refused permission from a cancel.
|
||||
*/
|
||||
export async function captureRegion(options: CaptureOptions): Promise<CaptureResult | null> {
|
||||
const platform = options.platform ?? process.platform;
|
||||
const hasCommand = options.hasCommand ?? commandExists;
|
||||
const baseRun = options.run ?? runCommand;
|
||||
|
||||
const backend = await selectBackend(
|
||||
hasCommand,
|
||||
platform === "darwin" ? CAPTURE_BACKENDS.filter((b) => b.name === "screencapture") : CAPTURE_BACKENDS,
|
||||
);
|
||||
if (!backend) throw new NoCaptureToolError(platform);
|
||||
|
||||
await mkdir(options.outputDir, { recursive: true });
|
||||
const output = join(options.outputDir, snipFileName(options.now ?? new Date()));
|
||||
|
||||
/** Everything the tools said, whether or not they exited cleanly. */
|
||||
const diagnostics: string[] = [];
|
||||
const run: RunCommand = async (tool, args, runOptions) => {
|
||||
try {
|
||||
const result = await baseRun(tool, args, runOptions);
|
||||
if (result.stderr.trim()) diagnostics.push(result.stderr.trim());
|
||||
return result;
|
||||
} catch (thrown) {
|
||||
const stderr = thrown instanceof CaptureToolError ? thrown.stderr : "";
|
||||
if (stderr.trim()) diagnostics.push(stderr.trim());
|
||||
throw thrown;
|
||||
}
|
||||
};
|
||||
|
||||
let failure: unknown;
|
||||
try {
|
||||
await backend.capture({ output, signal: options.signal, run });
|
||||
} catch (thrown) {
|
||||
failure = thrown;
|
||||
}
|
||||
|
||||
const written = await stat(output).catch(() => null);
|
||||
if (written && written.size > 0) {
|
||||
return { path: output, bytes: written.size, backend: backend.name };
|
||||
}
|
||||
|
||||
const detail = diagnostics.join("\n");
|
||||
|
||||
if (SCREEN_RECORDING_DENIED.test(detail)) throw new CapturePermissionError(platform);
|
||||
|
||||
// A tool that explained itself is a failure worth reporting. Silence with no
|
||||
// image is the user pressing Escape, which is the common case and says nothing.
|
||||
if (detail) throw new CaptureToolError(backend.requires[0], detail, failure);
|
||||
|
||||
return null;
|
||||
}
|
||||
+190
-3
@@ -88,6 +88,35 @@ const OPENING_FENCE = /^(\s*)(`{3,}|~{3,})\s*([^\s`~]*)\s*$/;
|
||||
const CLOSING_FENCE = /^\s*(`{3,}|~{3,})\s*$/;
|
||||
const STANDALONE_CODE_SPAN = /^\s*`([^`\n]+)`\s*[.!]?\s*$/;
|
||||
|
||||
/**
|
||||
* A line that is nothing but a bare http(s) URL, give or take a bullet.
|
||||
*
|
||||
* Vicinae only makes a URL clickable when markdown presents it as a link. Its
|
||||
* parser attaches the `table` and `strikethrough` GFM extensions and nothing
|
||||
* else, so the `autolink` extension is off: a bare URL renders as plain dead
|
||||
* text. `MarkdownModel` turns `CMARK_NODE_LINK` into `<a href=...>`, and the
|
||||
* selection controller opens that anchor on release, so `[label](url)` is the
|
||||
* one form the user can actually click.
|
||||
*/
|
||||
const BARE_URL = /^https?:\/\/\S+$/;
|
||||
/** Sentence punctuation a model appends to a URL that is not part of it. */
|
||||
const TRAILING_SENTENCE_PUNCTUATION = /[.,;:!?]+$/;
|
||||
/** One inline code span that holds nothing but a URL. */
|
||||
const STANDALONE_URL_SPAN = /^`([^`\n]+)`[.!]?$/;
|
||||
/** A bullet or numbered-list marker, kept so a list of links stays a list. */
|
||||
const LIST_MARKER = /^([ \t]*(?:[-*+]|\d+[.)])[ \t]+)(.*)$/;
|
||||
|
||||
/** A GFM table row: it opens and closes with a pipe. */
|
||||
const TABLE_ROW = /^\s*\|.*\|\s*$/;
|
||||
/**
|
||||
* The delimiter row under a header.
|
||||
*
|
||||
* The test is not how many dashes there are, because GFM wants only one per
|
||||
* column. It is that the line could be nothing else: pipes, dashes, colons and
|
||||
* spaces, with at least one dash to tell it apart from a row of empty cells.
|
||||
*/
|
||||
const TABLE_DELIMITER_ROW = /^\s*\|?[\s:|-]*-[\s:|-]*\|?\s*$/;
|
||||
|
||||
function significantLines(code: string): string[] {
|
||||
return code
|
||||
.split("\n")
|
||||
@@ -272,12 +301,170 @@ export function labelCodeFences(markdown: string): string {
|
||||
return out.join("\n");
|
||||
}
|
||||
|
||||
/** A URL with the sentence punctuation after it removed. */
|
||||
function asBareUrl(candidate: string): string | undefined {
|
||||
const trimmed = candidate.trim();
|
||||
if (!BARE_URL.test(trimmed)) return undefined;
|
||||
return trimmed.replace(TRAILING_SENTENCE_PUNCTUATION, "");
|
||||
}
|
||||
|
||||
/**
|
||||
* Wraps a URL as a markdown link, using the address itself as the label so the
|
||||
* destination stays readable rather than turning into a generic word.
|
||||
*
|
||||
* A destination containing parentheses (plenty of Wikipedia articles) has to
|
||||
* be wrapped in pointy brackets or the parser reads the first `)` as the end of
|
||||
* the destination and leaves the rest as trailing prose.
|
||||
*/
|
||||
function asLink(url: string): string {
|
||||
// Square brackets in the *label* would end the link text early, which is why
|
||||
// an IPv6 literal such as `http://[::1]/` needs them escaped. The destination
|
||||
// tolerates brackets, but not parentheses, which is what the pointy form is
|
||||
// for.
|
||||
const label = url.replace(/[[\]]/g, "\\$&");
|
||||
const destination = /[()]/.test(url) ? `<${url}>` : url;
|
||||
return `[${label}](${destination})`;
|
||||
}
|
||||
|
||||
/**
|
||||
* The URLs of a fence whose every non-empty line is one, or `undefined` when
|
||||
* the block is real code.
|
||||
*
|
||||
* Deliberately narrow: one line that is not a URL disqualifies the whole block,
|
||||
* so a `curl` invocation that happens to contain an address is never touched.
|
||||
*/
|
||||
function urlOnlyBlock(lines: string[]): string[] | undefined {
|
||||
const urls: string[] = [];
|
||||
|
||||
for (const line of lines) {
|
||||
const trimmed = line.trim();
|
||||
if (trimmed.length === 0) continue;
|
||||
|
||||
const url = asBareUrl(trimmed);
|
||||
if (!url) return undefined;
|
||||
urls.push(url);
|
||||
}
|
||||
|
||||
return urls.length > 0 ? urls : undefined;
|
||||
}
|
||||
|
||||
/** Rewrites a line that is only a URL, or only a span holding one, into a link. */
|
||||
function linkifyLine(line: string): string {
|
||||
const marker = LIST_MARKER.exec(line);
|
||||
const prefix = marker ? marker[1] : (line.match(/^[ \t]*/)?.[0] ?? "");
|
||||
const rest = marker ? marker[2] : line.slice(prefix.length);
|
||||
|
||||
const span = STANDALONE_URL_SPAN.exec(rest.trim());
|
||||
const url = asBareUrl(span ? span[1] : rest);
|
||||
|
||||
return url ? `${prefix}${asLink(url)}` : line;
|
||||
}
|
||||
|
||||
/**
|
||||
* Turns URLs the model wrote as code back into links.
|
||||
*
|
||||
* This is the reported bug: asked for a few GitHub projects, the model answers
|
||||
* with a fenced block of bare addresses. A fence renders through
|
||||
* `MdCodeBlock.qml` as a dark panel with a Copy button and no link handling at
|
||||
* all, so the one thing the user wants to do with a URL is the one thing they
|
||||
* cannot. The same applies to a lone `` `https://...` `` span.
|
||||
*
|
||||
* Three shapes are recovered, and only these:
|
||||
*
|
||||
* 1. A fence whose entire body is URLs.
|
||||
* 2. A line that is exactly one inline span holding a URL.
|
||||
* 3. A line that is exactly a URL, bulleted or not.
|
||||
*
|
||||
* Existing fences holding anything else are copied through untouched, so real
|
||||
* code is never rewritten.
|
||||
*/
|
||||
export function restoreUrlLinks(markdown: string): string {
|
||||
const lines = markdown.split("\n");
|
||||
const out: string[] = [];
|
||||
let index = 0;
|
||||
|
||||
while (index < lines.length) {
|
||||
if (!OPENING_FENCE.test(lines[index])) {
|
||||
out.push(linkifyLine(lines[index]));
|
||||
index += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
const body: string[] = [];
|
||||
let cursor = index + 1;
|
||||
while (cursor < lines.length && !CLOSING_FENCE.test(lines[cursor])) {
|
||||
body.push(lines[cursor]);
|
||||
cursor += 1;
|
||||
}
|
||||
|
||||
const closing = cursor < lines.length;
|
||||
const urls = urlOnlyBlock(body);
|
||||
|
||||
if (urls) out.push(...urls.map(asLink));
|
||||
else {
|
||||
out.push(lines[index], ...body);
|
||||
if (closing) out.push(lines[cursor]);
|
||||
}
|
||||
|
||||
index = cursor + (closing ? 1 : 0);
|
||||
}
|
||||
|
||||
return out.join("\n");
|
||||
}
|
||||
|
||||
/**
|
||||
* Inserts the blank line a table needs to start at all.
|
||||
*
|
||||
* A GFM table cannot interrupt a paragraph, so a table written straight after a
|
||||
* line of prose is not a table: it renders as a paragraph of pipes and dashes,
|
||||
* which is how a perfectly good answer turns into noise. The blank line looks
|
||||
* unnecessary, so models leave it out.
|
||||
*
|
||||
* Only a real table start is touched, which is a row followed by a delimiter
|
||||
* row, so prose that happens to contain a pipe is left alone. Fences are copied
|
||||
* through untouched for the same reason, since a table shown as example code
|
||||
* must stay exactly as written.
|
||||
*/
|
||||
export function separateTables(markdown: string): string {
|
||||
const lines = markdown.split("\n");
|
||||
const out: string[] = [];
|
||||
let index = 0;
|
||||
|
||||
while (index < lines.length) {
|
||||
if (OPENING_FENCE.test(lines[index])) {
|
||||
index = copyFence(lines, out, index);
|
||||
continue;
|
||||
}
|
||||
|
||||
const startsTable =
|
||||
TABLE_ROW.test(lines[index]) && index + 1 < lines.length && TABLE_DELIMITER_ROW.test(lines[index + 1]);
|
||||
|
||||
// Only when something would otherwise swallow it. A table already at a block
|
||||
// boundary, after a blank line or after a fenced block, needs nothing: the
|
||||
// fence has already ended the preceding block.
|
||||
const previous = out[out.length - 1];
|
||||
const atBlockBoundary =
|
||||
previous === undefined || previous.trim().length === 0 || CLOSING_FENCE.test(previous);
|
||||
|
||||
if (startsTable && !atBlockBoundary) out.push("");
|
||||
|
||||
out.push(lines[index]);
|
||||
index += 1;
|
||||
}
|
||||
|
||||
return out.join("\n");
|
||||
}
|
||||
|
||||
/**
|
||||
* The full pre-render pass, in order.
|
||||
*
|
||||
* Promotion runs first so a promoted block is seen as an ordinary fence by the
|
||||
* labeller, which is also harmless when promotion already inferred a language.
|
||||
* Tables are separated first because they change block boundaries: a blank line
|
||||
* has to exist before anything else looks at the line. Links are restored next
|
||||
* so a recovered URL leaves no span for the promoter to fence again. Promotion
|
||||
* then runs before labelling, so a promoted block is seen as an ordinary fence
|
||||
* by the labeller, which is also harmless when promotion already inferred a
|
||||
* language.
|
||||
*/
|
||||
export function repairCodeBlocks(markdown: string): string {
|
||||
return labelCodeFences(promoteStandaloneCodeSpans(markdown));
|
||||
return labelCodeFences(promoteStandaloneCodeSpans(restoreUrlLinks(separateTables(markdown))));
|
||||
}
|
||||
|
||||
+151
-38
@@ -1,10 +1,14 @@
|
||||
import { Action, ActionPanel, Icon, List } from "@vicinae/api";
|
||||
import { Action, ActionPanel, Icon, LaunchType, List, launchCommand } from "@vicinae/api";
|
||||
import { useCallback, useEffect, useMemo, useState } from "react";
|
||||
import { readAttachment } from "../attachments";
|
||||
import { useChat } from "../hooks/useChat";
|
||||
import { buildThreadMarkdown, type TranscriptOptions } from "../markdown";
|
||||
import { useVisionSupport } from "../hooks/useVision";
|
||||
import { buildThreadMarkdown, renderAttachments, type TranscriptOptions } from "../markdown";
|
||||
import { getLamPreferences } from "../preferences";
|
||||
import { deleteSession, listSessions, loadSession, summarizeSession } from "../sessions";
|
||||
import { buildThread } from "../thread";import type { Session, SessionSummary } from "../types";
|
||||
import { resolveStartup } from "../startup";
|
||||
import { buildThread } from "../thread";
|
||||
import type { PendingAttachment, Session, SessionSummary } from "../types";
|
||||
import { recencyGroup, truncate } from "../utils";
|
||||
|
||||
/**
|
||||
@@ -44,15 +48,30 @@ import { recencyGroup, truncate } from "../utils";
|
||||
* selection to the first row on every model change
|
||||
* (`GenericListView.selectFirstOnReset`), and streaming changes the model on
|
||||
* every flush, so ordering by recency keeps the chat in progress near the top.
|
||||
* - Capturing a region does **not** happen here. It is delegated to the `snip`
|
||||
* command, because this view owns a window and the window would be in the
|
||||
* shot. See {@link ChatViewProps}.
|
||||
*/
|
||||
|
||||
const EMPTY_TRANSCRIPT = "**Ask Lam anything**\n\nType a question in the search bar and press Return.";
|
||||
|
||||
/** Sent when a snip is attached and Return is pressed before anything is typed. */
|
||||
const DEFAULT_IMAGE_PROMPT = "What is in this image?";
|
||||
|
||||
/** The reminder shown while a snip is attached but not yet sent. */
|
||||
const PENDING_NOTE = "_Ask a question about it, or press Return to send it as it is._";
|
||||
|
||||
export interface ChatViewProps {
|
||||
/** Opens a specific conversation instead of resuming the latest one. */
|
||||
initialSession?: Session;
|
||||
/** Text handed over by a deeplink or fallback command. */
|
||||
initialPrompt?: string;
|
||||
/** A conversation to reload by id, which is how the snip command hands one back. */
|
||||
initialSessionId?: string;
|
||||
/** A captured image to read in, which is how the snip command hands one over. */
|
||||
initialAttachmentPath?: string;
|
||||
/** Something the launching command needs to say, such as a failed capture. */
|
||||
launchNote?: string;
|
||||
}
|
||||
|
||||
interface SessionRow {
|
||||
@@ -62,14 +81,26 @@ interface SessionRow {
|
||||
isActive: boolean;
|
||||
}
|
||||
|
||||
export function ChatView({ initialSession, initialPrompt }: ChatViewProps) {
|
||||
export function ChatView({
|
||||
initialSession,
|
||||
initialPrompt,
|
||||
initialSessionId,
|
||||
initialAttachmentPath,
|
||||
launchNote,
|
||||
}: ChatViewProps) {
|
||||
const preferences = useMemo(() => getLamPreferences(), []);
|
||||
const chat = useChat({ preferences, initialSession });
|
||||
const { replaceSession } = chat;
|
||||
const { canSeeImages } = useVisionSupport(preferences);
|
||||
|
||||
const [input, setInput] = useState(initialPrompt ?? "");
|
||||
const [summaries, setSummaries] = useState<SessionSummary[] | undefined>(undefined);
|
||||
const [resumed, setResumed] = useState(Boolean(initialSession));
|
||||
// One flag, not two. See the startup effect below for why that matters.
|
||||
const [startup, setStartup] = useState<"loading" | "ready">(initialSession ? "ready" : "loading");
|
||||
// A captured image is held here until the user says what they want to know
|
||||
// about it. Nothing is sent to the model on capture alone.
|
||||
const [pending, setPending] = useState<PendingAttachment | undefined>(undefined);
|
||||
const [snipNote, setSnipNote] = useState<string | undefined>(undefined);
|
||||
|
||||
const activeId = chat.session.id;
|
||||
|
||||
@@ -85,35 +116,47 @@ export function ChatView({ initialSession, initialPrompt }: ChatViewProps) {
|
||||
}, [reload, chat.session.updatedAt]);
|
||||
|
||||
/**
|
||||
* Reopening the command continues the most recent conversation, but only if it
|
||||
* was used recently. Past the configured idle window it starts a fresh chat
|
||||
* instead. The previous conversation is never removed, it just drops down the
|
||||
* sidebar. Setting the preference to "Never" keeps the last chat open forever.
|
||||
* Everything that has to happen before the first render, in one place.
|
||||
*
|
||||
* This used to be two effects guarding two booleans, and a hand-off launch
|
||||
* satisfied the guard of *both* while setting neither, so `resumed` stayed
|
||||
* false and the view sat on "Loading conversations…" forever. Anything that
|
||||
* blocks the first render belongs in one effect with one flag, and the flag is
|
||||
* cleared in a `finally` so that no branch, including a thrown one, can leave
|
||||
* the view spinning.
|
||||
*/
|
||||
useEffect(() => {
|
||||
if (initialSession) return;
|
||||
|
||||
let cancelled = false;
|
||||
|
||||
void (async () => {
|
||||
try {
|
||||
const [latest] = await listSessions();
|
||||
if (!latest) return;
|
||||
const outcome = await resolveStartup(
|
||||
{
|
||||
hasInitialSession: Boolean(initialSession),
|
||||
initialSessionId,
|
||||
initialAttachmentPath,
|
||||
resumeTimeoutMs: preferences.resumeTimeoutMs,
|
||||
now: Date.now(),
|
||||
},
|
||||
{ loadSession, listSessions, readAttachment },
|
||||
);
|
||||
|
||||
const idleMs = Date.now() - latest.updatedAt;
|
||||
const { resumeTimeoutMs } = preferences;
|
||||
if (resumeTimeoutMs !== undefined && idleMs > resumeTimeoutMs) return;
|
||||
if (cancelled) return;
|
||||
|
||||
const session = await loadSession(latest.id);
|
||||
if (!cancelled && session) replaceSession(session);
|
||||
if (outcome.session) replaceSession(outcome.session);
|
||||
if (outcome.attachment) setPending(outcome.attachment);
|
||||
if (outcome.attachmentError) {
|
||||
setSnipNote(`**Could not open the screenshot:** ${outcome.attachmentError}`);
|
||||
}
|
||||
} finally {
|
||||
if (!cancelled) setResumed(true);
|
||||
if (!cancelled) setStartup("ready");
|
||||
}
|
||||
})();
|
||||
|
||||
return () => {
|
||||
cancelled = true;
|
||||
};
|
||||
}, [initialSession, preferences, replaceSession]);
|
||||
}, [initialSession, initialSessionId, initialAttachmentPath, preferences.resumeTimeoutMs, replaceSession]);
|
||||
|
||||
const thread = useMemo(
|
||||
() =>
|
||||
@@ -131,11 +174,31 @@ export function ChatView({ initialSession, initialPrompt }: ChatViewProps) {
|
||||
[preferences.showToolCalls, preferences.showReasoning],
|
||||
);
|
||||
|
||||
const threadMarkdown = useMemo(
|
||||
() =>
|
||||
thread.length === 0 ? EMPTY_TRANSCRIPT : buildThreadMarkdown(thread, transcriptOptions),
|
||||
[thread, transcriptOptions],
|
||||
);
|
||||
/**
|
||||
* The transcript, preceded by any notes about this view's own state.
|
||||
*
|
||||
* Notes go first because the transcript reads newest-first: the top of the
|
||||
* panel is the text directly under the search bar, which is where the user is
|
||||
* already looking.
|
||||
*/
|
||||
const threadMarkdown = useMemo(() => {
|
||||
const notes = [
|
||||
launchNote,
|
||||
snipNote,
|
||||
// The preview comes first so the captured image is visible before the user
|
||||
// decides what to ask about it, rather than only after sending.
|
||||
pending ? `${renderAttachments([pending])}\n\n${PENDING_NOTE}` : undefined,
|
||||
].filter((note): note is string => Boolean(note));
|
||||
|
||||
// The placeholder is what tells a user that the search bar is the composer.
|
||||
// When a note above already says that, printing both put the same
|
||||
// instruction twice, one line under the other.
|
||||
const empty = notes.length === 0 ? EMPTY_TRANSCRIPT : "";
|
||||
const transcript =
|
||||
thread.length === 0 ? empty : buildThreadMarkdown(thread, transcriptOptions);
|
||||
|
||||
return [...notes, transcript].filter((part) => part.length > 0).join("\n\n");
|
||||
}, [launchNote, snipNote, pending, thread, transcriptOptions]);
|
||||
|
||||
const rows: SessionRow[] = useMemo(() => {
|
||||
const base = summaries ?? [];
|
||||
@@ -151,10 +214,7 @@ export function ChatView({ initialSession, initialPrompt }: ChatViewProps) {
|
||||
if (base.some((summary) => summary.id === activeId)) return mapped;
|
||||
|
||||
const active = summarizeSession(chat.session);
|
||||
return [
|
||||
{ id: active.id, title: active.title, updatedAt: active.updatedAt, isActive: true },
|
||||
...mapped,
|
||||
];
|
||||
return [{ id: active.id, title: active.title, updatedAt: active.updatedAt, isActive: true }, ...mapped];
|
||||
}, [summaries, activeId, chat.session]);
|
||||
|
||||
/**
|
||||
@@ -194,15 +254,28 @@ export function ChatView({ initialSession, initialPrompt }: ChatViewProps) {
|
||||
[activeId, chat, reload],
|
||||
);
|
||||
|
||||
/** The snip a "show me the screenshot" action would open: the pending one, else the newest. */
|
||||
const latestAttachment =
|
||||
pending ?? [...chat.session.turns].reverse().find((turn) => turn.attachments.length > 0)?.attachments[0];
|
||||
|
||||
const send = useCallback(
|
||||
(text: string) => {
|
||||
chat.ask(text);
|
||||
const typed = text.trim();
|
||||
// With a snip attached, Return has to send even before anything is typed,
|
||||
// otherwise the only way to ask about an image is to type out a question
|
||||
// the image already implies.
|
||||
const prompt = typed || (pending ? DEFAULT_IMAGE_PROMPT : "");
|
||||
if (!prompt) return;
|
||||
|
||||
chat.ask(prompt, pending);
|
||||
setPending(undefined);
|
||||
setSnipNote(undefined);
|
||||
setInput("");
|
||||
},
|
||||
[chat.ask],
|
||||
[chat.ask, pending],
|
||||
);
|
||||
|
||||
const hasPrompt = input.trim().length > 0;
|
||||
const hasPrompt = input.trim().length > 0 || pending !== undefined;
|
||||
|
||||
const stopAction = <Action title="Stop Generating" icon={Icon.Stop} onAction={chat.cancel} />;
|
||||
const sendAction = <Action title="Send" icon={Icon.SpeechBubble} onAction={() => send(input)} />;
|
||||
@@ -216,10 +289,46 @@ export function ChatView({ initialSession, initialPrompt }: ChatViewProps) {
|
||||
/>
|
||||
);
|
||||
|
||||
/**
|
||||
* Hands the capture to the `snip` command instead of doing it here.
|
||||
*
|
||||
* Two reasons, both structural. The window has to be out of the way, and only
|
||||
* the snip command is positioned to close it before the selector appears. And
|
||||
* launching a different command is what makes the hand-off safe: Vicinae's
|
||||
* `activateEntrypoint` treats a launch of the *currently active* view command
|
||||
* as a toggle and closes the window instead of reopening it, so relaying
|
||||
* through `snip` avoids ever asking for `chat` while `chat` is active.
|
||||
*
|
||||
* Absent while streaming, because the hand-off unloads this view and would
|
||||
* abandon an answer mid-sentence.
|
||||
*/
|
||||
const snipAction =
|
||||
canSeeImages && !chat.isStreaming ? (
|
||||
<Action
|
||||
title="Snip a Region"
|
||||
icon={Icon.Screenshot}
|
||||
shortcut={{ key: "s", modifiers: ["cmd", "shift"] }}
|
||||
onAction={() =>
|
||||
void launchCommand({
|
||||
name: "snip",
|
||||
type: LaunchType.UserInitiated,
|
||||
context: { sessionId: chat.session.id },
|
||||
})
|
||||
}
|
||||
/>
|
||||
) : null;
|
||||
|
||||
// A screenshot is a file, so it opens in the system image viewer. Without this
|
||||
// the only way to see what was sent would be to find it on disk by hand.
|
||||
const openSnipAction = latestAttachment ? (
|
||||
<Action.Open title="Open Screenshot" icon={Icon.Image} target={latestAttachment.path} />
|
||||
) : null;
|
||||
|
||||
/**
|
||||
* The primary action adapts to the search bar. With text in it, Return sends;
|
||||
* with it empty (where sending would be a no-op anyway), Return opens the
|
||||
* highlighted conversation instead.
|
||||
* highlighted conversation instead. A pending snip counts as text, so Return
|
||||
* sends it rather than navigating away.
|
||||
*/
|
||||
const rowActions = (row: SessionRow) => {
|
||||
const openAction = (
|
||||
@@ -231,6 +340,8 @@ export function ChatView({ initialSession, initialPrompt }: ChatViewProps) {
|
||||
{chat.isStreaming ? stopAction : hasPrompt ? sendAction : openAction}
|
||||
{hasPrompt ? openAction : null}
|
||||
{newChatAction}
|
||||
{snipAction}
|
||||
{openSnipAction}
|
||||
<Action
|
||||
title="Delete Conversation"
|
||||
icon={Icon.Trash}
|
||||
@@ -244,13 +355,13 @@ export function ChatView({ initialSession, initialPrompt }: ChatViewProps) {
|
||||
);
|
||||
};
|
||||
|
||||
if (!resumed || summaries === undefined) {
|
||||
if (startup === "loading" || summaries === undefined) {
|
||||
return <List isLoading navigationTitle="Lam" searchBarPlaceholder="Loading conversations…" />;
|
||||
}
|
||||
|
||||
const placeholder = chat.status
|
||||
? `${chat.status}. Press ⏎ to stop`
|
||||
: "Lam is thinking. Press ⏎ to stop";
|
||||
const placeholder = chat.status ? `${chat.status}. Press ⏎ to stop` : "Lam is thinking. Press ⏎ to stop";
|
||||
|
||||
const idlePlaceholder = pending ? "Ask about the screenshot…" : "Ask Lam anything…";
|
||||
|
||||
return (
|
||||
<List
|
||||
@@ -262,11 +373,13 @@ export function ChatView({ initialSession, initialPrompt }: ChatViewProps) {
|
||||
// Raycast puts the conversation title in a header; Vicinae's nearest
|
||||
// equivalent is the status-bar title, so the open chat is named there too.
|
||||
navigationTitle={truncate(chat.session.title, 40)}
|
||||
searchBarPlaceholder={chat.isStreaming ? placeholder : "Ask Lam anything…"}
|
||||
searchBarPlaceholder={chat.isStreaming ? placeholder : idlePlaceholder}
|
||||
actions={
|
||||
<ActionPanel>
|
||||
{chat.isStreaming ? stopAction : sendAction}
|
||||
{newChatAction}
|
||||
{snipAction}
|
||||
{openSnipAction}
|
||||
</ActionPanel>
|
||||
}
|
||||
>
|
||||
|
||||
+16
-3
@@ -7,8 +7,17 @@ export const DEFAULT_MODEL = "gpt-oss:120b";
|
||||
export const STORAGE_INDEX_KEY = "lam.sessions.index";
|
||||
export const STORAGE_SESSION_PREFIX = "lam.session.";
|
||||
|
||||
/** How many times the model may call tools before Lam gives up and reports back. */
|
||||
export const MAX_TOOL_ITERATIONS = 5;
|
||||
/**
|
||||
* How many rounds of tool calls one turn may spend before the model is told to
|
||||
* answer with what it has.
|
||||
*
|
||||
* Five was too few for a question that needs several searches and a few pages,
|
||||
* and the failure mode was the worst one available: the budget ran out, every
|
||||
* preamble the model had written was discarded, and the turn ended with a notice
|
||||
* about the limit rather than an answer. The closing round in `agent.ts` now
|
||||
* turns that into an answer, and this is the room it gets to gather first.
|
||||
*/
|
||||
export const MAX_TOOL_ITERATIONS = 8;
|
||||
|
||||
/** Oldest sessions past this count are evicted, along with their storage keys. */
|
||||
export const MAX_SESSIONS = 100;
|
||||
@@ -24,7 +33,9 @@ export const DEFAULT_SYSTEM_PROMPT = `You are Lam, a friendly assistant in the V
|
||||
|
||||
Sound like a person, not a document. Use contractions, talk about "you" and "I", and a short natural remark is welcome ("Sure, easy one", "Ah, that trips people up"). Performed enthusiasm and filler added to seem helpful are not.
|
||||
|
||||
Lead with the answer, then the detail. This renders in a narrow window, so stay brief: a tight sentence or two, or a short list, then stop. Keep paragraphs short and skip large headings and wide tables.
|
||||
Lead with the answer, then the detail. This renders in a narrow window, so stay brief: a tight sentence or two, or a short list, then stop. Keep paragraphs short.
|
||||
|
||||
Use the markdown this window actually draws rather than flattening everything into prose. It renders tables, bulleted and numbered lists, headings, blockquotes, bold, italics, inline code and fenced code, so pick the shape that fits. A table is the right shape for comparing three or more things across the same few attributes, so write the table itself: never hand back a table as a bulleted list of rows, and never describe a table you could simply write. Keep it to about four narrow columns so it fits, give it a header row, and leave a blank line before it. A list is for steps or for independent items. Reach for a heading only when an answer genuinely has sections, since at this size a heading usually costs a line and buys nothing.
|
||||
|
||||
Write plainly:
|
||||
- Never use an em dash. Use a comma, a colon, or a full stop.
|
||||
@@ -36,6 +47,8 @@ Write plainly:
|
||||
|
||||
Fence every command and every snippet, and always name the language: bash for shell and terminal commands, then json, sql, yaml, or the language for code (ts, js, python, rust, go). Never leave code inline in a sentence, and never open a fence without a language. Put no $ or shell prompt inside the block so it can be copied and run as-is. Ignore the formatting of earlier answers in this conversation: if one put a command in inline code, fence it anyway.
|
||||
|
||||
A web address is a link, never code. A markdown link, written as [label](https://example.com), is clickable in this window, while a URL inside backticks or a fenced block is dead text the user cannot open. So never fence a URL, never put one in inline code, and never answer "here are the links" with a snippet. Label a link with what it is ("OpenClickly on GitHub"), and when you give several, put each on its own bullet with its own link.
|
||||
|
||||
A fenced block is never a whole answer on its own. Say in one plain sentence what the code does or when to reach for it, so the reply reads like something a person wrote rather than a bare snippet.
|
||||
|
||||
Call a tool when it makes the answer more accurate. Anything time-sensitive or past your cutoff should be checked rather than guessed, and never invent tool output. If a tool fails, say so and answer with what you know.
|
||||
|
||||
+240
@@ -0,0 +1,240 @@
|
||||
import { LocalStorage } from "@vicinae/api";
|
||||
import { z } from "zod";
|
||||
import { errorMessage } from "./utils";
|
||||
|
||||
/**
|
||||
* IP geolocation, opt-in.
|
||||
*
|
||||
* This is the one place in the extension that tells a third party anything about
|
||||
* the user. The README said no geolocation service ever saw the user's address,
|
||||
* and this is what makes that conditional, so it is off unless asked for, it is
|
||||
* skipped entirely when the user has already said where they are, and the tool
|
||||
* output names the provider so the model can qualify what it repeats.
|
||||
*
|
||||
* How accurate is it? Less than the name suggests, and the honest answer is that
|
||||
* an IP address locates the point where the traffic left the ISP, not the person.
|
||||
* Measured from one machine on Globe Telecom, an IPv4 query resolved to Cebu City
|
||||
* and an IPv6 query, moments later, to Talisay in Calabarzon: about 500km apart.
|
||||
* The cache is therefore not just a privacy measure, and the tool output calls
|
||||
* the result a rough guess rather than a city.
|
||||
*
|
||||
* The answer is cached for hours rather than looked up per call. A city does not
|
||||
* change between questions, and every lookup hands the provider another copy of
|
||||
* the user's IP address.
|
||||
*
|
||||
* Imports Vicinae for `LocalStorage` only, which is the same footing as
|
||||
* `sessions.ts` and `vision.ts`. Everything else about it is plain `fetch`, so
|
||||
* the parsing and the cache are unit tested without a runtime.
|
||||
*/
|
||||
|
||||
/** Where the lookup goes. Swapping providers means changing this and the parser below. */
|
||||
export const GEO_PROVIDER = "https://ipwho.is/";
|
||||
export const GEO_PROVIDER_NAME = "ipwho.is";
|
||||
|
||||
/** How long a resolved place is trusted. Long enough to not re-leak the IP on every question. */
|
||||
export const GEO_TTL_MS = 6 * 60 * 60 * 1000;
|
||||
|
||||
/** An IP lookup is a nice-to-have, so it must never hold up an answer. */
|
||||
const GEO_TIMEOUT_MS = 8_000;
|
||||
|
||||
const STORAGE_KEY = "lam.geo";
|
||||
|
||||
export interface GeoPlace {
|
||||
/** Ready to show, e.g. "Cebu City, Central Visayas, Philippines". */
|
||||
place: string;
|
||||
city?: string;
|
||||
region?: string;
|
||||
country?: string;
|
||||
latitude?: number;
|
||||
longitude?: number;
|
||||
timezone?: string;
|
||||
/** Named in the tool output, so the model can say where this came from. */
|
||||
source: string;
|
||||
fetchedAt: number;
|
||||
}
|
||||
|
||||
export type GeoLookup = { status: "ok"; place: GeoPlace } | { status: "failed"; message: string };
|
||||
|
||||
export interface GeoOptions {
|
||||
signal?: AbortSignal;
|
||||
/** Injected so the cache is deterministic under test. */
|
||||
now?: number;
|
||||
/** Ignores a cached answer. */
|
||||
refresh?: boolean;
|
||||
}
|
||||
|
||||
const CachedSchema = z.object({
|
||||
place: z.string(),
|
||||
city: z.string().optional(),
|
||||
region: z.string().optional(),
|
||||
country: z.string().optional(),
|
||||
latitude: z.number().optional(),
|
||||
longitude: z.number().optional(),
|
||||
timezone: z.string().optional(),
|
||||
source: z.string(),
|
||||
fetchedAt: z.number(),
|
||||
});
|
||||
|
||||
/** The subset of the provider's response this needs, with every field optional so a shape change degrades instead of throwing. */
|
||||
const ResponseSchema = z.object({
|
||||
success: z.boolean().optional(),
|
||||
message: z.string().optional(),
|
||||
city: z.string().nullable().optional(),
|
||||
region: z.string().nullable().optional(),
|
||||
country: z.string().nullable().optional(),
|
||||
latitude: z.number().nullable().optional(),
|
||||
longitude: z.number().nullable().optional(),
|
||||
// ipwho.is nests the timezone; some providers hand back a plain string.
|
||||
timezone: z
|
||||
.union([z.string(), z.object({ id: z.string().optional() })])
|
||||
.nullable()
|
||||
.optional(),
|
||||
});
|
||||
|
||||
/**
|
||||
* Joins the parts of a place, dropping duplicates.
|
||||
*
|
||||
* Providers often return the city twice (once as `city`, once as `region`) for
|
||||
* city-states and single-city regions, and "Singapore, Singapore, Singapore"
|
||||
* reads like a bug.
|
||||
*/
|
||||
export function buildPlace(parts: { city?: string; region?: string; country?: string }): string {
|
||||
const seen = new Set<string>();
|
||||
const kept: string[] = [];
|
||||
|
||||
for (const part of [parts.city, parts.region, parts.country]) {
|
||||
const trimmed = part?.trim();
|
||||
const key = trimmed?.toLowerCase();
|
||||
if (!trimmed || !key || seen.has(key)) continue;
|
||||
|
||||
seen.add(key);
|
||||
kept.push(trimmed);
|
||||
}
|
||||
|
||||
return kept.join(", ");
|
||||
}
|
||||
|
||||
/** Aborts on either the caller's signal or a timeout, whichever comes first. */
|
||||
async function fetchWithTimeout(
|
||||
url: string,
|
||||
signal: AbortSignal | undefined,
|
||||
timeoutMs: number,
|
||||
): Promise<Response> {
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), timeoutMs);
|
||||
const forward = () => controller.abort();
|
||||
|
||||
signal?.addEventListener("abort", forward, { once: true });
|
||||
|
||||
try {
|
||||
return await fetch(url, {
|
||||
signal: controller.signal,
|
||||
headers: {
|
||||
Accept: "application/json",
|
||||
// Named rather than anonymous. Node's default UA is bare, and "who is
|
||||
// asking" is the one thing a geolocation provider has a fair interest in.
|
||||
"User-Agent": "lam-agent-vicinae",
|
||||
},
|
||||
});
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
signal?.removeEventListener("abort", forward);
|
||||
}
|
||||
}
|
||||
|
||||
async function readCache(now: number): Promise<GeoPlace | undefined> {
|
||||
try {
|
||||
const raw = await LocalStorage.getItem<string>(STORAGE_KEY);
|
||||
if (!raw) return undefined;
|
||||
|
||||
const parsed = CachedSchema.safeParse(JSON.parse(raw));
|
||||
if (!parsed.success) return undefined;
|
||||
if (now - parsed.data.fetchedAt > GEO_TTL_MS) return undefined;
|
||||
|
||||
return parsed.data;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
async function writeCache(place: GeoPlace): Promise<void> {
|
||||
await LocalStorage.setItem(STORAGE_KEY, JSON.stringify(place)).catch(() => undefined);
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves the user's rough place from their IP address.
|
||||
*
|
||||
* Never throws: a lookup failure leaves the region-level answer intact, and the
|
||||
* message is passed on so the model can say what went wrong rather than quietly
|
||||
* presenting a coarser answer as though it were the same thing.
|
||||
*/
|
||||
export async function ipPlace(options: GeoOptions = {}): Promise<GeoLookup> {
|
||||
const now = options.now ?? Date.now();
|
||||
|
||||
if (!options.refresh) {
|
||||
const cached = await readCache(now);
|
||||
if (cached) return { status: "ok", place: cached };
|
||||
}
|
||||
|
||||
let response: Response;
|
||||
try {
|
||||
response = await fetchWithTimeout(GEO_PROVIDER, options.signal, GEO_TIMEOUT_MS);
|
||||
} catch (thrown) {
|
||||
return { status: "failed", message: `could not reach ${GEO_PROVIDER_NAME} (${errorMessage(thrown)})` };
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
return { status: "failed", message: `${GEO_PROVIDER_NAME} answered HTTP ${response.status}` };
|
||||
}
|
||||
|
||||
let payload: unknown;
|
||||
try {
|
||||
payload = await response.json();
|
||||
} catch {
|
||||
return { status: "failed", message: `${GEO_PROVIDER_NAME} did not answer with JSON` };
|
||||
}
|
||||
|
||||
const parsed = ResponseSchema.safeParse(payload);
|
||||
if (!parsed.success) {
|
||||
return { status: "failed", message: `${GEO_PROVIDER_NAME} answered in an unexpected shape` };
|
||||
}
|
||||
|
||||
const data = parsed.data;
|
||||
if (data.success === false) {
|
||||
return {
|
||||
status: "failed",
|
||||
message: data.message ?? `${GEO_PROVIDER_NAME} refused the lookup`,
|
||||
};
|
||||
}
|
||||
|
||||
const city = data.city ?? undefined;
|
||||
const region = data.region ?? undefined;
|
||||
const country = data.country ?? undefined;
|
||||
const place = buildPlace({ city, region, country });
|
||||
|
||||
if (!place) {
|
||||
return { status: "failed", message: `${GEO_PROVIDER_NAME} returned no usable place` };
|
||||
}
|
||||
|
||||
const timezone = typeof data.timezone === "string" ? data.timezone : data.timezone?.id;
|
||||
|
||||
const resolved: GeoPlace = {
|
||||
place,
|
||||
city,
|
||||
region,
|
||||
country,
|
||||
latitude: data.latitude ?? undefined,
|
||||
longitude: data.longitude ?? undefined,
|
||||
timezone: timezone ?? undefined,
|
||||
source: GEO_PROVIDER_NAME,
|
||||
fetchedAt: now,
|
||||
};
|
||||
|
||||
await writeCache(resolved);
|
||||
return { status: "ok", place: resolved };
|
||||
}
|
||||
|
||||
/** Drops the cached place, so a move or a provider change is not pinned for hours. */
|
||||
export async function forgetGeoPlace(): Promise<void> {
|
||||
await LocalStorage.removeItem(STORAGE_KEY).catch(() => undefined);
|
||||
}
|
||||
+14
-2
@@ -8,8 +8,17 @@ import type { ApiMessage, ChatTurn } from "./types";
|
||||
* messages from a previous turn are intentionally dropped: they are noise once
|
||||
* the answer is written, and replaying them would re-send large page contents
|
||||
* on every subsequent request.
|
||||
*
|
||||
* Attached images are treated the same way, and for the same reason. They are
|
||||
* the largest thing a request can carry, so only the question being asked now
|
||||
* brings one. A follow-up about a screenshot needs the image attached again.
|
||||
*/
|
||||
export function buildMessages(systemPrompt: string, turns: ChatTurn[], prompt: string): ApiMessage[] {
|
||||
export function buildMessages(
|
||||
systemPrompt: string,
|
||||
turns: ChatTurn[],
|
||||
prompt: string,
|
||||
images?: string[],
|
||||
): ApiMessage[] {
|
||||
const messages: ApiMessage[] = [{ role: "system", content: systemPrompt }];
|
||||
|
||||
for (const turn of turns) {
|
||||
@@ -18,7 +27,10 @@ export function buildMessages(systemPrompt: string, turns: ChatTurn[], prompt: s
|
||||
messages.push({ role: "assistant", content: turn.response });
|
||||
}
|
||||
|
||||
messages.push({ role: "user", content: prompt });
|
||||
const question: ApiMessage = { role: "user", content: prompt };
|
||||
if (images && images.length > 0) question.images = images;
|
||||
messages.push(question);
|
||||
|
||||
return messages;
|
||||
}
|
||||
|
||||
|
||||
@@ -1,11 +1,13 @@
|
||||
import { useCallback, useEffect, useRef, useState } from "react";
|
||||
import { runAgent } from "../agent";
|
||||
import { toStoredAttachment } from "../attachments";
|
||||
import { ipPlace } from "../geo";
|
||||
import { buildMessages, composeSystemPrompt } from "../history";
|
||||
import { OllamaError } from "../ollama";
|
||||
import type { ResolvedPreferences } from "../preferences";
|
||||
import { createSession, createTurn, saveSession, withTitleFromFirstTurn } from "../sessions";
|
||||
import { toolsFor } from "../tools";
|
||||
import type { Session } from "../types";
|
||||
import type { PendingAttachment, Session } from "../types";
|
||||
import { errorMessage, isAbortError } from "../utils";
|
||||
|
||||
/**
|
||||
@@ -41,7 +43,7 @@ export interface UseChat {
|
||||
/** Short activity line such as "Searching the web", shown while a turn is in flight. */
|
||||
status?: string;
|
||||
error?: string;
|
||||
ask: (prompt: string) => void;
|
||||
ask: (prompt: string, attachment?: PendingAttachment) => void;
|
||||
cancel: () => void;
|
||||
replaceSession: (session: Session) => void;
|
||||
newSession: () => void;
|
||||
@@ -119,16 +121,22 @@ export function useChat(options: UseChatOptions): UseChat {
|
||||
}, []);
|
||||
|
||||
const ask = useCallback(
|
||||
(rawPrompt: string) => {
|
||||
(rawPrompt: string, attachment?: PendingAttachment) => {
|
||||
const prompt = rawPrompt.trim();
|
||||
if (!prompt || liveRef.current) return;
|
||||
|
||||
const preferences = preferencesRef.current;
|
||||
const base = sessionRef.current;
|
||||
|
||||
// Only the path, mime type and dimensions are kept on the turn. The base64
|
||||
// payload is used for this one request and then dropped, so it never
|
||||
// reaches storage. The dimensions have to survive because the transcript
|
||||
// lays the preview out long after the payload is gone.
|
||||
const attachments = attachment ? [toStoredAttachment(attachment)] : [];
|
||||
|
||||
// The turn is inserted immediately with an empty response so the UI shows
|
||||
// the question while the answer streams in beneath it.
|
||||
const turn = createTurn(prompt, preferences.model);
|
||||
const turn = createTurn(prompt, preferences.model, Date.now(), attachments);
|
||||
const started = withTitleFromFirstTurn({
|
||||
...base,
|
||||
model: preferences.model,
|
||||
@@ -152,6 +160,7 @@ export function useChat(options: UseChatOptions): UseChat {
|
||||
composeSystemPrompt(preferences.systemPrompt, preferences.webEnabled),
|
||||
base.turns,
|
||||
prompt,
|
||||
attachment ? [attachment.base64] : undefined,
|
||||
);
|
||||
|
||||
void (async () => {
|
||||
@@ -166,7 +175,14 @@ export function useChat(options: UseChatOptions): UseChat {
|
||||
think: preferences.think,
|
||||
messages,
|
||||
tools: toolsFor({ webEnabled: preferences.webEnabled }),
|
||||
toolContext: { host: preferences.host, apiKey: preferences.apiKey },
|
||||
toolContext: {
|
||||
host: preferences.host,
|
||||
apiKey: preferences.apiKey,
|
||||
place: preferences.place,
|
||||
// Only wired up when the user switched it on, so an off setting
|
||||
// cannot reach the network for a location at all.
|
||||
ipPlace: preferences.ipGeolocation ? ipPlace : undefined,
|
||||
},
|
||||
signal: controller.signal,
|
||||
events: {
|
||||
onContentDelta: (delta) => {
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
import { useEffect, useState } from "react";
|
||||
import type { ResolvedPreferences } from "../preferences";
|
||||
import { visionSupport } from "../vision";
|
||||
|
||||
/**
|
||||
* Whether the current model accepts images, for gating the snip action.
|
||||
*
|
||||
* Fails closed: only a confirmed `"yes"` turns the feature on. A model that
|
||||
* cannot see would otherwise accept a screenshot and answer as though it had,
|
||||
* which is worse than the action not being offered.
|
||||
*
|
||||
* The wording for a *failed* check lives in the snip command rather than here,
|
||||
* because that is the only place with somewhere to say it (a HUD). This hook
|
||||
* only decides whether to show an action.
|
||||
*/
|
||||
|
||||
export interface VisionState {
|
||||
/** True only once Ollama has confirmed the model reports the `vision` capability. */
|
||||
canSeeImages: boolean;
|
||||
/** True while an answer for this model is still outstanding. */
|
||||
isChecking: boolean;
|
||||
}
|
||||
|
||||
export function useVisionSupport(preferences: ResolvedPreferences): VisionState {
|
||||
const [canSeeImages, setCanSeeImages] = useState(false);
|
||||
const [isChecking, setIsChecking] = useState(true);
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
const controller = new AbortController();
|
||||
|
||||
setIsChecking(true);
|
||||
|
||||
void visionSupport({
|
||||
host: preferences.host,
|
||||
apiKey: preferences.apiKey,
|
||||
model: preferences.model,
|
||||
signal: controller.signal,
|
||||
})
|
||||
.then((support) => {
|
||||
if (!cancelled) setCanSeeImages(support === "yes");
|
||||
})
|
||||
.finally(() => {
|
||||
if (!cancelled) setIsChecking(false);
|
||||
});
|
||||
|
||||
return () => {
|
||||
cancelled = true;
|
||||
controller.abort();
|
||||
};
|
||||
}, [preferences.host, preferences.apiKey, preferences.model]);
|
||||
|
||||
return { canSeeImages, isChecking };
|
||||
}
|
||||
+69
-13
@@ -1,30 +1,76 @@
|
||||
/**
|
||||
* Time and place, derived locally.
|
||||
*
|
||||
* `get_location` is intentionally timezone-derived rather than a GPS or IP
|
||||
* lookup. It answers the questions people actually ask a launcher ("what time
|
||||
* is it there", "is it late in London") without shipping the user's IP address
|
||||
* to a third-party geolocation service on every tool call. It is also pure, so
|
||||
* it is cheap to test.
|
||||
* `get_location` is timezone-derived rather than a GPS or IP lookup, so the
|
||||
* user's IP is never shipped to a third-party geolocation service on every tool
|
||||
* call. It is also pure, so it is cheap to test.
|
||||
*
|
||||
* The correction that matters here is that a timezone name is **not** a city.
|
||||
* The IANA database names a zone after one representative place, and a single
|
||||
* zone can cover a whole country: "Asia/Manila" is the Philippines,
|
||||
* "Asia/Kolkata" is India, "Europe/Oslo" is Norway, "Australia/Brisbane" is
|
||||
* Queensland. Reporting "Manila" as the user's city is simply wrong for anyone
|
||||
* in Cebu, and being confidently wrong is worse than being vague, because a
|
||||
* model asked about the weather will use it without hesitating. So the zone's
|
||||
* label is still reported, but named for what it is instead of being dressed up
|
||||
* as the user's city.
|
||||
*
|
||||
* The only exact answer available without asking a third party is the Location
|
||||
* preference, which the user fills in themselves. It is reported verbatim and
|
||||
* preferred over anything inferred.
|
||||
*/
|
||||
|
||||
export interface LocationContext {
|
||||
timezone: string;
|
||||
utcOffset: string;
|
||||
locale: string;
|
||||
/** Country code from the locale, e.g. "PH". The user's own setting rather than an inference. */
|
||||
regionCode?: string;
|
||||
/** Readable name for that region, e.g. "Philippines". */
|
||||
region?: string;
|
||||
city?: string;
|
||||
/**
|
||||
* The timezone's own area label, e.g. "Manila".
|
||||
*
|
||||
* Deliberately not called a city. See the note at the top of this file.
|
||||
*/
|
||||
timezoneArea?: string;
|
||||
/** The user's configured Location preference, verbatim. The only exact value here. */
|
||||
place?: string;
|
||||
}
|
||||
|
||||
/** Splits an IANA zone like "Asia/Manila" or "America/Argentina/Salta" into readable parts. */
|
||||
export function describeTimezone(timeZone: string): { region?: string; city?: string } {
|
||||
/**
|
||||
* Splits an IANA zone like "Asia/Manila" or "America/Argentina/Salta".
|
||||
*
|
||||
* Only the last segment is kept, and the continent grouping is dropped: "Asia"
|
||||
* next to a locale region of "Philippines" reads as two answers to the same
|
||||
* question, and the zone's grouping answers a question nobody asked.
|
||||
*/
|
||||
export function describeTimezoneArea(timeZone: string): { area?: string } {
|
||||
const parts = timeZone.split("/").filter(Boolean);
|
||||
if (parts.length < 2) return {};
|
||||
|
||||
const region = parts[0];
|
||||
const city = parts[parts.length - 1].replace(/_/g, " ");
|
||||
return { area: parts[parts.length - 1].replace(/_/g, " ") || undefined };
|
||||
}
|
||||
|
||||
return { region, city: city || undefined };
|
||||
/**
|
||||
* The region from the locale subtag.
|
||||
*
|
||||
* This is a declaration rather than an inference: a locale of "en-PH" says the
|
||||
* user set their region to the Philippines. That makes it a better answer than
|
||||
* anything read out of a zone name, and it costs nothing.
|
||||
*/
|
||||
export function localeRegion(locale: string): { code: string; name: string } | undefined {
|
||||
try {
|
||||
const code = new Intl.Locale(locale).region;
|
||||
if (!code) return undefined;
|
||||
|
||||
const name = new Intl.DisplayNames(["en"], { type: "region" }).of(code);
|
||||
return { code, name: name ?? code };
|
||||
} catch {
|
||||
// A malformed locale, or an ICU build without region display names. Not
|
||||
// worth failing the whole answer over.
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
/** "GMT+08:00" on a modern V8; falls back to a short name if `longOffset` is unavailable. */
|
||||
@@ -56,7 +102,11 @@ function resolveTimeZone(timeZone?: string): string {
|
||||
}
|
||||
}
|
||||
|
||||
export function getLocationContext(date: Date = new Date(), timeZone?: string): LocationContext {
|
||||
export function getLocationContext(
|
||||
date: Date = new Date(),
|
||||
timeZone?: string,
|
||||
place?: string,
|
||||
): LocationContext {
|
||||
const zone = resolveTimeZone(timeZone);
|
||||
|
||||
let locale = "en-US";
|
||||
@@ -66,11 +116,17 @@ export function getLocationContext(date: Date = new Date(), timeZone?: string):
|
||||
// Keep the fallback locale.
|
||||
}
|
||||
|
||||
const region = localeRegion(locale);
|
||||
const configured = place?.trim();
|
||||
|
||||
return {
|
||||
timezone: zone,
|
||||
utcOffset: utcOffsetLabel(date, zone),
|
||||
locale,
|
||||
...describeTimezone(zone),
|
||||
regionCode: region?.code,
|
||||
region: region?.name,
|
||||
timezoneArea: describeTimezoneArea(zone).area,
|
||||
place: configured && configured.length > 0 ? configured : undefined,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
+28
-3
@@ -1,5 +1,6 @@
|
||||
import { imageSource } from "./attachments";
|
||||
import { repairCodeBlocks } from "./code";
|
||||
import type { ChatTurn, ThreadMessage } from "./types";
|
||||
import type { Attachment, ChatTurn, ThreadMessage } from "./types";
|
||||
|
||||
/**
|
||||
* Markdown rendering for the detail panel.
|
||||
@@ -22,6 +23,24 @@ import type { ChatTurn, ThreadMessage } from "./types";
|
||||
|
||||
export const STREAMING_CURSOR = "▍";
|
||||
|
||||
/**
|
||||
* Renders attachments as markdown images.
|
||||
*
|
||||
* Vicinae does resolve a local file: `MdImage.qml` draws it block-level and
|
||||
* centred, uses the alt text as an italic caption underneath, and caps the
|
||||
* height at 200px no matter what is asked for. So this is a thumbnail rather
|
||||
* than something readable, which is why the action panel also offers Open
|
||||
* Screenshot for the full image.
|
||||
*
|
||||
* The alt text is short on purpose. Whatever is here is drawn centred under the
|
||||
* image, so a sentence reads as a stray line of prose floating in the middle of
|
||||
* the pane. It is also the only thing left if the file has since been deleted,
|
||||
* which is why it is not empty.
|
||||
*/
|
||||
export function renderAttachments(attachments: Attachment[]): string {
|
||||
return attachments.map((attachment) => `})`).join("\n\n");
|
||||
}
|
||||
|
||||
/** Which optional sections the transcript includes. Driven by preferences. */
|
||||
export interface TranscriptOptions {
|
||||
showToolCalls: boolean;
|
||||
@@ -49,7 +68,12 @@ function renderTools(message: ThreadMessage): string {
|
||||
|
||||
function renderMessage(message: ThreadMessage, options: TranscriptOptions): string {
|
||||
// The question is identified by its styling alone.
|
||||
if (message.role === "user") return quote(message.text);
|
||||
if (message.role === "user") {
|
||||
const images = renderAttachments(message.attachments);
|
||||
const asked = quote(message.text);
|
||||
|
||||
return images ? `${asked}\n\n${images}` : asked;
|
||||
}
|
||||
|
||||
// While a tool is running there is no answer text yet, so the activity line
|
||||
// stands in and the user can see what Lam is actually doing.
|
||||
@@ -138,9 +162,10 @@ export function sessionToPlainText(turns: ChatTurn[]): string {
|
||||
return turns
|
||||
.map((turn) => {
|
||||
const question = `**You**\n\n${turn.prompt}`;
|
||||
const attached = turn.attachments.length > 0 ? "\n\n_(screenshot attached)_" : "";
|
||||
const answer = turn.response.trim() || (turn.error ? "" : "_(no response)_");
|
||||
const failure = turn.error ? `\n\n**[error]** ${turn.error}` : "";
|
||||
return `${question}\n\n**Lam**\n\n${answer}${failure}`;
|
||||
return `${question}${attached}\n\n**Lam**\n\n${answer}${failure}`;
|
||||
})
|
||||
.join("\n\n---\n\n");
|
||||
}
|
||||
|
||||
@@ -219,6 +219,44 @@ export async function listModels(options: ListModelsOptions): Promise<ModelInfo[
|
||||
.sort((a, b) => a.name.localeCompare(b.name));
|
||||
}
|
||||
|
||||
export interface ModelCapabilitiesOptions {
|
||||
host: string;
|
||||
apiKey: string;
|
||||
model: string;
|
||||
signal?: AbortSignal;
|
||||
}
|
||||
|
||||
/**
|
||||
* The features a model reports for itself, straight from `POST /api/show`.
|
||||
*
|
||||
* `GET /api/tags` is not enough here: it returns names, sizes and quantization
|
||||
* levels and says nothing about what the model can actually do. The capability
|
||||
* list is what turns "image input" from a guess about the model's name into a
|
||||
* fact, so it is worth the extra request.
|
||||
*/
|
||||
export async function modelCapabilities(options: ModelCapabilitiesOptions): Promise<string[]> {
|
||||
const host = normalizeHost(options.host);
|
||||
|
||||
const response = await fetch(`${host}/api/show`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json", ...authHeaders(options.apiKey) },
|
||||
body: JSON.stringify({ model: options.model }),
|
||||
signal: options.signal,
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw describeHttpError(response.status, await safeText(response), host);
|
||||
}
|
||||
|
||||
const payload = (await response.json()) as { capabilities?: string[] };
|
||||
return payload.capabilities ?? [];
|
||||
}
|
||||
|
||||
/** Ollama reports image input as the `vision` capability. */
|
||||
export function supportsVision(capabilities: string[]): boolean {
|
||||
return capabilities.includes("vision");
|
||||
}
|
||||
|
||||
export interface WebSearchResult {
|
||||
title: string;
|
||||
url: string;
|
||||
|
||||
@@ -15,6 +15,8 @@ export interface LamPreferences {
|
||||
model?: string;
|
||||
ollamaHost?: string;
|
||||
systemPrompt?: string;
|
||||
location?: string;
|
||||
ipGeolocation?: boolean;
|
||||
webSearchEnabled?: boolean;
|
||||
thinking?: string;
|
||||
showToolCalls?: boolean;
|
||||
@@ -28,6 +30,16 @@ export interface ResolvedPreferences {
|
||||
model: string;
|
||||
host: string;
|
||||
systemPrompt: string;
|
||||
/**
|
||||
* The place the user says they are, verbatim, or an empty string.
|
||||
*
|
||||
* The only exact location this extension has. Everything else is inferred from
|
||||
* the device timezone and locale, which is accurate to a region rather than a
|
||||
* city, and a timezone name is not a city at all.
|
||||
*/
|
||||
place: string;
|
||||
/** Whether to resolve a rough place from the IP address. Off unless asked for. */
|
||||
ipGeolocation: boolean;
|
||||
webEnabled: boolean;
|
||||
/** Undefined when reasoning is off, which is what the API client expects. */
|
||||
think?: string;
|
||||
@@ -78,6 +90,10 @@ export function resolvePreferences(raw: LamPreferences): ResolvedPreferences {
|
||||
model: (raw.model ?? "").trim() || DEFAULT_MODEL,
|
||||
host: (raw.ollamaHost ?? "").trim() || DEFAULT_HOST,
|
||||
systemPrompt: (raw.systemPrompt ?? "").trim(),
|
||||
place: (raw.location ?? "").trim(),
|
||||
// Defaults to off: this is the one setting that sends something about the
|
||||
// user to a third party, so it is a deliberate choice rather than a default.
|
||||
ipGeolocation: raw.ipGeolocation ?? false,
|
||||
// Defaults to on: the preference is a checkbox the user has to actively untick.
|
||||
webEnabled: raw.webSearchEnabled ?? true,
|
||||
think: thinking && thinking !== "off" ? thinking : undefined,
|
||||
|
||||
@@ -14,6 +14,15 @@ export const ToolCallRecordSchema = z.object({
|
||||
isError: z.boolean(),
|
||||
});
|
||||
|
||||
export const AttachmentSchema = z.object({
|
||||
path: z.string(),
|
||||
mime: z.string().default("image/png"),
|
||||
// Absent on sessions written before the header was read, which is fine: a
|
||||
// preview without dimensions falls back to the renderer's own sizing.
|
||||
width: z.number().optional(),
|
||||
height: z.number().optional(),
|
||||
});
|
||||
|
||||
export const ChatTurnSchema = z.object({
|
||||
id: z.string(),
|
||||
prompt: z.string(),
|
||||
@@ -23,6 +32,9 @@ export const ChatTurnSchema = z.object({
|
||||
createdAt: z.number(),
|
||||
error: z.string().optional(),
|
||||
toolCalls: z.array(ToolCallRecordSchema).default([]),
|
||||
// Sessions written before image input existed have no attachments key, so the
|
||||
// default is what keeps them loading rather than being rejected as corrupt.
|
||||
attachments: z.array(AttachmentSchema).default([]),
|
||||
});
|
||||
|
||||
export const SessionSchema = z.object({
|
||||
|
||||
+8
-2
@@ -1,7 +1,7 @@
|
||||
import { LocalStorage } from "@vicinae/api";
|
||||
import { MAX_SESSIONS, MAX_STORED_CHARS_PER_TURN, MAX_TURNS_PER_SESSION, STORAGE_INDEX_KEY, STORAGE_SESSION_PREFIX, TITLE_MAX_LENGTH } from "./constants";
|
||||
import { SessionIndexSchema, SessionSchema } from "./schemas";
|
||||
import type { ChatTurn, Session, SessionSummary } from "./types";
|
||||
import type { Attachment, ChatTurn, Session, SessionSummary } from "./types";
|
||||
import { titleFromPrompt, truncate } from "./utils";
|
||||
|
||||
/**
|
||||
@@ -56,7 +56,12 @@ export function createSession(model: string, now: number = Date.now()): Session
|
||||
}
|
||||
|
||||
/** Creates the turn record for a prompt that is about to be sent. */
|
||||
export function createTurn(prompt: string, model: string, now: number = Date.now()): ChatTurn {
|
||||
export function createTurn(
|
||||
prompt: string,
|
||||
model: string,
|
||||
now: number = Date.now(),
|
||||
attachments: Attachment[] = [],
|
||||
): ChatTurn {
|
||||
return {
|
||||
id: newId(),
|
||||
prompt,
|
||||
@@ -65,6 +70,7 @@ export function createTurn(prompt: string, model: string, now: number = Date.now
|
||||
model,
|
||||
createdAt: now,
|
||||
toolCalls: [],
|
||||
attachments,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
import { environment } from "@vicinae/api";
|
||||
import { join } from "node:path";
|
||||
import { CapturePermissionError, NoCaptureToolError } from "./capture";
|
||||
import { errorMessage } from "./utils";
|
||||
|
||||
/**
|
||||
* The seam between the snip command and the chat command.
|
||||
*
|
||||
* The capture cannot happen inside the chat view. Vicinae shows the launcher
|
||||
* window when it activates a command, so a region selector started from a view
|
||||
* is dragging around a panel parked in the middle of the screen. Capturing from
|
||||
* a `no-view` command and handing the file over is what keeps the window out of
|
||||
* the way, which means the two commands need a small agreed contract.
|
||||
*
|
||||
* Imports Vicinae for `environment.supportPath` only, and nothing else in the
|
||||
* logic layer depends on it.
|
||||
*/
|
||||
|
||||
/** What the snip command passes to the chat command through `launchCommand`. */
|
||||
export interface SnipLaunchContext {
|
||||
/** Absolute path of the captured PNG. Absent when the capture failed. */
|
||||
snipPath?: string;
|
||||
/** Why a capture failed, so the chat can explain itself rather than look empty. */
|
||||
snipError?: string;
|
||||
/** The conversation the snip was started from, restored so it is not lost. */
|
||||
sessionId?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Snips live in the extension's own support directory rather than a temp
|
||||
* directory: a session stores the path, and a path that survives a reboot is the
|
||||
* difference between an old conversation showing its screenshot and pointing at
|
||||
* nothing.
|
||||
*/
|
||||
export function snipDirectory(): string {
|
||||
return join(environment.supportPath, "snips");
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads the handoff payload.
|
||||
*
|
||||
* A launch context is whatever the launching command put there, so every field
|
||||
* is checked rather than trusted: a wrong type would otherwise reach the file
|
||||
* system as a path.
|
||||
*/
|
||||
export function readSnipContext(launchContext: unknown): SnipLaunchContext {
|
||||
if (!launchContext || typeof launchContext !== "object") return {};
|
||||
|
||||
const { snipPath, snipError, sessionId } = launchContext as Record<string, unknown>;
|
||||
const asString = (value: unknown) => (typeof value === "string" && value.length > 0 ? value : undefined);
|
||||
|
||||
return {
|
||||
snipPath: asString(snipPath),
|
||||
snipError: asString(snipError),
|
||||
sessionId: asString(sessionId),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Turns a capture failure into something worth showing.
|
||||
*
|
||||
* The two failures that carry a hint are the only two the user can do anything
|
||||
* about, and the hints are useless if they are dropped: "no capture tool is
|
||||
* installed" and "the system is blocking screen capture" need completely
|
||||
* different responses from completely different places.
|
||||
*/
|
||||
export function describeCaptureFailure(thrown: unknown): string {
|
||||
const actionable =
|
||||
thrown instanceof CapturePermissionError || thrown instanceof NoCaptureToolError
|
||||
? thrown.hint
|
||||
: undefined;
|
||||
|
||||
return actionable ? `${errorMessage(thrown)} ${actionable}` : errorMessage(thrown);
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
import type { PendingAttachment, Session, SessionSummary } from "./types";
|
||||
import { errorMessage } from "./utils";
|
||||
|
||||
/**
|
||||
* Everything the chat view resolves before its first render.
|
||||
*
|
||||
* Pulled out of the component and made pure because this is the part that keeps
|
||||
* going wrong. Three things can happen on open, and all of them are
|
||||
* asynchronous: a launch hands over a captured file, it may also name the
|
||||
* conversation to return to, and an ordinary open resumes the most recent
|
||||
* conversation if it is fresh enough. Every path has to end in a rendered chat.
|
||||
*
|
||||
* As a plain function it is testable without mounting anything, which matters
|
||||
* because the alternative is finding out from a screenshot that a view is
|
||||
* spinning forever.
|
||||
*
|
||||
* Imports nothing from Vicinae, so it belongs to the logic layer.
|
||||
*/
|
||||
|
||||
export interface StartupInput {
|
||||
/** A conversation already in hand, which is how the sessions command pushes this view. */
|
||||
hasInitialSession: boolean;
|
||||
/** A conversation to reopen by id, handed over by the snip command. */
|
||||
initialSessionId?: string;
|
||||
/** A captured image to read in. */
|
||||
initialAttachmentPath?: string;
|
||||
/** Idle time past which an ordinary open starts fresh. Undefined means never time out. */
|
||||
resumeTimeoutMs?: number;
|
||||
/** Injected so the resume window is deterministic under test. */
|
||||
now: number;
|
||||
}
|
||||
|
||||
/** The side effects, injected so nothing here touches storage or the file system directly. */
|
||||
export interface StartupPorts {
|
||||
loadSession: (id: string) => Promise<Session | undefined>;
|
||||
listSessions: () => Promise<SessionSummary[]>;
|
||||
readAttachment: (path: string) => Promise<PendingAttachment>;
|
||||
}
|
||||
|
||||
export interface StartupResult {
|
||||
/** The conversation to show, when one was restored. */
|
||||
session?: Session;
|
||||
/** The image that was handed over, when one could be read. */
|
||||
attachment?: PendingAttachment;
|
||||
/** Why a handed-over image could not be read, so the view can say so. */
|
||||
attachmentError?: string;
|
||||
}
|
||||
|
||||
/** True when the launch handed something over, which also means "do not resume". */
|
||||
export function isHandoff(input: Pick<StartupInput, "initialSessionId" | "initialAttachmentPath">): boolean {
|
||||
return Boolean(input.initialSessionId || input.initialAttachmentPath);
|
||||
}
|
||||
|
||||
export async function resolveStartup(input: StartupInput, ports: StartupPorts): Promise<StartupResult> {
|
||||
const result: StartupResult = {};
|
||||
|
||||
if (input.initialSessionId) {
|
||||
// A hand-off names the conversation to return to, so that one is restored
|
||||
// rather than whichever chat happens to be most recent. By id matters
|
||||
// because taking a snip is not a turn, so nothing was saved in between.
|
||||
// A session that has since been deleted just means starting fresh.
|
||||
result.session = await ports.loadSession(input.initialSessionId).catch(() => undefined);
|
||||
}
|
||||
|
||||
if (input.initialAttachmentPath) {
|
||||
try {
|
||||
result.attachment = await ports.readAttachment(input.initialAttachmentPath);
|
||||
} catch (thrown) {
|
||||
// A missing file leaves an empty chat with an explanation, which beats
|
||||
// refusing to open at all.
|
||||
result.attachmentError = errorMessage(thrown);
|
||||
}
|
||||
}
|
||||
|
||||
// Nothing was handed over, so the idle-resume rule applies: reopening the
|
||||
// command continues the most recent conversation, but only if it was used
|
||||
// recently. The previous conversation is never removed either way, it just
|
||||
// drops down the sidebar.
|
||||
if (!input.hasInitialSession && !isHandoff(input)) {
|
||||
const latest = await ports
|
||||
.listSessions()
|
||||
.then((sessions) => sessions[0])
|
||||
.catch(() => undefined);
|
||||
|
||||
if (latest) {
|
||||
const idleMs = input.now - latest.updatedAt;
|
||||
const fresh = input.resumeTimeoutMs === undefined || idleMs <= input.resumeTimeoutMs;
|
||||
|
||||
if (fresh) {
|
||||
result.session = await ports.loadSession(latest.id).catch(() => undefined);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
@@ -43,6 +43,8 @@ export function buildThread(turns: ChatTurn[], live: LiveState): ThreadMessage[]
|
||||
// Only while there is nothing to read yet. Once tokens land, the
|
||||
// placeholder it feeds is gone anyway.
|
||||
status: isLive && text.length === 0 ? live.status : undefined,
|
||||
// Only a question carries attachments, so the answer never has one.
|
||||
attachments: [],
|
||||
});
|
||||
|
||||
messages.push({
|
||||
@@ -53,6 +55,7 @@ export function buildThread(turns: ChatTurn[], live: LiveState): ThreadMessage[]
|
||||
isStreaming: false,
|
||||
thinking: "",
|
||||
toolCalls: [],
|
||||
attachments: turn.attachments,
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
+76
-10
@@ -1,5 +1,6 @@
|
||||
import { formatNow, getLocationContext, weekdayName } from "./location";
|
||||
import { webFetch, webSearch } from "./ollama";
|
||||
import type { GeoLookup } from "./geo";
|
||||
import type { ToolDefinition } from "./types";
|
||||
import { errorMessage } from "./utils";
|
||||
|
||||
@@ -19,6 +20,21 @@ export interface ToolContext {
|
||||
signal?: AbortSignal;
|
||||
/** Injectable clock so tests are deterministic. */
|
||||
now?: Date;
|
||||
/**
|
||||
* The user's Location preference, if they set one.
|
||||
*
|
||||
* Injected rather than read here, because this module is deliberately free of
|
||||
* Vicinae imports and preferences arrive through that API.
|
||||
*/
|
||||
place?: string;
|
||||
/**
|
||||
* Resolves a rough place from the user's IP address, when that is switched on.
|
||||
*
|
||||
* Injected for the same reason as `place`, and absent when IP geolocation is
|
||||
* off, which is what keeps this module from being able to reach the network
|
||||
* for a location on its own.
|
||||
*/
|
||||
ipPlace?: (options: { signal?: AbortSignal }) => Promise<GeoLookup>;
|
||||
}
|
||||
|
||||
export interface ToolResult {
|
||||
@@ -45,7 +61,7 @@ export const TOOL_DEFINITIONS: ToolDefinition[] = [
|
||||
function: {
|
||||
name: "get_location",
|
||||
description:
|
||||
"Get the user's approximate location context: timezone, UTC offset, locale, region, and city. Use it for local questions such as weather, business hours, or travel timing. This is derived from the device timezone, not GPS, so treat it as a region rather than an address.",
|
||||
"Get where the user is, for local questions such as weather, business hours, or travel timing. Returns the place they configured in preferences, plus the timezone, UTC offset, locale and region. Without a configured place this is region-level, not an address, and the timezone area it reports is the timezone's own name rather than the user's city, so never treat it as one.",
|
||||
parameters: { type: "object", properties: {}, required: [] },
|
||||
},
|
||||
},
|
||||
@@ -140,7 +156,7 @@ export async function executeTool(
|
||||
return { content: describeNow(context.now), isError: false };
|
||||
|
||||
case "get_location":
|
||||
return { content: describeLocation(context.now), isError: false };
|
||||
return { content: await describeLocation(context), isError: false };
|
||||
|
||||
case "web_search":
|
||||
return await runWebSearch(args, context);
|
||||
@@ -172,18 +188,68 @@ function describeNow(now?: Date): string {
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
function describeLocation(now?: Date): string {
|
||||
const date = now ?? new Date();
|
||||
const location = getLocationContext(date);
|
||||
async function describeLocation(context: ToolContext): Promise<string> {
|
||||
const date = context.now ?? new Date();
|
||||
const configured = context.place?.trim() || undefined;
|
||||
|
||||
const lines = [
|
||||
// A place the user typed wins outright, and while it is set there is no reason
|
||||
// to hand their IP address to anyone.
|
||||
const lookup =
|
||||
configured || !context.ipPlace ? undefined : await context.ipPlace({ signal: context.signal });
|
||||
const ip = lookup?.status === "ok" ? lookup.place : undefined;
|
||||
|
||||
const location = getLocationContext(date, undefined, configured ?? ip?.place);
|
||||
|
||||
const lines: string[] = [];
|
||||
|
||||
// The place comes first because it is the only answer that is about where the
|
||||
// user actually is, rather than about which zone the clock is set to.
|
||||
if (location.place) lines.push(`Place: ${location.place}`);
|
||||
|
||||
lines.push(
|
||||
`Timezone: ${location.timezone}`,
|
||||
`UTC offset: ${location.utcOffset}`,
|
||||
`Locale: ${location.locale}`,
|
||||
];
|
||||
if (location.city) lines.push(`City: ${location.city}`);
|
||||
if (location.region) lines.push(`Region: ${location.region}`);
|
||||
lines.push("Note: derived from the device timezone. It indicates region, not a precise address.");
|
||||
);
|
||||
|
||||
if (location.region) {
|
||||
lines.push(`Region: ${location.region}${location.regionCode ? ` (${location.regionCode})` : ""}`);
|
||||
}
|
||||
if (location.timezoneArea) {
|
||||
lines.push(`Timezone area: ${location.timezoneArea}`);
|
||||
}
|
||||
|
||||
// A mismatch between the clock and the IP is worth surfacing: it usually means
|
||||
// a VPN or a device still set to somewhere the user has left.
|
||||
if (ip?.timezone && ip.timezone !== location.timezone) {
|
||||
lines.push(`IP-reported timezone: ${ip.timezone}, which differs from the device timezone`);
|
||||
}
|
||||
|
||||
if (configured) {
|
||||
lines.push(
|
||||
"Note: Place is the user's own setting, so prefer it for anything that depends on exactly where they are.",
|
||||
);
|
||||
} else if (ip) {
|
||||
lines.push(
|
||||
`Note: Place came from an IP address lookup via ${ip.source}, so treat it as a rough guess rather than the user's city. An IP address locates where the request left the ISP's network, which can be hundreds of kilometres away, and it can be out of date if they have moved since.`,
|
||||
);
|
||||
} else if (lookup?.status === "failed") {
|
||||
// Deliberately not the "no Location is set" wording below. A failed lookup
|
||||
// and an unset preference are different problems, and reporting the wrong
|
||||
// one sends the user looking in the wrong place. This is also the only
|
||||
// channel that reaches them: the model paraphrases, so it is told to say so.
|
||||
lines.push(
|
||||
`Note: IP geolocation is switched on but the lookup failed (${lookup.message}), so no place could be resolved. Tell the user this plainly. Setting Location in preferences avoids the lookup and the network entirely. Everything below is derived from the device timezone and locale, so it is region-level.`,
|
||||
);
|
||||
} else {
|
||||
lines.push(
|
||||
"Note: no Location is set in preferences. Everything above is derived from the device timezone and locale, so it is accurate to a region and not to a city. Ask which city they are in when an exact place matters.",
|
||||
);
|
||||
}
|
||||
|
||||
if (location.timezoneArea) {
|
||||
lines.push("The timezone area is the timezone's own name, not the user's city.");
|
||||
}
|
||||
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
@@ -37,8 +37,53 @@ export interface ApiMessage {
|
||||
tool_calls?: ToolCall[];
|
||||
/** Only set on `role: "tool"` messages, naming the tool that produced this content. */
|
||||
tool_name?: string;
|
||||
/**
|
||||
* Base64 image data, for vision models.
|
||||
*
|
||||
* Ollama only reads these on a `user` message (`gpt-oss` style tool roles have
|
||||
* no images). They are the one genuinely large thing in a request, so they are
|
||||
* attached to the question they belong to and never replayed later, the same
|
||||
* way tool output is not replayed.
|
||||
*/
|
||||
images?: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* A file the user attached to a question.
|
||||
*
|
||||
* Only the path is stored. A screenshot runs to megabytes, and
|
||||
* `MAX_STORED_CHARS_PER_TURN` exists precisely so one turn cannot dominate
|
||||
* LocalStorage, so the bytes stay on disk and are read when the turn is sent.
|
||||
*/
|
||||
export interface Attachment {
|
||||
/** Absolute path to the image on disk. */
|
||||
path: string;
|
||||
/** Sent to Ollama next to the base64 payload. */
|
||||
mime: string;
|
||||
/**
|
||||
* Pixel dimensions, read from the file's header.
|
||||
*
|
||||
* Kept because the transcript has to lay the preview out long after the file
|
||||
* was read, and Vicinae sizes an image's container to the request rather than
|
||||
* to the image, which leaves dead space unless the request has the right
|
||||
* aspect ratio. Absent for a format whose header we do not parse.
|
||||
*/
|
||||
width?: number;
|
||||
height?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* An attachment that has been read and is ready to send, held in the view while
|
||||
* the user types their question.
|
||||
*
|
||||
* The base64 payload lives here and nowhere else. It is needed for exactly one
|
||||
* request, so it is deliberately not part of `Attachment` and never reaches
|
||||
* storage.
|
||||
*/
|
||||
export interface PendingAttachment extends Attachment {
|
||||
/** Base64 of the image, with no `data:` URL prefix. Ollama wants it bare. */
|
||||
base64: string;
|
||||
}
|
||||
/** What a tool actually returned during a turn, kept for display and debugging. */
|
||||
export interface ToolCallRecord {
|
||||
name: string;
|
||||
@@ -58,6 +103,8 @@ export interface ChatTurn {
|
||||
/** Set when the turn failed, so a stored session can show what went wrong. */
|
||||
error?: string;
|
||||
toolCalls: ToolCallRecord[];
|
||||
/** Images the user attached, kept for the transcript. The bytes are not stored. */
|
||||
attachments: Attachment[];
|
||||
}
|
||||
|
||||
export interface Session {
|
||||
@@ -104,4 +151,6 @@ export interface ThreadMessage {
|
||||
error?: string;
|
||||
/** Activity line shown while this message is streaming, e.g. "Searching the web". */
|
||||
status?: string;
|
||||
/** Images attached to this question. Only ever non-empty on a user message. */
|
||||
attachments: Attachment[];
|
||||
}
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
import { LocalStorage } from "@vicinae/api";
|
||||
import { modelCapabilities, supportsVision } from "./ollama";
|
||||
|
||||
/**
|
||||
* The cached answer to "can this model see images?".
|
||||
*
|
||||
* `POST /api/show` is a round trip, and the answer only changes when the user
|
||||
* switches model. Without a cache the chat view would ask again on every mount
|
||||
* and the snip action would flicker in and out of the action panel while the
|
||||
* request was in flight.
|
||||
*
|
||||
* `"unknown"` is deliberately distinct from `"no"`. A failed request is not
|
||||
* evidence that a model cannot see, so the caller decides how much to trust it
|
||||
* and a failure is never written to the cache.
|
||||
*/
|
||||
|
||||
/** One key per host and model, so switching model invalidates by construction. */
|
||||
const VISION_CACHE_PREFIX = "lam.vision.";
|
||||
|
||||
export type VisionSupport = "yes" | "no" | "unknown";
|
||||
|
||||
export interface VisionCheck {
|
||||
host: string;
|
||||
apiKey: string;
|
||||
model: string;
|
||||
signal?: AbortSignal;
|
||||
/** Ignores the cache. Used when the answer might have gone stale. */
|
||||
refresh?: boolean;
|
||||
}
|
||||
|
||||
function cacheKey(host: string, model: string): string {
|
||||
return `${VISION_CACHE_PREFIX}${host}|${model}`;
|
||||
}
|
||||
|
||||
export async function visionSupport(check: VisionCheck): Promise<VisionSupport> {
|
||||
const key = cacheKey(check.host, check.model);
|
||||
|
||||
if (!check.refresh) {
|
||||
const cached = await LocalStorage.getItem<boolean>(key).catch(() => undefined);
|
||||
if (cached !== undefined) return cached ? "yes" : "no";
|
||||
}
|
||||
|
||||
try {
|
||||
const capable = supportsVision(
|
||||
await modelCapabilities({
|
||||
host: check.host,
|
||||
apiKey: check.apiKey,
|
||||
model: check.model,
|
||||
signal: check.signal,
|
||||
}),
|
||||
);
|
||||
|
||||
await LocalStorage.setItem(key, capable).catch(() => undefined);
|
||||
return capable ? "yes" : "no";
|
||||
} catch {
|
||||
return "unknown";
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Drops every cached answer.
|
||||
*
|
||||
* A model can gain vision without its name changing (a re-pull, a different
|
||||
* tag on the same account), so `Set Up Lam Agent` calls this rather than
|
||||
* leaving the user pinned to a stale "no".
|
||||
*/
|
||||
export async function forgetVisionSupport(): Promise<void> {
|
||||
const items = await LocalStorage.allItems().catch(() => ({}));
|
||||
|
||||
await Promise.all(
|
||||
Object.keys(items)
|
||||
.filter((key) => key.startsWith(VISION_CACHE_PREFIX))
|
||||
.map((key) => LocalStorage.removeItem(key).catch(() => undefined)),
|
||||
);
|
||||
}
|
||||
@@ -2,6 +2,8 @@ import { Action, ActionPanel, Icon, List, openExtensionPreferences } from "@vici
|
||||
import { useCallback, useEffect, useMemo, useState } from "react";
|
||||
import { describeResumeTimeout, getLamPreferences, needsSetup } from "./lib/preferences";
|
||||
import { listModels } from "./lib/ollama";
|
||||
import { forgetGeoPlace } from "./lib/geo";
|
||||
import { forgetVisionSupport, visionSupport, type VisionSupport } from "./lib/vision";
|
||||
import type { ModelInfo } from "./lib/types";
|
||||
import { errorMessage } from "./lib/utils";
|
||||
|
||||
@@ -36,6 +38,7 @@ export default function SetupCommand() {
|
||||
const [error, setError] = useState<string | undefined>(undefined);
|
||||
const [isLoading, setIsLoading] = useState(false);
|
||||
const [hasTested, setHasTested] = useState(false);
|
||||
const [vision, setVision] = useState<VisionSupport>("unknown");
|
||||
|
||||
const missingKey = needsSetup(preferences);
|
||||
|
||||
@@ -53,6 +56,23 @@ export default function SetupCommand() {
|
||||
setHasTested(true);
|
||||
setIsLoading(false);
|
||||
}
|
||||
|
||||
// A model can gain or lose image support without its name changing (a
|
||||
// re-pull, a retagged build), so re-testing drops the cached answer that the
|
||||
// chat view gates the snip action on rather than leaving a stale "no".
|
||||
await forgetVisionSupport();
|
||||
|
||||
// Same reasoning for the IP-derived place: a re-test is the moment to stop
|
||||
// trusting an answer that was resolved hours ago, possibly in another city.
|
||||
await forgetGeoPlace();
|
||||
setVision(
|
||||
await visionSupport({
|
||||
host: preferences.host,
|
||||
apiKey: preferences.apiKey,
|
||||
model: preferences.model,
|
||||
refresh: true,
|
||||
}),
|
||||
);
|
||||
}, [preferences]);
|
||||
|
||||
// Verifying on open means the user sees something real immediately rather
|
||||
@@ -149,6 +169,30 @@ export default function SetupCommand() {
|
||||
subtitle={describeResumeTimeout(preferences.resumeTimeoutMs)}
|
||||
actions={settingsActions}
|
||||
/>
|
||||
<List.Item
|
||||
icon={vision === "yes" ? Icon.Image : Icon.QuestionMarkCircle}
|
||||
title="Image Input"
|
||||
subtitle={
|
||||
vision === "yes"
|
||||
? `Supported by ${preferences.model}. Snip commands are available.`
|
||||
: vision === "no"
|
||||
? `${preferences.model} cannot see images. Pick a vision model to snip.`
|
||||
: "Not checked yet"
|
||||
}
|
||||
actions={settingsActions}
|
||||
/>
|
||||
<List.Item
|
||||
icon={preferences.place || preferences.ipGeolocation ? Icon.Person : Icon.QuestionMarkCircle}
|
||||
title="Location"
|
||||
subtitle={
|
||||
preferences.place
|
||||
? `${preferences.place} (from preferences)`
|
||||
: preferences.ipGeolocation
|
||||
? "Rough area, resolved from the IP address via ipwho.is"
|
||||
: "Region only, from the timezone and locale"
|
||||
}
|
||||
actions={settingsActions}
|
||||
/>
|
||||
</List.Section>
|
||||
|
||||
{error ? (
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
import { LaunchType, closeMainWindow, launchCommand, showHUD, type LaunchProps } from "@vicinae/api";
|
||||
import { captureRegion } from "./lib/capture";
|
||||
import { getLamPreferences } from "./lib/preferences";
|
||||
import { describeCaptureFailure, readSnipContext, snipDirectory } from "./lib/snip";
|
||||
import { visionSupport } from "./lib/vision";
|
||||
|
||||
/**
|
||||
* Vicinae's `closeMainWindow` defers the actual hide to a 50ms timer
|
||||
* (`closeWindow(settings, 50ms)` in `ui-service.hpp`), so a capture started the
|
||||
* instant the call resolves can still find the window on screen and select a
|
||||
* region around it. Waiting a little past that timer is the difference between
|
||||
* "the window is closed" and "the window is usually closed".
|
||||
*/
|
||||
const WINDOW_SETTLE_MS = 100;
|
||||
|
||||
function settleWindow(): Promise<void> {
|
||||
return new Promise((resolve) => setTimeout(resolve, WINDOW_SETTLE_MS));
|
||||
}
|
||||
|
||||
/**
|
||||
* Entrypoint for the "snip" command.
|
||||
*
|
||||
* `no-view`, and it has to be. Vicinae calls `showWindow()` when it activates a
|
||||
* command from a closed launcher, so the window is up by the time any extension
|
||||
* code runs; capturing from a view means selecting a region around a panel
|
||||
* sitting in the middle of the screen. This command closes the window first,
|
||||
* captures with nothing in the way, and only then hands the image to the chat,
|
||||
* which is the command that brings the window back.
|
||||
*
|
||||
* It is also the *only* capture path. The chat's own action launches this command
|
||||
* rather than capturing inline, so there is one implementation, one capability
|
||||
* check, and one place where the window is dealt with.
|
||||
*
|
||||
* Bind it to a global shortcut for one-key "snip and ask".
|
||||
*/
|
||||
export default async function SnipCommand(props: LaunchProps) {
|
||||
const preferences = getLamPreferences();
|
||||
const context = readSnipContext(props.launchContext);
|
||||
|
||||
// Before anything else. Everything below is pointless if the selector is
|
||||
// competing with a window for the middle of the screen.
|
||||
await closeMainWindow();
|
||||
await settleWindow();
|
||||
|
||||
// Settled before the capture, because taking a screenshot for a model that
|
||||
// cannot read it throws away the selection gesture the user just performed.
|
||||
// "unknown" says so in those words: a dropped check is not a verdict on the
|
||||
// model, and reporting it as one would send the user shopping for another.
|
||||
const vision = await visionSupport({
|
||||
host: preferences.host,
|
||||
apiKey: preferences.apiKey,
|
||||
model: preferences.model,
|
||||
});
|
||||
|
||||
if (vision !== "yes") {
|
||||
await showHUD(
|
||||
vision === "no"
|
||||
? `${preferences.model} cannot see images. Pick a vision model in preferences.`
|
||||
: `Could not check whether ${preferences.model} can see images. Run Set Up Lam Agent.`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
let snipPath: string | undefined;
|
||||
let snipError: string | undefined;
|
||||
|
||||
try {
|
||||
const capture = await captureRegion({ outputDir: snipDirectory() });
|
||||
snipPath = capture?.path;
|
||||
} catch (thrown) {
|
||||
snipError = describeCaptureFailure(thrown);
|
||||
}
|
||||
|
||||
// An abandoned selection says nothing at all: the user pressed Escape. But a
|
||||
// snip started from an open conversation has to come back, because the window
|
||||
// is now closed and the hotkey is otherwise the only way back.
|
||||
if (!snipPath && !snipError && !context.sessionId) return;
|
||||
|
||||
// Only the fields that exist are sent. `launchCommand` types its context as a
|
||||
// string map, and a key left undefined carries no information anyway.
|
||||
const payload: Record<string, string> = {};
|
||||
if (context.sessionId) payload.sessionId = context.sessionId;
|
||||
if (snipPath) payload.snipPath = snipPath;
|
||||
if (snipError) payload.snipError = snipError;
|
||||
|
||||
await launchCommand({
|
||||
name: "chat",
|
||||
type: LaunchType.UserInitiated,
|
||||
context: payload,
|
||||
});
|
||||
}
|
||||
Reference in New Issue
Block a user