fix: take the launcher window out of the shot when snipping
The snip was selecting a region around the launcher panel sitting in the middle of the screen. `activateEntrypoint` calls `showWindow()` when it activates a command from a closed launcher, so a view command's window is already up before any extension code runs. Capturing from inside the chat view could never have worked. - `snip` is now a `no-view` command. It calls `closeMainWindow()` first, waits out the 50ms hide that `ui-service.hpp` schedules, then captures with nothing on screen, then brings the window back by launching `chat` with the file. `settleWindow()` exists because a capture started the instant the close resolves could still find the window there. - The chat's action no longer captures inline. It launches `snip` and passes the session id, so there is one capture path, one capability check, and one place that deals with the window. - The hand-off relays through `snip` rather than going straight back to `chat` on purpose: `ActivateEntrypointOptions::toggleIfAlreadyActive` defaults to true, so launching the *already active* view command closes the window instead of reopening it. A launch cannot override that flag, but relaying through a different no-view command makes the toggle branch unreachable. - The action is hidden while streaming, since the hand-off unloads the view and would abandon an answer mid-sentence. - Vision is settled before the selector appears, so a model that cannot read images never costs a selection gesture. `no` and `unknown` get different HUD wording. - `lib/snip.ts` holds the hand-off contract; `lib/hooks/useSnip.ts` is gone, as nothing captures from a view any more. Verified: 252 tests, tsc, eslint, prettier, and vici build. The deployed manifest now reports `snip mode=no-view`.
This commit is contained in:
@@ -16,7 +16,7 @@ Hit a global shortcut, type a question, get a streamed answer without leaving wh
|
||||
- **Persistent conversations.** Every chat is saved locally. The sidebar lists them by recency, and reopening the command picks up where you left off, unless it has been idle longer than you allow.
|
||||
- **Readable code.** Fenced blocks render in a dark panel with the language named in a header, syntax colouring, and a Copy button.
|
||||
- **Tools.** Lam decides when to call them: current time, location context, web search, and page fetching.
|
||||
- **Screenshots, two ways.** Snip a region into the conversation you are already in from the action panel, or run **Snip to Lam** to start a new chat about it. Neither is offered unless the current model can actually see images.
|
||||
- **Screenshots, two ways.** Snip a region into the conversation you are already in from the action panel, or run **Snip to Lam** to start a new chat about it. The launcher window gets out of the way first, so it never ends up in the shot. Neither is offered unless the current model can actually see images.
|
||||
- **Chat only.** Lam answers questions. It does not touch your files or run commands. The one exception is capture, and it only ever happens because you pressed the key for it.
|
||||
|
||||
## Requirements
|
||||
@@ -106,7 +106,7 @@ vicinae 'vicinae://launch/lam-agent/chat?fallbackText=summarise%20this'
|
||||
| Command | Purpose |
|
||||
| --- | --- |
|
||||
| **Chat with Lam** | The chat surface. Bind this to your global shortcut. |
|
||||
| **Snip to Lam** | Capture a region and start a new chat about it. Bind this to a second shortcut for one-key "snip and ask". |
|
||||
| **Snip to Lam** | Capture a region and start a new chat about it. A `no-view` command, so the launcher window closes before the selector appears. Bind this to a second shortcut for one-key "snip and ask". |
|
||||
| **Chat Sessions** | Browse, reopen, and delete past conversations. |
|
||||
| **Set Up Lam Agent** | Verify the connection, list available models, and check whether the selected model can see images. |
|
||||
|
||||
@@ -144,7 +144,7 @@ Web search results and fetched pages can be large. Lam truncates them before the
|
||||
```
|
||||
src/
|
||||
chat.tsx # "chat" command entrypoint, thin wrapper
|
||||
snip.tsx # "snip" command entrypoint, captures then chats
|
||||
snip.tsx # "snip" command entrypoint, no-view: captures then hands over
|
||||
sessions.tsx # browse, reopen, delete conversations
|
||||
setup.tsx # connection check + live model list
|
||||
lib/
|
||||
@@ -162,8 +162,8 @@ src/
|
||||
code.ts # code fences and links, repaired
|
||||
preferences.ts # preferences, normalised
|
||||
hooks/useChat.ts # the chat state machine
|
||||
hooks/useSnip.ts # capture -> pending attachment
|
||||
hooks/useVision.ts # capability check, for gating snip actions
|
||||
hooks/useVision.ts # capability check, for gating the snip action
|
||||
snip.ts # the snip -> chat handoff contract
|
||||
components/ChatView.tsx # the chat surface
|
||||
```
|
||||
|
||||
@@ -197,7 +197,11 @@ A few decisions worth knowing about before you change things:
|
||||
|
||||
**Screen capture is owned, not borrowed.** There is no screenshot call anywhere in `@vicinae/api`, and `Clipboard` moves text, html, urls and a file path and nothing binary, so a bitmap can never be read back out of the clipboard either. The only way to get a region of the screen is to shell out, which is fair game because extensions run under Node. The useful consequence is that the extension picks the output path and reads the file it wrote, rather than guessing what some other application left on the clipboard. Cancel detection leans on the file for the same reason: every tool reports an abandoned selection differently, and at least one exits `0` on cancel, so whether an image actually landed on disk is the only signal they agree on. Snips go in the extension's support directory rather than a temp directory, because a stored conversation keeps the path and a path that survives a reboot is the difference between an old chat showing its screenshot and pointing at nothing.
|
||||
|
||||
**Image support is a fact from the API, not a guess from the model name, and it is cached.** `GET /api/tags` returns names, sizes and quantization levels and says nothing about capability, so the check is a `POST /api/show` that reads the `vision` flag. That is a round trip per model, so the answer is persisted under a key built from the host and model name, which means switching model invalidates it by construction. The state is a tri-state rather than a boolean: a failed request is `"unknown"`, never `"no"`, because telling someone their model cannot see when the network merely dropped is a false claim about their setup that sends them shopping for a model they do not need. The actions still fail closed, so only a confirmed `"yes"` offers a snip.
|
||||
**Capture runs in a `no-view` command, because the chat view owns a window.** This is the part that is easy to get wrong. `activateEntrypoint` calls `showWindow()` when it activates a command from a closed launcher, so the window is already on screen before any extension code runs, and a region selector started from a view is dragging a rectangle around a panel parked in the middle of the screen. So `snip` is declared `no-view`, calls `closeMainWindow()` as its first statement, captures with nothing in the way, and only then brings the window back by launching `chat` with the file. That command is also the single capture path: the chat's own action launches `snip` rather than capturing inline, so there is one implementation and one place that deals with the window.
|
||||
|
||||
The hand-off goes through `snip` specifically rather than straight to `chat` on purpose. `activateEntrypoint` treats a launch of the command that is *already active* as a toggle and closes the window instead of reopening it (`toggleIfAlreadyActive` defaults to true, and a launch cannot override it). Relaying through a different, no-view command means never asking for `chat` while `chat` is active, so the toggle branch is unreachable. The chat's own snip action is also hidden while a turn is streaming, because the hand-off unloads the view and would abandon an answer mid-sentence.
|
||||
|
||||
**Image support is a fact from the API, not a guess from the model name, and it is cached.** `GET /api/tags` returns names, sizes and quantization levels and says nothing about capability, so the check is a `POST /api/show` that reads the `vision` flag. That is a round trip per model, so the answer is persisted under a key built from the host and model name, which means switching model invalidates it by construction. The state is a tri-state rather than a boolean: a failed request is `"unknown"`, never `"no"`, because telling someone their model cannot see when the network merely dropped is a false claim about their setup that sends them shopping for a model they do not need. The snip command settles this *before* the selector appears, so a model that cannot read images never costs the user a selection gesture, and the two outcomes get different HUD wording. The in-chat action still fails closed: only a confirmed `"yes"` offers it.
|
||||
|
||||
**The capture is never a tool.** It stays in the action panel and on a keybinding, so it only ever runs because a person asked for it. A model that could decide to screenshot the screen is a different risk profile from one that answers questions, which is the same line the read-only tools already sit behind.
|
||||
|
||||
|
||||
+2
-2
@@ -19,8 +19,8 @@
|
||||
"name": "snip",
|
||||
"title": "Snip to Lam",
|
||||
"subtitle": "Lam Agent",
|
||||
"description": "Capture a region of the screen and start a new chat about it. Offered only when the model in preferences supports image input, so check with \"Set Up Lam Agent\" if the shortcut seems to do nothing.",
|
||||
"mode": "view"
|
||||
"description": "Capture a region of the screen and start a new chat about it. The launcher window closes first so it cannot end up in the shot. Offered only when the model in preferences supports image input.",
|
||||
"mode": "no-view"
|
||||
},
|
||||
{
|
||||
"name": "sessions",
|
||||
|
||||
@@ -46,3 +46,26 @@ export function __setPreferenceValues(values: Record<string, unknown>): void {
|
||||
export function getPreferenceValues<T = Record<string, unknown>>(): T {
|
||||
return preferenceOverride as T;
|
||||
}
|
||||
|
||||
/**
|
||||
* The runtime `environment`. Only `supportPath` is exercised, by the snip
|
||||
* directory, and a stable fake path keeps those assertions independent of where
|
||||
* the test happens to run.
|
||||
*/
|
||||
export const environment = {
|
||||
supportPath: "/tmp/lam-agent-support",
|
||||
assetsPath: "/tmp/lam-agent-assets",
|
||||
extensionName: "lam-agent",
|
||||
ownerOrAuthorName: "adrianbonpin",
|
||||
commandName: "chat",
|
||||
commandMode: "view" as const,
|
||||
isDevelopment: true,
|
||||
appearance: "dark" as const,
|
||||
theme: "dark" as const,
|
||||
textSize: "medium" as const,
|
||||
launchType: "userInitiated" as const,
|
||||
raycastVersion: "0.0.0",
|
||||
vicinaeVersion: { tag: "test", commit: "test" },
|
||||
isRaycast: false,
|
||||
canAccess: () => true,
|
||||
};
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { readSnipContext, snipDirectory } from "../lib/snip";
|
||||
|
||||
/**
|
||||
* The snip handoff crosses a command boundary as a serialized launch context, so
|
||||
* every field arrives as unknown. This is the boundary where a wrong type would
|
||||
* otherwise reach the file system as a path.
|
||||
*/
|
||||
|
||||
describe("readSnipContext", () => {
|
||||
it("reads the fields the snip command writes", () => {
|
||||
expect(
|
||||
readSnipContext({
|
||||
snipPath: "/tmp/snip.png",
|
||||
snipError: "no tool",
|
||||
sessionId: "abc123",
|
||||
}),
|
||||
).toEqual({ snipPath: "/tmp/snip.png", snipError: "no tool", sessionId: "abc123" });
|
||||
});
|
||||
|
||||
it("returns nothing for a launch with no context at all", () => {
|
||||
expect(readSnipContext(undefined)).toEqual({});
|
||||
expect(readSnipContext(null)).toEqual({});
|
||||
});
|
||||
|
||||
it("ignores a context that is not an object", () => {
|
||||
expect(readSnipContext("snip.png")).toEqual({});
|
||||
expect(readSnipContext(42)).toEqual({});
|
||||
expect(readSnipContext(["/tmp/snip.png"])).toEqual({});
|
||||
});
|
||||
|
||||
it("drops a value that is not a string rather than passing it on", () => {
|
||||
// A number or object here would reach `readAttachment` as a path.
|
||||
expect(readSnipContext({ snipPath: 42, sessionId: { id: "x" } })).toEqual({});
|
||||
});
|
||||
|
||||
it("treats an empty string as absent", () => {
|
||||
expect(readSnipContext({ snipPath: "", sessionId: "" })).toEqual({});
|
||||
});
|
||||
|
||||
it("ignores unknown keys, so a stray field cannot widen the contract", () => {
|
||||
expect(readSnipContext({ snipPath: "/tmp/snip.png", somethingElse: "/etc/passwd" })).toEqual({
|
||||
snipPath: "/tmp/snip.png",
|
||||
});
|
||||
});
|
||||
|
||||
it("reads a payload that only names a conversation, which is a cancelled snip", () => {
|
||||
expect(readSnipContext({ sessionId: "abc123" })).toEqual({ sessionId: "abc123" });
|
||||
});
|
||||
});
|
||||
|
||||
describe("snipDirectory", () => {
|
||||
it("keeps snips under the extension's support directory, not a temp directory", () => {
|
||||
// A stored conversation holds the path, so it has to survive a reboot.
|
||||
expect(snipDirectory()).toBe("/tmp/lam-agent-support/snips");
|
||||
});
|
||||
});
|
||||
+14
-1
@@ -1,5 +1,6 @@
|
||||
import { type LaunchProps } from "@vicinae/api";
|
||||
import { ChatView } from "./lib/components/ChatView";
|
||||
import { readSnipContext } from "./lib/snip";
|
||||
|
||||
/**
|
||||
* Entrypoint for the "chat" command.
|
||||
@@ -11,7 +12,19 @@ import { ChatView } from "./lib/components/ChatView";
|
||||
* launcher-and-start-typing behaviour. `fallbackText` is populated when the
|
||||
* command is opened through a deeplink carrying text, so
|
||||
* `vicinae://launch/lam-agent/chat?fallbackText=...` arrives pre-filled.
|
||||
*
|
||||
* It is also the command the `snip` command returns through, carrying a captured
|
||||
* file and, when the snip came from an open conversation, that conversation's id.
|
||||
*/
|
||||
export default function ChatCommand(props: LaunchProps) {
|
||||
return <ChatView initialPrompt={props.fallbackText} />;
|
||||
const snip = readSnipContext(props.launchContext);
|
||||
|
||||
return (
|
||||
<ChatView
|
||||
initialPrompt={props.fallbackText}
|
||||
initialSessionId={snip.sessionId}
|
||||
initialAttachmentPath={snip.snipPath}
|
||||
launchNote={snip.snipError ? `**Could not take a screenshot:** ${snip.snipError}` : undefined}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -1,14 +1,14 @@
|
||||
import { Action, ActionPanel, Icon, List } from "@vicinae/api";
|
||||
import { Action, ActionPanel, Icon, LaunchType, List, launchCommand } from "@vicinae/api";
|
||||
import { useCallback, useEffect, useMemo, useState } from "react";
|
||||
import { readAttachment } from "../attachments";
|
||||
import { useChat } from "../hooks/useChat";
|
||||
import { useSnip } from "../hooks/useSnip";
|
||||
import { useVisionSupport } from "../hooks/useVision";
|
||||
import { buildThreadMarkdown, type TranscriptOptions } from "../markdown";
|
||||
import { getLamPreferences } from "../preferences";
|
||||
import { deleteSession, listSessions, loadSession, summarizeSession } from "../sessions";
|
||||
import { buildThread } from "../thread";
|
||||
import type { PendingAttachment, Session, SessionSummary } from "../types";
|
||||
import { recencyGroup, truncate } from "../utils";
|
||||
import { errorMessage, recencyGroup, truncate } from "../utils";
|
||||
|
||||
/**
|
||||
* The chat surface: conversations on the left, the open conversation on the right.
|
||||
@@ -47,6 +47,9 @@ import { recencyGroup, truncate } from "../utils";
|
||||
* selection to the first row on every model change
|
||||
* (`GenericListView.selectFirstOnReset`), and streaming changes the model on
|
||||
* every flush, so ordering by recency keeps the chat in progress near the top.
|
||||
* - Capturing a region does **not** happen here. It is delegated to the `snip`
|
||||
* command, because this view owns a window and the window would be in the
|
||||
* shot. See {@link ChatViewProps}.
|
||||
*/
|
||||
|
||||
const EMPTY_TRANSCRIPT = "**Ask Lam anything**\n\nType a question in the search bar and press Return.";
|
||||
@@ -54,7 +57,7 @@ const EMPTY_TRANSCRIPT = "**Ask Lam anything**\n\nType a question in the search
|
||||
/** Sent when a snip is attached and Return is pressed before anything is typed. */
|
||||
const DEFAULT_IMAGE_PROMPT = "What is in this image?";
|
||||
|
||||
/** The reminder shown while a snip is captured but not yet sent. */
|
||||
/** The reminder shown while a snip is attached but not yet sent. */
|
||||
const PENDING_NOTE = "_Screenshot attached. Ask a question about it, or press Return to send it as it is._";
|
||||
|
||||
export interface ChatViewProps {
|
||||
@@ -62,10 +65,12 @@ export interface ChatViewProps {
|
||||
initialSession?: Session;
|
||||
/** Text handed over by a deeplink or fallback command. */
|
||||
initialPrompt?: string;
|
||||
/** A snip that has already been captured, used when a view is pushed with one. */
|
||||
initialAttachment?: PendingAttachment;
|
||||
/** Captures a snip on mount, which is what the standalone command does. */
|
||||
startWithSnip?: boolean;
|
||||
/** A conversation to reload by id, which is how the snip command hands one back. */
|
||||
initialSessionId?: string;
|
||||
/** A captured image to read in, which is how the snip command hands one over. */
|
||||
initialAttachmentPath?: string;
|
||||
/** Something the launching command needs to say, such as a failed capture. */
|
||||
launchNote?: string;
|
||||
}
|
||||
|
||||
interface SessionRow {
|
||||
@@ -75,27 +80,28 @@ interface SessionRow {
|
||||
isActive: boolean;
|
||||
}
|
||||
|
||||
export function ChatView({ initialSession, initialPrompt, initialAttachment, startWithSnip }: ChatViewProps) {
|
||||
export function ChatView({
|
||||
initialSession,
|
||||
initialPrompt,
|
||||
initialSessionId,
|
||||
initialAttachmentPath,
|
||||
launchNote,
|
||||
}: ChatViewProps) {
|
||||
const preferences = useMemo(() => getLamPreferences(), []);
|
||||
const chat = useChat({ preferences, initialSession });
|
||||
const { replaceSession } = chat;
|
||||
const { isSniping, snip } = useSnip();
|
||||
const { canSeeImages } = useVisionSupport(preferences);
|
||||
|
||||
const {
|
||||
support: vision,
|
||||
canSeeImages,
|
||||
isChecking: isCheckingVision,
|
||||
recheck,
|
||||
} = useVisionSupport(preferences);
|
||||
/** A launch that hands something over has to wait for it before it can render. */
|
||||
const needsPreparation = Boolean(initialSessionId) || Boolean(initialAttachmentPath);
|
||||
|
||||
const [input, setInput] = useState(initialPrompt ?? "");
|
||||
const [summaries, setSummaries] = useState<SessionSummary[] | undefined>(undefined);
|
||||
// A snip command starts a new chat on purpose, so the idle-resume below is
|
||||
// skipped and the view must not wait for it.
|
||||
const [resumed, setResumed] = useState(Boolean(initialSession) || Boolean(startWithSnip));
|
||||
const [prepared, setPrepared] = useState(!needsPreparation);
|
||||
const [resumed, setResumed] = useState(Boolean(initialSession));
|
||||
// A captured image is held here until the user says what they want to know
|
||||
// about it. Nothing is sent to the model on capture alone.
|
||||
const [pending, setPending] = useState<PendingAttachment | undefined>(initialAttachment);
|
||||
const [pending, setPending] = useState<PendingAttachment | undefined>(undefined);
|
||||
const [snipNote, setSnipNote] = useState<string | undefined>(undefined);
|
||||
|
||||
const activeId = chat.session.id;
|
||||
@@ -111,18 +117,55 @@ export function ChatView({ initialSession, initialPrompt, initialAttachment, sta
|
||||
reload();
|
||||
}, [reload, chat.session.updatedAt]);
|
||||
|
||||
/**
|
||||
* Restores whatever the launching command handed over.
|
||||
*
|
||||
* This is the second half of the snip handoff: the capture command closed the
|
||||
* window to take the shot, so it comes back here with a file to read and, when
|
||||
* the snip was started from a conversation, the id of the conversation to
|
||||
* return to. Reloading by id rather than trusting "most recent" matters
|
||||
* because the snip itself is not a turn, so nothing was saved in between.
|
||||
*/
|
||||
useEffect(() => {
|
||||
if (!needsPreparation) return;
|
||||
|
||||
let cancelled = false;
|
||||
|
||||
void (async () => {
|
||||
if (initialSessionId) {
|
||||
const session = await loadSession(initialSessionId);
|
||||
if (!cancelled && session) replaceSession(session);
|
||||
}
|
||||
|
||||
if (initialAttachmentPath) {
|
||||
try {
|
||||
const attachment = await readAttachment(initialAttachmentPath);
|
||||
if (!cancelled) setPending(attachment);
|
||||
} catch (thrown) {
|
||||
if (!cancelled) setSnipNote(`**Could not open the screenshot:** ${errorMessage(thrown)}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (!cancelled) setPrepared(true);
|
||||
})();
|
||||
|
||||
return () => {
|
||||
cancelled = true;
|
||||
};
|
||||
}, [needsPreparation, initialSessionId, initialAttachmentPath, replaceSession]);
|
||||
|
||||
/**
|
||||
* Reopening the command continues the most recent conversation, but only if it
|
||||
* was used recently. Past the configured idle window it starts a fresh chat
|
||||
* instead. The previous conversation is never removed, it just drops down the
|
||||
* sidebar. Setting the preference to "Never" keeps the last chat open forever.
|
||||
*
|
||||
* The snip command opts out entirely: it exists to start a new chat about one
|
||||
* screenshot, and quietly continuing the previous conversation would bury the
|
||||
* image in an unrelated thread.
|
||||
* A hand-off launch opts out: it either names the conversation to return to or
|
||||
* deliberately starts a new one, and quietly continuing some other chat would
|
||||
* bury the screenshot in an unrelated thread.
|
||||
*/
|
||||
useEffect(() => {
|
||||
if (initialSession || startWithSnip) return;
|
||||
if (initialSession || needsPreparation) return;
|
||||
|
||||
let cancelled = false;
|
||||
void (async () => {
|
||||
@@ -144,7 +187,7 @@ export function ChatView({ initialSession, initialPrompt, initialAttachment, sta
|
||||
return () => {
|
||||
cancelled = true;
|
||||
};
|
||||
}, [initialSession, startWithSnip, preferences, replaceSession]);
|
||||
}, [initialSession, needsPreparation, preferences, replaceSession]);
|
||||
|
||||
const thread = useMemo(
|
||||
() =>
|
||||
@@ -170,37 +213,15 @@ export function ChatView({ initialSession, initialPrompt, initialAttachment, sta
|
||||
* already looking.
|
||||
*/
|
||||
const threadMarkdown = useMemo(() => {
|
||||
const notes: string[] = [];
|
||||
|
||||
// Only the standalone command has to explain a model that cannot see; inside
|
||||
// the chat the action is simply absent. A failed check is worded separately
|
||||
// from a text-only model, because saying "your model cannot see" when the
|
||||
// request merely failed is a claim about the user's setup that is not true.
|
||||
if (startWithSnip && !isCheckingVision && vision !== "yes") {
|
||||
notes.push(
|
||||
vision === "no"
|
||||
? `**${preferences.model} cannot see images.** Choose a model with vision support in preferences, and this command becomes a one-key snip.`
|
||||
: `**Could not check whether ${preferences.model} can see images.** Use Check Again below, or snip from the chat's action panel once the connection is back.`,
|
||||
const notes = [launchNote, snipNote, pending ? PENDING_NOTE : undefined].filter(
|
||||
(note): note is string => Boolean(note),
|
||||
);
|
||||
}
|
||||
|
||||
if (snipNote) notes.push(snipNote);
|
||||
if (pending) notes.push(PENDING_NOTE);
|
||||
|
||||
const transcript =
|
||||
thread.length === 0 ? EMPTY_TRANSCRIPT : buildThreadMarkdown(thread, transcriptOptions);
|
||||
|
||||
return [...notes, transcript].join("\n\n");
|
||||
}, [
|
||||
startWithSnip,
|
||||
isCheckingVision,
|
||||
vision,
|
||||
preferences.model,
|
||||
snipNote,
|
||||
pending,
|
||||
thread,
|
||||
transcriptOptions,
|
||||
]);
|
||||
}, [launchNote, snipNote, pending, thread, transcriptOptions]);
|
||||
|
||||
const rows: SessionRow[] = useMemo(() => {
|
||||
const base = summaries ?? [];
|
||||
@@ -259,30 +280,6 @@ export function ChatView({ initialSession, initialPrompt, initialAttachment, sta
|
||||
[activeId, chat, reload],
|
||||
);
|
||||
|
||||
/**
|
||||
* Captures a region and holds it until a question is sent with it.
|
||||
*
|
||||
* A cancelled selection deliberately says nothing: the user pressed Escape and
|
||||
* telling them so is noise. A failed capture has to explain itself, because
|
||||
* the usual cause is a capture tool that is not installed.
|
||||
*/
|
||||
const takeSnip = useCallback(async () => {
|
||||
setSnipNote(undefined);
|
||||
const outcome = await snip();
|
||||
|
||||
if (outcome.status === "captured") setPending(outcome.attachment);
|
||||
else if (outcome.status === "failed")
|
||||
setSnipNote(`**Could not take a screenshot:** ${outcome.message}`);
|
||||
}, [snip]);
|
||||
|
||||
// The standalone command captures on arrival. Waiting for the capability check
|
||||
// matters here: capturing a screenshot only to be told the model cannot read
|
||||
// it would throw away the selection the user just made.
|
||||
useEffect(() => {
|
||||
if (!startWithSnip || !canSeeImages || initialAttachment) return;
|
||||
void takeSnip();
|
||||
}, [startWithSnip, canSeeImages, initialAttachment, takeSnip]);
|
||||
|
||||
/** The snip a "show me the screenshot" action would open: the pending one, else the newest. */
|
||||
const latestAttachment =
|
||||
pending ??
|
||||
@@ -320,18 +317,31 @@ export function ChatView({ initialSession, initialPrompt, initialAttachment, sta
|
||||
);
|
||||
|
||||
/**
|
||||
* Present only when Ollama has confirmed the model reports `vision`.
|
||||
* Hands the capture to the `snip` command instead of doing it here.
|
||||
*
|
||||
* Offering it otherwise would let the user frame a careful screenshot and
|
||||
* then have a model that cannot see answer as though it had, which is a worse
|
||||
* failure than the action not being there.
|
||||
* Two reasons, both structural. The window has to be out of the way, and only
|
||||
* the snip command is positioned to close it before the selector appears. And
|
||||
* launching a different command is what makes the hand-off safe: Vicinae's
|
||||
* `activateEntrypoint` treats a launch of the *currently active* view command
|
||||
* as a toggle and closes the window instead of reopening it, so relaying
|
||||
* through `snip` avoids ever asking for `chat` while `chat` is active.
|
||||
*
|
||||
* Absent while streaming, because the hand-off unloads this view and would
|
||||
* abandon an answer mid-sentence.
|
||||
*/
|
||||
const snipAction = canSeeImages ? (
|
||||
const snipAction =
|
||||
canSeeImages && !chat.isStreaming ? (
|
||||
<Action
|
||||
title="Snip a Region"
|
||||
icon={Icon.Screenshot}
|
||||
shortcut={{ key: "s", modifiers: ["cmd", "shift"] }}
|
||||
onAction={() => void takeSnip()}
|
||||
onAction={() =>
|
||||
void launchCommand({
|
||||
name: "snip",
|
||||
type: LaunchType.UserInitiated,
|
||||
context: { sessionId: chat.session.id },
|
||||
})
|
||||
}
|
||||
/>
|
||||
) : null;
|
||||
|
||||
@@ -341,13 +351,6 @@ export function ChatView({ initialSession, initialPrompt, initialAttachment, sta
|
||||
<Action.Open title="Open Screenshot" icon={Icon.Image} target={latestAttachment.path} />
|
||||
) : null;
|
||||
|
||||
// Only the standalone command reaches a state with nowhere to go: a failed
|
||||
// capability check leaves it with no snip and no action that would start one.
|
||||
const checkAgainAction =
|
||||
startWithSnip && !isCheckingVision && vision === "unknown" ? (
|
||||
<Action title="Check Again" icon={Icon.ArrowClockwise} onAction={recheck} />
|
||||
) : null;
|
||||
|
||||
/**
|
||||
* The primary action adapts to the search bar. With text in it, Return sends;
|
||||
* with it empty (where sending would be a no-op anyway), Return opens the
|
||||
@@ -366,7 +369,6 @@ export function ChatView({ initialSession, initialPrompt, initialAttachment, sta
|
||||
{newChatAction}
|
||||
{snipAction}
|
||||
{openSnipAction}
|
||||
{checkAgainAction}
|
||||
<Action
|
||||
title="Delete Conversation"
|
||||
icon={Icon.Trash}
|
||||
@@ -380,12 +382,10 @@ export function ChatView({ initialSession, initialPrompt, initialAttachment, sta
|
||||
);
|
||||
};
|
||||
|
||||
if (!resumed || summaries === undefined) {
|
||||
if (!resumed || !prepared || summaries === undefined) {
|
||||
return <List isLoading navigationTitle="Lam" searchBarPlaceholder="Loading conversations…" />;
|
||||
}
|
||||
|
||||
const busy = chat.isStreaming || isSniping;
|
||||
|
||||
const placeholder = chat.status
|
||||
? `${chat.status}. Press ⏎ to stop`
|
||||
: "Lam is thinking. Press ⏎ to stop";
|
||||
@@ -394,7 +394,7 @@ export function ChatView({ initialSession, initialPrompt, initialAttachment, sta
|
||||
|
||||
return (
|
||||
<List
|
||||
isLoading={busy}
|
||||
isLoading={chat.isStreaming}
|
||||
isShowingDetail
|
||||
filtering={false}
|
||||
searchText={input}
|
||||
@@ -402,16 +402,13 @@ export function ChatView({ initialSession, initialPrompt, initialAttachment, sta
|
||||
// Raycast puts the conversation title in a header; Vicinae's nearest
|
||||
// equivalent is the status-bar title, so the open chat is named there too.
|
||||
navigationTitle={truncate(chat.session.title, 40)}
|
||||
searchBarPlaceholder={
|
||||
chat.isStreaming ? placeholder : isSniping ? "Select a region…" : idlePlaceholder
|
||||
}
|
||||
searchBarPlaceholder={chat.isStreaming ? placeholder : idlePlaceholder}
|
||||
actions={
|
||||
<ActionPanel>
|
||||
{chat.isStreaming ? stopAction : sendAction}
|
||||
{newChatAction}
|
||||
{snipAction}
|
||||
{openSnipAction}
|
||||
{checkAgainAction}
|
||||
</ActionPanel>
|
||||
}
|
||||
>
|
||||
@@ -426,7 +423,7 @@ export function ChatView({ initialSession, initialPrompt, initialAttachment, sta
|
||||
// Title only. Raycast's sidebar shows nothing else, and stacking a
|
||||
// subtitle plus a timestamp on a 38px row elided the title down to
|
||||
// a few characters. The check icon is what marks the open chat.
|
||||
detail={<List.Item.Detail markdown={threadMarkdown} isLoading={busy} />}
|
||||
detail={<List.Item.Detail markdown={threadMarkdown} isLoading={chat.isStreaming} />}
|
||||
actions={rowActions(row)}
|
||||
/>
|
||||
))}
|
||||
|
||||
@@ -1,61 +0,0 @@
|
||||
import { environment } from "@vicinae/api";
|
||||
import { join } from "node:path";
|
||||
import { useCallback, useState } from "react";
|
||||
import { readAttachment } from "../attachments";
|
||||
import { NoCaptureToolError, captureRegion } from "../capture";
|
||||
import type { PendingAttachment } from "../types";
|
||||
import { errorMessage } from "../utils";
|
||||
|
||||
/**
|
||||
* Taking a snip, from the UI's point of view.
|
||||
*
|
||||
* The outcome is a discriminated union rather than `null`, because "the user
|
||||
* pressed Escape" and "there is no capture tool on this machine" want opposite
|
||||
* responses: the first should say nothing at all, the second has to explain
|
||||
* itself. A hook rather than a bare function so both entry points (the chat
|
||||
* action and the standalone command) share one implementation and one spinner.
|
||||
*/
|
||||
|
||||
export type SnipOutcome =
|
||||
| { status: "captured"; attachment: PendingAttachment }
|
||||
| { status: "cancelled" }
|
||||
| { status: "failed"; message: string };
|
||||
|
||||
export interface UseSnip {
|
||||
isSniping: boolean;
|
||||
snip: () => Promise<SnipOutcome>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Snips live in the extension's own support directory rather than a temp
|
||||
* directory: a session stores the path, and a path that survives a reboot is
|
||||
* the difference between an old conversation showing its screenshot and
|
||||
* referring to a file that no longer exists.
|
||||
*/
|
||||
export function snipDirectory(): string {
|
||||
return join(environment.supportPath, "snips");
|
||||
}
|
||||
|
||||
export function useSnip(): UseSnip {
|
||||
const [isSniping, setIsSniping] = useState(false);
|
||||
|
||||
const snip = useCallback(async (): Promise<SnipOutcome> => {
|
||||
setIsSniping(true);
|
||||
|
||||
try {
|
||||
const capture = await captureRegion({ outputDir: snipDirectory() });
|
||||
|
||||
// No image on disk means the region selection was abandoned.
|
||||
if (!capture) return { status: "cancelled" };
|
||||
|
||||
return { status: "captured", attachment: await readAttachment(capture.path) };
|
||||
} catch (thrown) {
|
||||
const hint = thrown instanceof NoCaptureToolError ? ` ${thrown.hint}` : "";
|
||||
return { status: "failed", message: `${errorMessage(thrown)}${hint}` };
|
||||
} finally {
|
||||
setIsSniping(false);
|
||||
}
|
||||
}, []);
|
||||
|
||||
return { isSniping, snip };
|
||||
}
|
||||
+14
-33
@@ -1,40 +1,29 @@
|
||||
import { useCallback, useEffect, useRef, useState } from "react";
|
||||
import { useEffect, useState } from "react";
|
||||
import type { ResolvedPreferences } from "../preferences";
|
||||
import { visionSupport, type VisionSupport } from "../vision";
|
||||
import { visionSupport } from "../vision";
|
||||
|
||||
/**
|
||||
* Whether the current model accepts images, for gating the snip actions.
|
||||
* Whether the current model accepts images, for gating the snip action.
|
||||
*
|
||||
* The state is the same tri-state the cached lookup returns, not a boolean. A
|
||||
* failed check and a text-only model want different words: claiming a model
|
||||
* "cannot see images" when the request merely failed is a false statement about
|
||||
* the user's setup, and it sends them looking for a different model for no
|
||||
* reason.
|
||||
* Fails closed: only a confirmed `"yes"` turns the feature on. A model that
|
||||
* cannot see would otherwise accept a screenshot and answer as though it had,
|
||||
* which is worse than the action not being offered.
|
||||
*
|
||||
* The actions themselves still fail closed: only a confirmed `"yes"` offers a
|
||||
* snip.
|
||||
* The wording for a *failed* check lives in the snip command rather than here,
|
||||
* because that is the only place with somewhere to say it (a HUD). This hook
|
||||
* only decides whether to show an action.
|
||||
*/
|
||||
|
||||
export interface VisionState {
|
||||
/** `"unknown"` means the check failed, which is not the same as `"no"`. */
|
||||
support: VisionSupport;
|
||||
/** True only once Ollama has confirmed the model reports the `vision` capability. */
|
||||
canSeeImages: boolean;
|
||||
/** True while an answer for this model is still outstanding. */
|
||||
isChecking: boolean;
|
||||
/** Asks again, ignoring the cache. */
|
||||
recheck: () => void;
|
||||
}
|
||||
|
||||
export function useVisionSupport(preferences: ResolvedPreferences): VisionState {
|
||||
const [support, setSupport] = useState<VisionSupport>("unknown");
|
||||
const [canSeeImages, setCanSeeImages] = useState(false);
|
||||
const [isChecking, setIsChecking] = useState(true);
|
||||
const [attempt, setAttempt] = useState(0);
|
||||
|
||||
// Set by `recheck` and consumed by the next run. A ref rather than a
|
||||
// dependency so that a manual retry does not also make every later model
|
||||
// change bypass the cache.
|
||||
const refreshRef = useRef(false);
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
@@ -47,14 +36,11 @@ export function useVisionSupport(preferences: ResolvedPreferences): VisionState
|
||||
apiKey: preferences.apiKey,
|
||||
model: preferences.model,
|
||||
signal: controller.signal,
|
||||
refresh: refreshRef.current,
|
||||
})
|
||||
.then((result) => {
|
||||
if (cancelled) return;
|
||||
setSupport(result);
|
||||
.then((support) => {
|
||||
if (!cancelled) setCanSeeImages(support === "yes");
|
||||
})
|
||||
.finally(() => {
|
||||
refreshRef.current = false;
|
||||
if (!cancelled) setIsChecking(false);
|
||||
});
|
||||
|
||||
@@ -62,12 +48,7 @@ export function useVisionSupport(preferences: ResolvedPreferences): VisionState
|
||||
cancelled = true;
|
||||
controller.abort();
|
||||
};
|
||||
}, [preferences.host, preferences.apiKey, preferences.model, attempt]);
|
||||
}, [preferences.host, preferences.apiKey, preferences.model]);
|
||||
|
||||
const recheck = useCallback(() => {
|
||||
refreshRef.current = true;
|
||||
setAttempt((value) => value + 1);
|
||||
}, []);
|
||||
|
||||
return { support, canSeeImages: support === "yes", isChecking, recheck };
|
||||
return { canSeeImages, isChecking };
|
||||
}
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
import { environment } from "@vicinae/api";
|
||||
import { join } from "node:path";
|
||||
|
||||
/**
|
||||
* The seam between the snip command and the chat command.
|
||||
*
|
||||
* The capture cannot happen inside the chat view. Vicinae shows the launcher
|
||||
* window when it activates a command, so a region selector started from a view
|
||||
* is dragging around a panel parked in the middle of the screen. Capturing from
|
||||
* a `no-view` command and handing the file over is what keeps the window out of
|
||||
* the way, which means the two commands need a small agreed contract.
|
||||
*
|
||||
* Imports Vicinae for `environment.supportPath` only, and nothing else in the
|
||||
* logic layer depends on it.
|
||||
*/
|
||||
|
||||
/** What the snip command passes to the chat command through `launchCommand`. */
|
||||
export interface SnipLaunchContext {
|
||||
/** Absolute path of the captured PNG. Absent when the capture failed. */
|
||||
snipPath?: string;
|
||||
/** Why a capture failed, so the chat can explain itself rather than look empty. */
|
||||
snipError?: string;
|
||||
/** The conversation the snip was started from, restored so it is not lost. */
|
||||
sessionId?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Snips live in the extension's own support directory rather than a temp
|
||||
* directory: a session stores the path, and a path that survives a reboot is the
|
||||
* difference between an old conversation showing its screenshot and pointing at
|
||||
* nothing.
|
||||
*/
|
||||
export function snipDirectory(): string {
|
||||
return join(environment.supportPath, "snips");
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads the handoff payload.
|
||||
*
|
||||
* A launch context is whatever the launching command put there, so every field
|
||||
* is checked rather than trusted: a wrong type would otherwise reach the file
|
||||
* system as a path.
|
||||
*/
|
||||
export function readSnipContext(launchContext: unknown): SnipLaunchContext {
|
||||
if (!launchContext || typeof launchContext !== "object") return {};
|
||||
|
||||
const { snipPath, snipError, sessionId } = launchContext as Record<string, unknown>;
|
||||
const asString = (value: unknown) => (typeof value === "string" && value.length > 0 ? value : undefined);
|
||||
|
||||
return {
|
||||
snipPath: asString(snipPath),
|
||||
snipError: asString(snipError),
|
||||
sessionId: asString(sessionId),
|
||||
};
|
||||
}
|
||||
+85
-10
@@ -1,18 +1,93 @@
|
||||
import { ChatView } from "./lib/components/ChatView";
|
||||
import { LaunchType, closeMainWindow, launchCommand, showHUD, type LaunchProps } from "@vicinae/api";
|
||||
import { NoCaptureToolError, captureRegion } from "./lib/capture";
|
||||
import { getLamPreferences } from "./lib/preferences";
|
||||
import { readSnipContext, snipDirectory } from "./lib/snip";
|
||||
import { errorMessage } from "./lib/utils";
|
||||
import { visionSupport } from "./lib/vision";
|
||||
|
||||
/**
|
||||
* Vicinae's `closeMainWindow` defers the actual hide to a 50ms timer
|
||||
* (`closeWindow(settings, 50ms)` in `ui-service.hpp`), so a capture started the
|
||||
* instant the call resolves can still find the window on screen and select a
|
||||
* region around it. Waiting a little past that timer is the difference between
|
||||
* "the window is closed" and "the window is usually closed".
|
||||
*/
|
||||
const WINDOW_SETTLE_MS = 100;
|
||||
|
||||
function settleWindow(): Promise<void> {
|
||||
return new Promise((resolve) => setTimeout(resolve, WINDOW_SETTLE_MS));
|
||||
}
|
||||
|
||||
/**
|
||||
* Entrypoint for the "snip" command.
|
||||
*
|
||||
* Captures a region, then hands the image to a *new* chat. Nothing is sent on
|
||||
* capture: a screenshot on its own is a question the user has not asked yet, so
|
||||
* {@link ChatView} holds it until they say what they want to know about it.
|
||||
* `no-view`, and it has to be. Vicinae calls `showWindow()` when it activates a
|
||||
* command from a closed launcher, so the window is up by the time any extension
|
||||
* code runs; capturing from a view means selecting a region around a panel
|
||||
* sitting in the middle of the screen. This command closes the window first,
|
||||
* captures with nothing in the way, and only then hands the image to the chat,
|
||||
* which is the command that brings the window back.
|
||||
*
|
||||
* The view gates the capture on the model actually reporting vision support, so
|
||||
* a text-only model explains itself rather than taking a screenshot nothing can
|
||||
* read.
|
||||
* It is also the *only* capture path. The chat's own action launches this command
|
||||
* rather than capturing inline, so there is one implementation, one capability
|
||||
* check, and one place where the window is dealt with.
|
||||
*
|
||||
* Bind this to a global shortcut for one-key "snip and ask".
|
||||
* Bind it to a global shortcut for one-key "snip and ask".
|
||||
*/
|
||||
export default function SnipCommand() {
|
||||
return <ChatView startWithSnip />;
|
||||
export default async function SnipCommand(props: LaunchProps) {
|
||||
const preferences = getLamPreferences();
|
||||
const context = readSnipContext(props.launchContext);
|
||||
|
||||
// Before anything else. Everything below is pointless if the selector is
|
||||
// competing with a window for the middle of the screen.
|
||||
await closeMainWindow();
|
||||
await settleWindow();
|
||||
|
||||
// Settled before the capture, because taking a screenshot for a model that
|
||||
// cannot read it throws away the selection gesture the user just performed.
|
||||
// "unknown" says so in those words: a dropped check is not a verdict on the
|
||||
// model, and reporting it as one would send the user shopping for another.
|
||||
const vision = await visionSupport({
|
||||
host: preferences.host,
|
||||
apiKey: preferences.apiKey,
|
||||
model: preferences.model,
|
||||
});
|
||||
|
||||
if (vision !== "yes") {
|
||||
await showHUD(
|
||||
vision === "no"
|
||||
? `${preferences.model} cannot see images. Pick a vision model in preferences.`
|
||||
: `Could not check whether ${preferences.model} can see images. Run Set Up Lam Agent.`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
let snipPath: string | undefined;
|
||||
let snipError: string | undefined;
|
||||
|
||||
try {
|
||||
const capture = await captureRegion({ outputDir: snipDirectory() });
|
||||
snipPath = capture?.path;
|
||||
} catch (thrown) {
|
||||
snipError =
|
||||
thrown instanceof NoCaptureToolError ? `${errorMessage(thrown)} ${thrown.hint}` : errorMessage(thrown);
|
||||
}
|
||||
|
||||
// An abandoned selection says nothing at all: the user pressed Escape. But a
|
||||
// snip started from an open conversation has to come back, because the window
|
||||
// is now closed and the hotkey is otherwise the only way back.
|
||||
if (!snipPath && !snipError && !context.sessionId) return;
|
||||
|
||||
// Only the fields that exist are sent. `launchCommand` types its context as a
|
||||
// string map, and a key left undefined carries no information anyway.
|
||||
const payload: Record<string, string> = {};
|
||||
if (context.sessionId) payload.sessionId = context.sessionId;
|
||||
if (snipPath) payload.snipPath = snipPath;
|
||||
if (snipError) payload.snipError = snipError;
|
||||
|
||||
await launchCommand({
|
||||
name: "chat",
|
||||
type: LaunchType.UserInitiated,
|
||||
context: payload,
|
||||
});
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user