fix: raise citation markers and stop adjacent ones merging #22

Merged
adrianbonpin merged 1 commits from feat/superscript-citations into prod 2026-10-07 08:20:03 +00:00
7 changed files with 103 additions and 25 deletions
Showing only changes of commit eff0d4ea05 - Show all commits
+4
View File
@@ -300,6 +300,10 @@ The whole pass is plain TypeScript with no Vicinae imports, so it unit tests wit
Footnotes deserve a note of their own, because the window cannot draw them. Vicinae parses markdown with `CMARK_OPT_DEFAULT` and attaches only the table and strikethrough extensions, so a `[^1]` arrives as literal text. The citation pass therefore does the whole job: it links the marker, drops a footnote definition that would only repeat an entry the tools already registered, and builds the list from those entries, which is also what stops the list from being something a model can quietly invent. Footnotes deserve a note of their own, because the window cannot draw them. Vicinae parses markdown with `CMARK_OPT_DEFAULT` and attaches only the table and strikethrough extensions, so a `[^1]` arrives as literal text. The citation pass therefore does the whole job: it links the marker, drops a footnote definition that would only repeat an entry the tools already registered, and builds the list from those entries, which is also what stops the list from being something a model can quietly invent.
In a sentence the marker is raised, so a claim resting on a search result reads as `rain is likely¹` rather than with the brackets at full size. It is a Unicode superscript digit rather than a `<sup>` tag, because a link label is plain text that nothing downstream can break, and a digit is raised by design in every font. The entry under the answer keeps the plain number, `[1] Rain in Cebu`, where being able to scan a list matters more than the convention.
Markers the model wrote touching are separated as they are drawn, because `¹²` reads as twelve, or as twenty one, and either way as one number. The prompt asks for exactly that touching shape, so `[1][2]` and the same numbers written apart both arrive as `¹, ²`.
Numbers are handed out where the tools run, once per turn, and printed in the tool result the model reads. So `[3]` means one page for the model and for the reader, three searches in one turn do not each start again at `1`, and a page found by a search and read again later keeps the number it already had. A marker stays as written unless every number in it is a source with a destination this window should open: half a link reads as a citation that works, and a `javascript:` address out of a search result is a real vulnerability rather than a rendering quirk. Numbers are handed out where the tools run, once per turn, and printed in the tool result the model reads. So `[3]` means one page for the model and for the reader, three searches in one turn do not each start again at `1`, and a page found by a search and read again later keeps the number it already had. A marker stays as written unless every number in it is a source with a destination this window should open: half a link reads as a citation that works, and a `javascript:` address out of a search result is a real vulnerability rather than a rendering quirk.
The list shows the sources the answer cited, falling back to everything the turn consulted when it cited nothing, since that is the one case where the list is the only evidence of what was read. It waits for a citation before appearing at all, so it never arrives in full and then shrinks. The list shows the sources the answer cited, falling back to everything the turn consulted when it cited nothing, since that is the one case where the list is the only evidence of what was read. It waits for a citation before appearing at all, so it never arrives in full and then shrinks.
@@ -67,9 +67,18 @@ would have been asked are recorded here instead.
line; a citation list is part of the answer's credibility. line; a citation list is part of the answer's credibility.
- **A4. The copied conversation includes the list**, in the same form, so - **A4. The copied conversation includes the list**, in the same form, so
provenance survives a paste into a document. provenance survives a paste into a document.
- **A5. The marker is `[1]`, not a superscript.** A superscript inside a link is - **A5. The marker in a sentence is a superscript, and the list keeps the plain
a smaller target and, at this pane width, harder to read than the bracket form number.** The model still writes `[1]`; the window draws it raised, as `[¹]`, and
that is already the convention in most model output. the entry under the answer reads `[1] Title` at normal size, where scanning a list
matters more than convention. This reverses an earlier decision that markers stay
at baseline, taken on the grounds that a raised marker is a smaller target; the
user asked for the raised form after reading the first version, and the target is
the same width in practice because only the digits are raised.
- **A6. Superscripts are Unicode digits, not a `<sup>` tag.** The label of a markdown
link is plain text that no pass can break, while inline HTML inside a label asks
the sanitizer, the parser and Qt to agree about something this repository cannot
test. Digits are raised by design in every font, so the mark still reads correctly
when a font lacks the rarer glyphs and Qt falls back for them.
## Design ## Design
@@ -152,7 +161,11 @@ The pass follows the rules the rest of the pipeline already follows.
- **Prose only.** Fenced segments and code spans are skipped, using the same - **Prose only.** Fenced segments and code spans are skipped, using the same
machinery the maths pass uses. A `[1]` inside a snippet is text. machinery the maths pass uses. A `[1]` inside a snippet is text.
- **`[n]`, `[^n]` and comma lists** (`[1, 3]`) are recognised. - **`[n]`, `[^n]`, comma lists, and runs.** `[1, 3]` is two sources inside one
marker, and `[1][2]` is two markers written touching. Both are drawn as raised
digits separated by a comma, because `¹²` reads as a single number and leaves the
reader no way to tell where one citation ends, which is a defect the first version
of this shipped with.
- **All or nothing.** A marker is linked only when every index it names is a - **All or nothing.** A marker is linked only when every index it names is a
known source. `[1, 9]` where 9 was never seen stays exactly as written, rather known source. `[1, 9]` where 9 was never seen stays exactly as written, rather
than becoming half a link. than becoming half a link.
+34 -9
View File
@@ -116,23 +116,21 @@ describe("linkCitations", () => {
{ index: 3, title: "Third", url: "https://third.example" }, { index: 3, title: "Third", url: "https://third.example" },
]; ];
it("turns a marker into a link whose label is the marker", () => { it("raises a marker into a superscript link", () => {
const result = linkCitations("Rain is likely [1].", sources); const result = linkCitations("Rain is likely [1].", sources);
expect(result.text).toBe("Rain is likely [\\[1\\]](https://first.example)."); expect(result.text).toBe("Rain is likely [¹](https://first.example).");
expect([...result.cited]).toEqual([1]); expect([...result.cited]).toEqual([1]);
}); });
it("reads the footnote shape as the same thing", () => { it("reads the footnote shape as the same thing", () => {
expect(linkCitations("Rain [^1].", sources).text).toBe("Rain [\\[1\\]](https://first.example)."); expect(linkCitations("Rain [^1].", sources).text).toBe("Rain [¹](https://first.example).");
}); });
it("links a list of markers and keeps the comma", () => { it("links a list of markers and keeps the comma", () => {
const result = linkCitations("Both [1, 3] agree.", sources); const result = linkCitations("Both [1, 3] agree.", sources);
expect(result.text).toBe( expect(result.text).toBe("Both [¹](https://first.example), [³](https://third.example) agree.");
"Both [\\[1\\]](https://first.example), [\\[3\\]](https://third.example) agree.",
);
expect([...result.cited].sort()).toEqual([1, 3]); expect([...result.cited].sort()).toEqual([1, 3]);
}); });
@@ -145,6 +143,33 @@ describe("linkCitations", () => {
expect([...result.cited]).toEqual([]); expect([...result.cited]).toEqual([]);
}); });
it("separates markers written touching, which would otherwise read as one number", () => {
// The prompt asks for exactly this shape, and `¹³` is a number: nothing about it
// says it is two sources, or where the first one stops and the second begins.
const result = linkCitations("Both [1][3] agree.", sources);
expect(result.text).toBe("Both [¹](https://first.example), [³](https://third.example) agree.");
expect([...result.cited].sort()).toEqual([1, 3]);
});
it("treats a space between two markers the same way", () => {
expect(linkCitations("Both [1] [3] agree.", sources).text).toBe(
"Both [¹](https://first.example), [³](https://third.example) agree.",
);
});
it("leaves a whole run alone when one number in it is not a source", () => {
expect(linkCitations("See [1][9].", sources).text).toBe("See [1][9].");
});
it("does not join markers across a line break", () => {
// A run belongs to one line. Crossing a newline would pull two paragraphs of
// citations into one sentence.
expect(linkCitations("See [1]\n[3].", sources).text).toBe(
"See [¹](https://first.example)\n[³](https://third.example).",
);
});
it("leaves a marker alone when the destination is not one this window should open", () => { it("leaves a marker alone when the destination is not one this window should open", () => {
const unsafe = [{ index: 1, title: "Nasty", url: "javascript:alert(1)" }]; const unsafe = [{ index: 1, title: "Nasty", url: "javascript:alert(1)" }];
const result = linkCitations("See [1].", unsafe); const result = linkCitations("See [1].", unsafe);
@@ -171,7 +196,7 @@ describe("linkCitations", () => {
it("drops a footnote definition for a source it already lists", () => { it("drops a footnote definition for a source it already lists", () => {
const answer = "Rain [^1].\n\n[^1]: First, https://first.example"; const answer = "Rain [^1].\n\n[^1]: First, https://first.example";
expect(linkCitations(answer, sources).text).toBe("Rain [\\[1\\]](https://first.example).\n"); expect(linkCitations(answer, sources).text).toBe("Rain [¹](https://first.example).\n");
}); });
it("keeps a markdown reference definition, which backs a link the model wrote", () => { it("keeps a markdown reference definition, which backs a link the model wrote", () => {
@@ -179,14 +204,14 @@ describe("linkCitations", () => {
// shape is unambiguously a duplicate of an entry we render ourselves. // shape is unambiguously a duplicate of an entry we render ourselves.
const answer = "Rain [^1].\n\n[1]: https://elsewhere.example"; const answer = "Rain [^1].\n\n[1]: https://elsewhere.example";
expect(linkCitations(answer, sources).text).toBe( expect(linkCitations(answer, sources).text).toBe(
"Rain [\\[1\\]](https://first.example).\n\n[1]: https://elsewhere.example", "Rain [¹](https://first.example).\n\n[1]: https://elsewhere.example",
); );
}); });
it("keeps a definition for a number it knows nothing about", () => { it("keeps a definition for a number it knows nothing about", () => {
const answer = "Rain [^1].\n\n[^7]: Something else"; const answer = "Rain [^1].\n\n[^7]: Something else";
expect(linkCitations(answer, sources).text).toBe( expect(linkCitations(answer, sources).text).toBe(
"Rain [\\[1\\]](https://first.example).\n\n[^7]: Something else", "Rain [¹](https://first.example).\n\n[^7]: Something else",
); );
}); });
+1 -1
View File
@@ -675,7 +675,7 @@ describe("sources", () => {
it("links the markers and lists the sources the answer leaned on", () => { it("links the markers and lists the sources the answer leaned on", () => {
const text = answered("Rain is likely [1]."); const text = answered("Rain is likely [1].");
expect(text).toContain("Rain is likely [\\[1\\]](https://first.example)."); expect(text).toContain("Rain is likely [¹](https://first.example).");
expect(text).toContain("_Sources_"); expect(text).toContain("_Sources_");
expect(text).toContain("- [\\[1\\]](https://first.example) First"); expect(text).toContain("- [\\[1\\]](https://first.example) First");
// The answer did not lean on the second one. // The answer did not lean on the second one.
+3 -3
View File
@@ -145,13 +145,13 @@ describe("citations in the pipeline", () => {
it("reports which citations it linked", () => { it("reports which citations it linked", () => {
const result = renderModelOutput("Rain [1].", { sources }); const result = renderModelOutput("Rain [1].", { sources });
expect(result.text).toBe("Rain [\\[1\\]](https://first.example)."); expect(result.text).toBe("Rain [¹](https://first.example).");
expect([...result.cited]).toEqual([1]); expect([...result.cited]).toEqual([1]);
}); });
it("links a marker that has to survive the maths and markup passes", () => { it("links a marker that has to survive the maths and markup passes", () => {
expect(rendered("Let $x^2$ be [1], and nothing else.", sources)).toContain( expect(rendered("Let $x^2$ be [1], and nothing else.", sources)).toContain(
"x² be [\\[1\\]](https://first.example)", "x² be [¹](https://first.example)",
); );
}); });
@@ -159,7 +159,7 @@ describe("citations in the pipeline", () => {
const text = rendered("```bash\ncurl [1]\n```\n\nSee [1].", sources); const text = rendered("```bash\ncurl [1]\n```\n\nSee [1].", sources);
expect(text).toContain("curl [1]"); expect(text).toContain("curl [1]");
expect(text).toContain("[\\[1\\]](https://first.example)."); expect(text).toContain("[¹](https://first.example).");
}); });
it("changes nothing when no sources are given", () => { it("changes nothing when no sources are given", () => {
+43 -7
View File
@@ -104,13 +104,46 @@ export interface CitationResult {
} }
/** /**
* A citation marker the model wrote, in the two shapes it reaches for. * One citation marker, in the two shapes a model reaches for, plus any markers
* written straight after it.
*
* Adjacent markers are matched as a single run, because a raised digit carries no
* separator of its own: `[1][2]` drawn as `¹²` reads as twelve, or as twenty one, and
* either way as one number rather than as two sources.
* *
* The trailing guards are what keep this off markdown the model wrote on purpose: * The trailing guards are what keep this off markdown the model wrote on purpose:
* `](` starts a link, and `]:` is a reference definition. The digit list allows * `](` starts a link, and `]:` is a reference definition. The digit list allows
* `[1, 3]`, which is how a sentence resting on two sources is usually written. * `[1, 3]`, which is how a sentence resting on two sources is often written.
*/ */
const CITATION_MARKER = /\[\^?(\d+(?:\s*,\s*\d+)*)\](?![(:])/g; const CITATION_RUN = /\[\^?\d+(?:\s*,\s*\d+)*\](?![(:])(?:[ \t]*\[\^?\d+(?:\s*,\s*\d+)*\](?![(:]))*/g;
/** The numbers inside one run, in the order they were written. */
const RUN_NUMBERS = /\d+/g;
/** The digits a superscript marker is built from, in order. */
const DIGITS = "0123456789";
const SUPERSCRIPT = "⁰¹²³⁴⁵⁶⁷⁸⁹";
/**
* A number as a superscript, which is what a citation mark is.
*
* Unicode rather than a `<sup>` tag on purpose. This is the label of a markdown
* link, and a plain label cannot be broken by any part of the pipeline, while
* inline HTML in a label asks two renderers to agree about something neither this
* extension nor its tests can see. The glyphs are raised by design, so the mark
* still reads as a superscript even when a font lacks the rarer ones and Qt falls
* back for them.
*/
function superscript(index: number): string {
let out = "";
for (const digit of String(index)) {
const at = DIGITS.indexOf(digit);
out += at >= 0 ? SUPERSCRIPT[at] : digit;
}
return out;
}
/** /**
* A footnote definition line, which is `[^1]: title, url` on a line of its own. * A footnote definition line, which is `[^1]: title, url` on a line of its own.
@@ -165,7 +198,8 @@ function stripDefinitions(text: string, known: Set<number>): string {
} }
/** /**
* `[1]`, `[^1]` and `[1, 3]` as links, plus the numbers that became links. * `[1]`, `[^1]`, `[1, 3]` and `[1][2]` as superscript links, plus the numbers that
* became links.
* *
* One run of prose only: fences are the caller's business, because the caller * One run of prose only: fences are the caller's business, because the caller
* already walks the document in segments for the maths pass and there is no * already walks the document in segments for the maths pass and there is no
@@ -179,8 +213,8 @@ export function linkCitations(text: string, sources: Source[]): CitationResult {
const prose = stripDefinitions(text, new Set(byIndex.keys())); const prose = stripDefinitions(text, new Set(byIndex.keys()));
const linked = mapOutsideCodeSpans(prose, (chunk) => const linked = mapOutsideCodeSpans(prose, (chunk) =>
chunk.replace(CITATION_MARKER, (whole: string, digits: string) => { chunk.replace(CITATION_RUN, (whole: string) => {
const wanted = digits.split(",").map((part) => Number(part.trim())); const wanted = [...whole.matchAll(RUN_NUMBERS)].map((match) => Number(match[0]));
const links: string[] = []; const links: string[] = [];
for (const index of wanted) { for (const index of wanted) {
@@ -190,10 +224,12 @@ export function linkCitations(text: string, sources: Source[]): CitationResult {
// marker that is partly a link reads as a citation that works. // marker that is partly a link reads as a citation that works.
if (!href) return whole; if (!href) return whole;
links.push(markdownLink(`[${index}]`, href)); links.push(markdownLink(superscript(index), href));
} }
for (const index of wanted) cited.add(index); for (const index of wanted) cited.add(index);
// Separated even when the model wrote them touching: `¹, ²` is two citations,
// where `¹²` is a number, and the reader has no way to tell it from a real one.
return links.join(", "); return links.join(", ");
}), }),
); );
+1 -1
View File
@@ -96,7 +96,7 @@ Fence every command and every snippet, and always name the language: bash for sh
A web address is a link, never code. A markdown link, written as [label](https://example.com), is clickable in this window, while a URL inside backticks or a fenced block is dead text the user cannot open. So never fence a URL, never put one in inline code, and never answer "here are the links" with a snippet. Label a link with what it is ("OpenClickly on GitHub"), and when you give several, put each on its own bullet with its own link. A web address is a link, never code. A markdown link, written as [label](https://example.com), is clickable in this window, while a URL inside backticks or a fenced block is dead text the user cannot open. So never fence a URL, never put one in inline code, and never answer "here are the links" with a snippet. Label a link with what it is ("OpenClickly on GitHub"), and when you give several, put each on its own bullet with its own link.
When a tool result gives you a source, cite it: put its number in square brackets right after the sentence it supports, like a claim resting on a search result [1]. Cite the page that carries the point rather than the search that found it, cite one number at a time when a sentence rests on two of them (write [1][2], not [1,2]), and never cite a number you were not shown. The window turns that number into a link and lists the sources under your answer, so write no footnote definitions and no sources or references list of your own. When a tool result gives you a source, cite it: put its number in square brackets right after the sentence it supports, like a claim resting on a search result [1]. Cite the page that carries the point rather than the search that found it, put each number in its own brackets when a sentence rests on two sources (write [1][2]), and never cite a number you were not shown. The window turns that number into a link and lists the sources under your answer, so write no footnote definitions and no sources or references list of your own.
A fenced block is never a whole answer on its own. Say in one plain sentence what the code does or when to reach for it, so the reply reads like something a person wrote rather than a bare snippet. A fenced block is never a whole answer on its own. Say in one plain sentence what the code does or when to reach for it, so the reply reads like something a person wrote rather than a bare snippet.