Open console query results in Positron's Data Explorer

CHANGELOG.md

@@ -1,5 +1,14 @@
 ## [Unreleased]
 
+### Added
+
+- In a Positron console, a query result can be opened in the Data Explorer
+  from an "Open in Data Explorer" link under its table, with sorting,
+  filtering, profiles, export and Convert to Code. A `-- @view` comment opens
+  a query's result in the Data Explorer as soon as it runs. Long results show
+  their first 10 rows in the console, configurable with `ggsql.consoleRows`
+  (posit-dev/positron#16137).
+
 ### Changed
 
 - The reader/dialect layer was refactored around a central dialect registry with

doc/get_started/tooling/positron-vscode.qmd

@@ -30,6 +30,20 @@ Executing ggsql statements containing [`VISUALISE`](../../syntax/clause/visualis
 
 ![](./screenshots/positron-plots-pane.png)
 
+## Data Explorer
+
+A query without `VISUALISE` shows its result as a table in the console, followed by an "Open in Data Explorer" link. A result of more than 20 rows shows only its first 10, with a count of the rest. To show more or fewer, change the `ggsql.consoleRows` setting: a result of up to twice that many rows is shown in full. The change applies to consoles started afterwards. Clicking the link opens the result in Positron's Data Explorer, where you can sort, filter, and profile it, and use "Convert to Code" to turn your sorts and filters back into SQL.
+
+To open a result in the Data Explorer as soon as it runs, start the query with a `-- @view` comment. You can also give the tab a title:
+
+```ggsql
+-- @view: Revenue by region
+SELECT region, sum(revenue) AS revenue FROM sales GROUP BY region
+```
+
+The result opens in a Data Explorer tab, and is also shown in the console as usual. Because it is an ordinary comment, `-- @view` works the same way in a source file, in a cell, and in code sent to the console by other tools. It applies to the query that follows it, and that query cannot have a `VISUALISE` clause. Outside Positron, where there is no Data Explorer, the result is shown as a table as usual.
+
+The Data Explorer works on a snapshot of the result, so it doesn't change when the underlying tables do. Run the query again to see new data.
 
 ## Editor and cells
 

ggsql-jupyter/CLAUDE.md

@@ -17,7 +17,11 @@ ggsql-jupyter/
 │   ├── kernel.rs       Jupyter messaging loop (ZMQ, message dispatch)
 │   ├── executor.rs     Runs queries via ggsql::Reader, returns rendered output
 │   ├── connection.rs   Reader lifecycle, connection-string parsing
-│   ├── data_explorer.rs Positron data-explorer comm channel
+│   ├── data_explorer/  Positron data-explorer comm channel
+│   │   ├── mod.rs      State, RPC dispatch, paging and profiles
+│   │   ├── filter.rs   Row filters, sort keys, schema search, Convert to Code → SQL
+│   │   └── export.rs   Copying a selection as CSV / TSV / HTML
+│   ├── results.rs      Query results retained for the data explorer
 │   ├── display.rs      Output formatting (image bundles, SQL → HTML table)
 │   ├── message.rs      Jupyter message structs (ZMQ frames, HMAC signing)
 │   └── util.rs
@@ -173,7 +177,32 @@ So `RenderHints::canvas` picks a height (golden ratio, close to ggplot2's defaul
 
 ## Positron-specific bits
 
-- `data_explorer.rs` implements Positron's data-explorer comm channel (registered query results become explorable tables).
+- `data_explorer/` implements Positron's data-explorer comm channel. An explorer's `Source` says which database its table is in: the session's own, for a table opened from the Connections pane, or the private results database below.
+
+### The data explorer as SQL
+
+Every page, filter, sort, profile and export is a query; the table is never loaded whole. A few things are deliberate:
+
+- **Filter values arrive as strings and are checked against the column's type** before they reach the SQL: a number must parse as a finite number, a boolean as true/false, and everything else becomes a quoted literal. Filters are always ANDed (`supports_conditions: unsupported`).
+- **A bad filter is left out, not fatal.** One the kernel cannot build, or one the database rejects (counted on its own to find out which), comes back with `is_valid: false` and an `error_message`; the rest still apply.
+- **Profiles and exports see the filtered rows**, through `relation()`; pages and exports also follow the sort.
+- **Sorting a retained result breaks ties on DuckDB's `rowid`**, its original row order, so paging is stable and clearing the sort restores the query's own `ORDER BY`. A Connections-pane table has no portable equivalent and gets no tiebreak.
+- **Regular-expression search needs `Capabilities::regex`** (`regexp_matches`, DuckDB only). The results database always has it; the session's database has it when its URI is `duckdb://`.
+- **Convert to Code writes SQL for the session's database**, not the private one, wrapping a retained result's original query as a derived table. A filter it cannot express there is left out with a `--` comment saying so.
+- **A failed request is a JSON-RPC error**, with `plot::comm::RpcError`'s encoding, not a `null` result.
+
+### Opening a console result in the data explorer
+
+The console renders a tabular result as an HTML table and has no output toolbar to hang a button on, so the affordance is a link, and it needs no change to Positron:
+
+1. **A console result is retained.** `results.rs` copies it into a private in-memory DuckDB under an unguessable key, bounded by `--max-results` (default 16). Only results no explorer has open count against the bound or are evicted, so an open tab never loses its table. Querying a copy rather than the source means one SQL dialect for everything the explorer asks, no remote query re-run per page, and nothing visible in the user's session.
+2. **A `stream` follows the `execute_result`**, carrying an OSC 8 hyperlink to `x-ggsql-view:<session-id>/<key>`. It cannot go in the bundle's `text/plain`: the console shows the HTML and ignores it. So that the link stays in view under a long result, a console table shows rows the way R's tibble does — all of them up to twice `--console-rows` (default 10), else the first `--console-rows` — where elsewhere it shows up to 100 (`display.rs`'s `html_rows_shown`). The option is also read from `GGSQL_CONSOLE_ROWS`, which is how the extension sets it.
+3. **Positron offers a clicked hyperlink to `openResource`** on the last active console session of each language. The extension's session wrapper (`ggsql-vscode/src/resultLinks.ts`) handles the scheme and runs `-- @view-result <key>` silently, in the session the link names.
+4. **`-- @view-result` opens a `positron.dataExplorer` comm with `inline_only: false`**, which Positron opens as an editor tab immediately. A result that already has an explorer open is not opened again: the kernel sends `open_editor` on the UI comm with that explorer's editor URI (`positron-data-explorer:positron-data-explorer-<comm id>`), which focuses the tab — what Positron's Variables pane does for a variable already in view. A key that is no longer retained is a cell error, which surfaces as a warning from the extension.
+
+Only `SessionKind::PositronConsole` gets this: anywhere else the escape sequence would be stray text. Notebooks have a better route — the inline data explorer — that this does not yet use.
+
+A query under the user-facing `-- @view` directive (see [the Positron docs](../doc/get_started/tooling/positron-vscode.qmd)) takes the same path from step 1, minus the click: it is retained in any Positron session, silent executions included, and opened at once. The table still shows in the cell, and a console still gets the link under it, to reopen a closed tab; a silent execution shows nothing but the tab. A `Standalone` session has no data explorer to open, so it gets the inline table and a `stderr` note instead.
 - The companion VS Code extension (`ggsql-vscode/`) ships a copy of this binary and also discovers installed ones via the `ggsql.kernelPath` setting, the Jupyter kernelspec directories, the native install locations, or `PATH`. It runs each one with `--version` to name the runtime it registers, so **keep `--version` working**: a kernel that does not answer it is still offered, but without a version in the picker. See [Finding the kernel](../ggsql-vscode/CLAUDE.md#finding-the-kernel).
 
 ## Build & install

ggsql-jupyter/Cargo.toml

@@ -33,12 +33,15 @@ arrow = { workspace = true }
 tokio = { workspace = true, features = ["sync"] }
 
 # CLI
-clap = { workspace = true }
+clap = { workspace = true, features = ["env"] }
 
 # Serialization
 serde = { workspace = true }
 serde_json = { workspace = true }
 
+# The data explorer's regular-expression column search
+regex = { workspace = true }
+
 # Error handling
 anyhow = { workspace = true }
 thiserror = { workspace = true }

ggsql-jupyter/src/data_explorer/export.rs

@@ -0,0 +1,108 @@
+//! A data explorer selection as CSV, TSV or HTML, for the clipboard.
+
+use crate::display::escape_html;
+
+/// Format `rows` for export. A `header` is written first when given; a null
+/// cell is empty.
+pub fn format(
+    header: Option<&[String]>,
+    rows: &[Vec<String>],
+    format: &str,
+) -> Result<String, String> {
+    let all = header
+        .into_iter()
+        .map(<[String]>::to_vec)
+        .chain(rows.iter().cloned());
+    match format {
+        "csv" => Ok(delimited(all, ",", csv_field)),
+        "tsv" => Ok(delimited(all, "\t", tsv_field)),
+        "html" => Ok(html(header, rows)),
+        other => Err(format!("Unsupported export format {other:?}")),
+    }
+}
+
+fn delimited(
+    rows: impl Iterator<Item = Vec<String>>,
+    sep: &str,
+    field: fn(&str) -> String,
+) -> String {
+    let lines: Vec<String> = rows
+        .map(|row| row.iter().map(|v| field(v)).collect::<Vec<_>>().join(sep))
+        .collect();
+    lines.join("\n")
+}
+
+/// Quoted only when it has to be, with embedded quotes doubled (RFC 4180).
+fn csv_field(v: &str) -> String {
+    if v.contains([',', '"', '\n', '\r']) {
+        format!("\"{}\"", v.replace('"', "\"\""))
+    } else {
+        v.to_string()
+    }
+}
+
+/// TSV has no quoting, so the characters that would break a row become spaces.
+fn tsv_field(v: &str) -> String {
+    v.replace(['\t', '\n', '\r'], " ")
+}
+
+fn html(header: Option<&[String]>, rows: &[Vec<String>]) -> String {
+    let mut out = String::from("<table>\n");
+    if let Some(header) = header {
+        out.push_str("<thead><tr>");
+        for h in header {
+            out.push_str(&format!("<th>{}</th>", escape_html(h)));
+        }
+        out.push_str("</tr></thead>\n");
+    }
+    out.push_str("<tbody>\n");
+    for row in rows {
+        out.push_str("<tr>");
+        for v in row {
+            out.push_str(&format!("<td>{}</td>", escape_html(v)));
+        }
+        out.push_str("</tr>\n");
+    }
+    out.push_str("</tbody>\n</table>");
+    out
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn rows() -> Vec<Vec<String>> {
+        vec![
+            vec!["1".into(), "a, \"b\"".into()],
+            vec!["2".into(), "tab\there".into()],
+        ]
+    }
+
+    #[test]
+    fn test_csv_quotes_only_what_needs_it() {
+        let header = ["n".to_string(), "s".to_string()];
+        assert_eq!(
+            format(Some(&header), &rows(), "csv").unwrap(),
+            "n,s\n1,\"a, \"\"b\"\"\"\n2,tab\there"
+        );
+    }
+
+    #[test]
+    fn test_tsv_flattens_tabs() {
+        assert_eq!(
+            format(None, &rows(), "tsv").unwrap(),
+            "1\ta, \"b\"\n2\ttab here"
+        );
+    }
+
+    #[test]
+    fn test_html_escapes() {
+        let out = format(None, &[vec!["<b>".into()]], "html").unwrap();
+        assert!(out.contains("<td>&lt;b&gt;</td>"), "{out}");
+    }
+
+    #[test]
+    fn test_unknown_format_is_an_error() {
+        assert!(format(None, &rows(), "xlsx").is_err());
+    }
+}