Download the exact captured file
---
name: flexviz-explore
description: >
Serve a LOCAL Parquet/CSV file as a live, interactive FlexViz dashboard, read
back the zooms and selections a human makes in it, change it from your side,
and report the findings. Use when the user wants to explore or visualize a
large local dataset (too big to plot inline), asks for an interactive or
cross-filter dashboard, or hands you a flexviz /view URL. Prerequisites: the
data file is on this machine, flexviz is installed in the project environment,
and the human's browser can reach this machine. Do NOT use for small datasets
that fit an inline plot, or in remote sessions where the human cannot reach
the served port.
---
# Explore large data with FlexViz
FlexViz serves interactive cross-filter dashboards from lazy Polars queries. The
data stays server-side; the browser gets bounded aggregates and small samples of
real values. Parquet larger than RAM streams, so peak memory stays flat as rows
grow (box plots excepted).
Every dashboard view is a URL that carries the complete spec: 1 to 4 KB, and a
browser tool echoes the page URL in every snapshot. So you never hold one.
Record each URL under a number with `flexviz history`, open it at `/h/N`, and
call it `fv:N`.
Privacy: rows stay in the lazy engine unless you collect them. The schema, your
samples, and the ranges the human selects do enter your context. A share URL
embeds column names and filters, so add `.flexviz/` to `.gitignore`.
## The loop
### 1. Inspect the schema
```bash
flexviz schema data.parquet
```
Not on PATH? Use the project venv (`.venv/bin/flexviz` or `uv run flexviz`) and
keep that form below. If the package is absent, ask the human before installing.
The output names the file, the source, and every column with its dtype. Pick an
x column (usually time) and the columns worth plotting. A `.head(5).collect()`
peek is fine. Never collect the full frame.
### 2. Serve the file (background process)
```bash
flexviz serve data.parquet --cache > serve.log 2>&1 &
```
Run the server, and every command below, from the project directory: the history
file (`.flexviz/history.jsonl`) and `/h/N` both resolve against a working
directory.
- Without `--port` the server takes a free port and prints its URL on the first
line of the log. Read the port there and use it in every step below.
- Each file becomes a source named by its stem (`data.parquet` -> `"data"`).
- `--cache` enables cross-filter cubes and live brushing. Use it whenever
the file will not change while serving.
- Wait for readiness: poll until the answer names YOUR source, for example
`until curl -s http://127.0.0.1:<port>/sources | grep -q '"data"'; do sleep 1; done`.
A bare "it answered" check is not enough: another server can answer instead.
- Keep the process running while the human explores, tell them the port and
PID, and stop it at the end. An orphan server otherwise holds the port.
- To add files later, restart with the FULL file list (old + new).
- Serve on loopback. Another interface exposes unauthenticated endpoints, so
only do it if the human explicitly accepts that.
`flexviz serve` scans the file as stored: it cannot cast a column, and it has
no column the file does not have. A timestamp held as `String` gives a string
x axis with no warning, and a figure on a derived column (hour of day, weekday,
a ratio) opens fine and then fails every update with a 500 and
`ColumnNotFoundError` in the serve log, so its panels stay blank. In both
cases register the source yourself and build step 3 on that same LazyFrame:
```python
import polars as pl
from flexviz import register_source, run_server
lf = pl.scan_parquet("data.parquet").with_columns(pl.col("timestamp").str.to_datetime())
register_source("data", lf, cache=True) # call this before run_server
run_server("127.0.0.1", 8077)
```
### 3. Build the dashboard and record it
Build and record in ONE script, and print only the number:
```python
import polars as pl
from flexviz import Dashboard, history
dash = Dashboard(pl.scan_parquet("data.parquet"), cache=True)
dash.add_figure().add_line(x="timestamp", y="value", group_by="sensor_id")
dash.add_figure().add_histogram(x="value", bins=50)
url = dash.share_url(server_url="http://127.0.0.1:<port>", source_name="data")
print(history.add(url, note="data.parquet: line + histogram, initial view", actor="agent"))
```
- Start small: at most 4 figures in the first dashboard. Pick the few columns
that matter and add more only when the human asks. A wall of panels is harder
to read than a focused view.
- This script only builds a spec: cheap, lazy, and it exits at once. The serve
process answers every interaction.
- `source_name` must match the served stem, and `cache=True` must match
`--cache`.
- Two or more figures that show the same time column? Add
`dash.link_axes(on="timestamp")`, so a zoom on one moves them all. With only
one such figure there is nothing to link, and the call raises.
- No categorical column to `group_by`? Split metrics across figures instead,
and use `add_histogram2d` to relate numeric columns. Add at most one
`add_corr_heatmap` for the whole dashboard, and only when many numeric columns
make one worthwhile. It is an overview, not a per-figure default.
### 4. Open the entry
Open `http://127.0.0.1:<port>/h/N` for the number step 3 printed, and give the
human the same address. The `/h/N` address is what they open. In everything
you write, the entry is `fv:N`. Tell them: drag on one chart to cross-filter the
others, zoom to re-aggregate at higher detail, double-click to reset.
A page that opens is not a page that drew: check the serve log for a
traceback after the first open of every new spec.
### 5. Read the state back
All interaction state lives in ONE browser tab, so which tab the human uses
decides what you can read. Assume separate browsers until they confirm
otherwise.
**Shared tab.** Your browser tool drives a window on this machine that the human
can also use: a headed Playwright session (never `--headless`), or an extension
attached to the human's own browser. One tab holds the state, so you read it:
```js
window.flexvizState({compact: true}) // via your browser evaluate tool
```
It returns `{version, state, client_state, revision}`. `revision` increases when the
serialized state differs from the previous read; it does not say who changed it. Example:
<!-- compact-state -->
```json
{"version": "0.7", "revision": 3, "state": {
"viewport": {"fig1/x": {"min": 100, "max": 300}},
"selections": [{"source_figure_uid": "fig1",
"predicates": [{"clauses": [{"column": "value", "range": [10.0, 25.0]}]}]}],
"group_domains": {}, "cross_filter_mode": "update"},
"client_state": {"hover_mode": "off", "live_brush": "auto",
"axis_locks": {}, "axis_lock_ranges": {}, "axis_links": []}}
```
- `version`/`revision`: the spec version, and a counter that increases when the serialized state differs from the previous read.
- `state`: server-visible — viewport (`fig_uid/axis`), selections, group_domains, cross_filter_mode.
- `client_state`: client-only display state; the engine ignores it. `axis_links`
lists groups of viewport keys that always hold one range.
**Separate browsers.** A headless session, or a human at another machine. Your
tab and their tab hold independent state, so polling yours tells you nothing
about their zooms and selections. They hand the state over: ask them to click
**Share** in the toolbar (it copies a URL that captures their current view) and
to run this in the project directory:
```bash
flexviz history add "<paste it here>" --actor human --note "what I was looking at"
```
They tell you only the number it prints; you read it with `flexviz history show
N`. If they will not run a command, let them paste the URL and run it
yourself. That is the one place a URL enters your context, and it enters once.
The address bar does NOT track interactions; only Share captures them.
### 6. Record the human's state before you act
With separate browsers the human's `history add` already made this entry, so
skip the step. In the shared tab the page cannot write the history file, so you
record what you read.
Your browser tool returns the `flexvizState` readback as JSON text, not a Python
object, and some tools encode it twice. Save it, then load it tolerantly:
```python
import json
from flexviz import history
raw = json.load(open("state.json")) # whatever your browser tool wrote
compact = json.loads(raw) if isinstance(raw, str) else raw
print(history.record_state(3, compact["state"], compact["client_state"],
note="human brushed 09:00-11:00 on sensor 12"))
```
`record_state` appends a NEW numbered entry: it reuses entry 3's dashboard, swaps
in the state you read, and returns the new number (host and port come from
entry 3, so no URL reaches your context). Do it before you change anything: the
new entry is what the report cites later.
### 7. Change the dashboard
A state change (viewport, selections, hover mode, axis locks) goes through the
write half, in the tab you control. Smallest payloads. `F` below is a figure UID,
the one you read as a viewport key in step 5 (`"<uid>/x"`), never the panel's DOM
id or its index. A datetime axis stores the date string Plotly reports, e.g.
`"2024-01-01 00:00:00"`, not ISO-8601:
```js
await window.flexvizApply({state: {viewport: {"F/x": {min: a, max: b}}}})
await window.flexvizApply({state: {selections: []}}) // clear all selections
```
To author a cross-filter, `source_figure_uid` must be a figure that brushes the
predicate's column: filter `value` from the figure whose axis is `value`, not
from a time-series line. A mismatch is coerced silently to that figure's own
axis, not rejected:
```js
await window.flexvizApply({state: {selections: [
{source_figure_uid: "<uid of the value figure>",
predicates: [{clauses: [{column: "value", range: [2.5, 6.0]}]}]}
]}})
```
Only `state`, `client_state` and `layout` apply; every other key is ignored
with a warning. `state` and `client_state` merge one level deep, so a
partial `state` keeps its sibling keys (`viewport`, `group_domains`,
`cross_filter_mode`). It re-renders and resolves with the compact state. It
rejects with the server's reason when the re-request fails, and the page keeps
its previous state. When `client_state.axis_links` lists a viewport key, set
every key of that group to the same range; a patch that sets only one is
rejected. Locks follow the same rule: lock every key of a group or none, with
one range in `axis_lock_ranges`. With
separate browsers the human does not see it, so record the state with
`record_state(..., actor="agent")` and give them the new `/h/N` instead;
the default actor is `"human"`, because step 6 is its first use.
A structure change (add or remove a figure, change traces) needs
a new spec, because panels are built server-side. Rebuild in Python as in step
3, `history.add(...)`, then open the new `/h/N`. Note your own entries too, so
the history reads as a sequence of who did what.
### 8. Tell them what you found
End the first pass in chat: the findings in a few lines, each one naming the
entry that shows it as `fv:3`, never as "panel 1". Filter the LazyFrame to the
brushed range to put a number next to a figure. End that same message with the
offer, after the findings, never before: "Say report and I write findings.md
with live views, tell me what to look at next, and say whether you use the
window I opened or your own browser."
Write that file only when they ask. A line that holds nothing but `fv:5` becomes
the live dashboard of entry 5, zoomable for as long as the server runs. `--md`
writes a second copy with bare links, for GitHub or chat:
```bash
flexviz report findings.md -o findings.html --md findings.expanded.md
```
## API cheat-sheet
Defaults shown. The `#` notes mark the arguments whose meaning the signature
does not give: those are where a call runs cleanly and charts the wrong thing.
For the rest, read `flexviz/figure.py` or https://docs.flexviz.tech.
```python
Dashboard(data, cache=False) # data: pl.LazyFrame/DataFrame, pandas, pyarrow
# cache=True enables cross-filter cubes (live brushing)
dash.add_figure(title=...) # -> Figure; chainable builders below
dash.link_axes(on="col") # or (fig_a, fig_b, axis="x") or ((fig, "x"), ...);
# linked axes share one range; numeric/temporal only
dash.share_url(server_url, source_name, rows=None, cols=None,
draggable=None, # GridStack, locked initially; False = static/read-only
cache=None, live_brush=None, layout=None)
LayoutSpec(gap="8px", draggable=True, grid_editable=False,
grid_items=None, # [GridItem(...)]; None auto-places the figures
toolbar=ToolbarConfig()) # show_reset/deselect/cfmode/hover/
# lock_all_axes/wheel/grid/share/export/import
GridItem(fig_uid, x=0, y=0, w=6, h=5) # 12-column grid; height is h * 80 px
history.add(url, note="", actor="agent") # -> int; the only place a URL belongs
fig.add_line(x, y, name=None, color=None, n_points=1000,
downsample="minmax", # or "lttb" | "fpcs" | "nth"
assume_sorted_x=False, group_by=None)
fig.add_histogram(x=None, y=None, bins=20, histnorm="count",
name=None, group_by=None)
fig.add_bar(labels, values=None, # values=None counts rows per label
agg="sum", # "mean"|"median"|"min"|"max"|"n_unique";
# ignored while values is None
orientation="v", bar_mode="group", group_by=None)
fig.add_boxplot(x=None, y=None, name=None, group_by=None)
fig.add_pie(labels, values=None, agg="sum", hole=0.0) # values=None counts rows
fig.add_treemap(path=[...], values=None, agg="sum") # path: hierarchy columns;
# values=None counts rows
fig.add_histogram2d(x, y, x_bins=20, y_bins=20,
z=None, histfunc=None) # z + histfunc are a pair: both or
# neither, either alone raises;
# no z counts rows.
fig.add_corr_heatmap(columns=None, method="pearson", triangular=False)
fig.add_geo_histogram2d(lat, lon, lat_bins=64, lon_bins=64,
z=None, histfunc=None) # same pair rule as above
# The bin grid spans the full lat/lon extent, so a few far-out points push every
# real point into one cell. Zoom the map: the grid then spans the map bounds.
fig.add_geo_line(lat, lon, n_points=1000)
fig.title(text); fig.xlabel(text); fig.ylabel(text); fig.legend(show=True)
fig.update_layout(**plotly_layout) # any Plotly layout key passes through;
# dicts merge one level deep
```
Legend placement, margins, log axes and fonts are renderer options, so they go
through `update_layout`. A legend below the chart needs `margin={"b": 80}` too.
Panel size is dashboard layout: `cols=1` gives a full-width panel and
`GridItem.h` sets the height. `group_by="col"` splits a trace into one child
per group value, with stable colors.
## Rules
Token discipline:
- Never print or restate a share URL. Refer to entries as `fv:N`. A handover
can bring one URL in; nothing may take it out again.
- Run `flexviz history show N --url` only when a human explicitly asks for
the link.
- Never read `.flexviz/history.jsonl` or `findings*.html` directly: both hold
full share URLs. Use `flexviz history list` and `show N` instead. If
`flexviz` is not on PATH, use the venv form from step 1, not the raw files.
- Poll `flexvizState({compact: true})`, never the full accessor. The full one
repeats every figure and trace, which you already know.
- Do not screenshot the dashboard to "see" it. The state is exact, pixels are
not, and a full-page screenshot costs more than the state it replaces.
Other rules:
- Never collect the full dataset into your context. The data stays in the
lazy engine.
- The history file belongs to one working directory and grows across sessions.
Numbers never restart, and an old entry still opens at `/h/N` after a server
restart, if the same file is served under the same source name. Two agents in
one directory can take the same number, so give each session its own.