Foxglove can search robot footage. Check what the search leaves out.

Foxglove’s new Agent Sidebar can search recordings, build visualization layouts, answer questions about robot data and run platform tools from a plain-language request. The company also added a local MCP server so outside assistants such as Claude, ChatGPT or Cursor can work with the same data through the Foxglove desktop app. Foxglove says these agent features are in beta for all customers.

The more specific change is Semantic Search. Instead of remembering a device label or exact field name, an engineer can ask for a visual event such as highway merges at night with poor lane detection. Foxglove says object and scene search works on individual frames, while action search uses NVIDIA Cosmos models to reason across a sequence.

Under the hood, Foxglove says it samples one frame per image or video topic per second during ingestion, turns those frames into embeddings and ranks matches by cosine similarity. Device, time and topic filters narrow the candidate set before the visual ranking. Smart Search and the agent-assisted query route are available now; Foxglove’s post says access to Semantic Search itself still has to be requested.

Comparison Mode is less flashy and may be the easier win. It puts several recordings on the same labeled timeline, side by side or overlaid. A team can compare a failed run with a successful one, a simulation with real hardware, or a robot before and after a code change. Foxglove says Comparison Mode is available to all customers.

Semantic search is retrieval. It depends on the words in the query, the frames that were sampled, the embedding model and the filters applied before ranking. Each dependency can be reasonable and still leave a rare event outside the result set.

The one-frame-per-second sampling detail makes this concrete. A brief slip, flicker, blocked view or unsafe approach may sit between sampled frames. Action search can inspect sequences, but Foxglove has not published a recall rate for each kind of rare robotics failure in these launch posts. The sensible reading is that the search narrows the haystack. It has not certified the haystack.

Video is only one witness anyway. A carton may look steady while grip force drifts. A vehicle may stay in lane while localization confidence drops. The promising part of Foxglove’s design is that a retrieved moment remains connected to synchronized telemetry, lidar, logs and other sensor data. Use that connection. The thumbnail is the lead, not the verdict.

Start with a small set of incidents your team has already labeled by hand. Include the obvious crash and the ugly near-misses: the grasp that recovered, the stop that came late, the camera that went dark for half a second and the run that looked normal until a current trace moved. Ask the new search several ordinary versions of the question a tired engineer would actually type.

Write down which incidents appeared, which did not and how long it took to reach the underlying sensor evidence. Repeat that set after a model, index or prompt change. A faster search is helpful. A faster search that quietly drops two known failures is a regression with better manners.

For serious investigations, keep a second route that does not depend on visual similarity. Search structured thresholds, error codes and timestamps; compare the suspect run with a baseline; then inspect the synchronized streams around the event. If a query returns nothing, say exactly that: no match found under this query and filter. Do not turn it into the much stronger sentence that nothing happened.

Ren Ortiz likes that a sentence can open the right physical moment instead of sending someone through hours of video. He would immediately pin the result to the other signals: gripper force, motor current, camera timestamp and the first frame where the box changed pose. If those disagree, the disagreement is the incident.

Priya Rao would keep a fixed set of known failures and rerun the same searches after every material update. She cares less about a polished answer than whether the tool still finds the short, expensive events a person previously found by hand. A search system earns trust by exposing its misses, not by returning an impressive first page.

Those positions are not in conflict. Ren wants the tool close to the machine. Priya wants a stable way to notice when it stops seeing what mattered. Together they turn natural-language search from a demo into a quicker route to evidence.

Robotics data is a good place for an AI assistant because the current alternative is often tedious: remember the schema, open several recordings, align timelines and scrub until the failure appears. Foxglove is putting the assistant inside that work instead of asking engineers to paste a summary into a separate chatbot.

The boundary should stay plain. Let the assistant find candidate runs, prepare a useful layout and bring related evidence together. Let the synchronized data and the person responsible for the robot decide what happened and whether the fix held.

Search is valuable because rare failures are hard to find. That same rarity is why an empty search result deserves suspicion. Save the afternoon when the tool finds the clip. Keep looking when it does not.