overthinkos/jupyter

jupyter -- Lightweight JupyterLab with real-time collaboration

Layer Properties

| Property | Value | |----------|-------| | Dependencies | supervisord | | Sub-layers | jupyter-mcp | | Ports | 8888 | | Service | jupyter (supervisord) | | Volume | workspace at ~/workspace | | MCP provides | jupyter at http://{{.ContainerName}}:8888/mcp (streamable HTTP) | | Install files | layer.yml (rpm: git), pixi.toml |

Key Packages

conda-forge: JupyterLab >= 4.4.0, ipywidgets, ipykernel, jupyterlab-git, jupyter-resource-usage, matplotlib, seaborn, pandas, numpy, scikit-learn, scipy, polars, pyarrow, duckdb, black, pytest

PyPI: jupyter-collaboration >= 4.1.0

RPM: git

Environment

No custom environment variables. Inherits PATH from pixi layer (~/.pixi/envs/default/bin).

Collaboration

Real-time collaborative editing via jupyter-collaboration (Y-CRDT). Multiple users editing the same notebook see shared cursors and live changes. The backend uses jupyter_server_ydoc with pycrdt-websocket for CRDT synchronization. No external database required — document state is managed in-memory by default, with optional SQLite persistence.

Collaboration is enabled by default when jupyter-collaboration is installed. The WebSocket endpoint at /api/collaboration/room/ handles document sync.

Token Authentication

The service supports optional token-based authentication via the JUPYTER_TOKEN_FILE environment variable. If set and the file exists, its contents are used as the auth token. Otherwise, authentication is disabled (empty token, no password) for development convenience.

Volume

The workspace volume is mounted at ~/workspace. JupyterLab serves notebooks from this directory via --notebook-dir=$HOME/workspace.

Usage

# image.yml
jupyter:
  base: fedora
  layers:
    - agent-forwarding
    - jupyter
    - dbus
    - ov
  ports:
    - "8888:8888"

Differences from jupyter-ml Layer

| | jupyter | jupyter-ml | |---|---|---| | Base dependency | supervisord | cuda, supervisord | | GPU | No | CUDA required | | Platforms | amd64 + arm64 | amd64 only | | Focus | Collaboration + data science | ML/AI training | | Key packages | jupyter-collaboration, pandas, polars | PyTorch, vLLM, Unsloth, llama.cpp | | Image size | ~3.4 GB | ~15+ GB |

MCP Server Discovery

The jupyter layer declares mcp_provides for cross-container and pod MCP discovery. The URL template http://{{.ContainerName}}:8888/mcp resolves to the actual container name at deploy time (e.g., http://ov-jupyter:8888/mcp), or to http://localhost:8888/mcp in combined images where both services run in the same container.

• Transport: Streamable HTTP (http)
• 13 tools available via the jupyter-mcp extension (see below)
• Hermes auto-configures via the OV_MCP_SERVERS env var -- no manual MCP registration needed
• Pod-aware: resolves to localhost in combined images where hermes and jupyter share a container

MCP Server Extension (jupyter_mcp)

The layer includes a built-in MCP (Model Context Protocol) server at /mcp on port 8888. AI agents can create, read, edit, and execute notebook cells programmatically — with changes syncing live to all connected JupyterLab clients via CRDT.

Architecture

Tornado (Jupyter Server, :8888)
  └── /mcp → TornadoASGIHandler → FastMCP ASGI App (Starlette, in-process)
                                      └── RTCAdapter → YNotebook (CRDT)

• FastMCP v3.x (standalone, by Prefect) with Streamable HTTP transport (MCP spec 2025-11-25)
• Tornado-ASGI bridge — custom handler translates Tornado requests to ASGI scope/receive/send (both share the same asyncio event loop)
• On-demand room creation — CRDT rooms are created headlessly when tools need them (no browser required). Replicates YDocWebSocketHandler.prepare() logic including lazy websocket server start
• Per-notebook asyncio locks — serializes mutations on the same notebook to prevent index-based race conditions during concurrent access
• Lazy YDocExtension resolution — resolved on first tool call, not at extension load time (avoids load-order dependency)

MCP Tools (13 total)

Notebook Management: | Tool | Parameters | Returns | |------|-----------|---------| | list_notebooks | — | [{"path": "...", "name": "..."}] | | get_notebook | path | Full notebook dict (CRDT if room open, else disk) | | create_notebook | path | {"path": "...", "name": "..."} |

Cell Operations (CRDT — changes sync live to all collaborators): | Tool | Parameters | Returns | |------|-----------|---------| | get_cell | path, index | Cell dict (source, cell_type, metadata, outputs) | | update_cell | path, index, source, cell_type? | "Cell N updated" | | insert_cell | path, index, source, cell_type="code" | "Cell inserted at index N" | | delete_cell | path, index | "Cell N deleted" | | execute_cell | path, index | [{"type": "stream", "content": {...}}] |

Room Management: | Tool | Parameters | Returns | |------|-----------|---------| | open_notebook_session | path | "Collaboration room opened for ..." | | close_notebook_session | path | "Collaboration room closed for ..." |

Change Watching: | Tool | Parameters | Returns | |------|-----------|---------| | watch_notebook | path, timeout=30 | {"changed": true, "cell_count": N} or {"changed": false} |

Collaboration Awareness: | Tool | Parameters | Returns | |------|-----------|---------| | get_active_users | — | [{"id": "...", "name": "..."}] | | get_active_sessions | — | [{"room_id": "..."}] |

Installation

The MCP extension is installed by the jupyter-mcp sub-layer (composed via layers: [jupyter-mcp]):

1. pip install "fastmcp>=3.2.0" + opentelemetry runtime deps (not pixi -- cross-platform resolver conflicts with opentelemetry-api on aarch64)
2. pip install --no-deps /ctx/jupyter_mcp (extension package from sub-layer directory)
3. Extension enabled via jupyter_server_config.d/jupyter_mcp.json

Source files

layers/jupyter-mcp/
├── jupyter_mcp/
│   ├── pyproject.toml              # hatchling package, no deps (--no-deps install)
│   └── jupyter_mcp/
│       ├── __init__.py             # Extension entry point
│       ├── app.py                  # Registers /mcp handler + ASGI lifespan
│       ├── tornado_asgi.py         # Tornado↔ASGI bridge (SSE streaming, disconnect handling)
│       ├── mcp_server.py           # FastMCP tool definitions (13 tools)
│       └── rtc_adapter.py          # CRDT access via YNotebook + kernel execution
# (in layer.yml tasks:)  # Build-time: pip install fastmcp + extension + enable
└── layer.yml

Multi-client support

Multiple MCP clients can edit the same notebook simultaneously:

• All sessions share the same CRDT document via the singleton RTCAdapter
• Per-notebook asyncio locks prevent index corruption during concurrent mutations
• watch_notebook uses a fan-out NotebookWatcher — each watcher gets an independent asyncio.Event, signaled when any CRDT change occurs
• The CRDT layer (pycrdt) handles merge conflicts automatically

Implementation Notes

• The supervisord service command calls jupyter lab directly, not via pixi run. Pixi multi-stage builds copy only ~/.pixi/envs/ to the final image — pixi.toml is not present at runtime.
• The start-jupyter pixi task in pixi.toml is for build-time verification only.

Tests

The layer ships 2 declarative checks embedded in the org.overthinkos.tests OCI label (see /ov:test for the full schema):

• Build-scope (run under ov image test):
  • workspace-dir — ${HOME}/workspace exists as a directory
• Deploy-scope (run under ov test against a live service):
  • jupyter-api — GET http://${CONTAINER_IP}:${HOST_PORT:8888}/api returns 200 with a body containing "version". The ${HOST_PORT:8888} substitution means the check works unchanged when deploy.yml remaps the host port.

Used In Images

• /ov-images:jupyter

Related Layers

• /ov-layers:jupyter-ml -- GPU-accelerated variant with full CUDA ML stack + same CRDT MCP server
• /ov-layers:jupyter-mcp -- MCP server implementation (sub-layer, 13 tools for programmatic notebook access)
• /ov-layers:notebook-templates -- Starter notebooks (data layer, used alongside this layer in images)
• /ov-layers:hermes -- MCP consumer (mcp_accepts)
• /ov-layers:supervisord -- process manager dependency
• /ov-layers:python -- Python runtime (transitive via supervisord)
• /ov:mcp -- end-to-end testing of the layer's MCP endpoint (ov test mcp ping, list-tools, call); the layer ships 3 deploy-scope mcp: declarative checks against list_notebooks/insert_cell/execute_cell

When to Use This Skill

Use when the user asks about:

• Real-time collaborative Jupyter notebooks
• jupyter-collaboration or Y-CRDT setup
• Lightweight Jupyter without GPU/ML dependencies
• Port 8888 service (check if GPU variant intended)
• The jupyter layer
• MCP server for notebook manipulation
• Programmatic notebook access from AI agents
• watch_notebook or change notifications
• Multi-client concurrent editing

Related

• /ov:layer — layer authoring reference (layer.yml schema, task verbs, service declarations)