# Art Computer End-to-End Architecture

*How a natural language command becomes artwork on the screen*

---

## Overview

Users can create and play artwork playlists through natural language text commands in the mobile app. The system parses these commands to generate DP-1 compliant playlists, stores them in a feed server, and delivers them to the Art Computer for playback.

---

## Architecture Components

### Core Services
- **Mobile App** - Art Computer and Toko controller
- **DP1 Feed Server** ([`dp1-feed-v2`](https://github.com/display-protocol/dp1-feed-v2)) — playlist storage and distribution (open-source reference; see [Hosted Feed (Feral File)](../../hosted-feed/index.md) for the managed option)
- **Relayer** - WebSocket bridge between mobile and device
- **Device OS** ([`ffos`](https://github.com/feral-file/ffos)) - FF OS, the Art Computer's runtime and system services
- **Device daemon** ([`ffos-user`](https://github.com/feral-file/ffos-user)) - `feral-controld`, the Go service that holds the relayer connection, serves the LAN API on port 1111, routes commands, and owns setup
- **Display Engine** - Chromium kiosk running the bundled player, driven over the Chrome DevTools Protocol

---

## Complete Flow Diagram

```mermaid
sequenceDiagram
    participant User
    participant Mobile as Mobile App
    participant CommandAPI as Command API<br/>(Cloudflare Workers)
    participant DP1Feed as DP1 Feed Server<br/>(Go/Gin + PostgreSQL)
    participant Relayer as Relayer<br/>(WebSocket Bridge)
    participant Controld as feral-controld<br/>(Go Daemon)
    participant Player as Chromium Player<br/>(HTML/JS)

    User->>Mobile: Text Command<br/>"Show me Unsupervised art"
    
    Mobile->>CommandAPI: POST /api/v1/text<br/>command + device_names
    CommandAPI->>CommandAPI: AI Intent Parsing<br/>NLP + Entity Recognition
    CommandAPI->>DP1Feed: Generate DP1 Playlist<br/>JSON with signatures
    DP1Feed-->>CommandAPI: Playlist URL + ID
    CommandAPI-->>Mobile: Stream Response<br/>{dp1_call, intent, response}
    
    Mobile->>Relayer: POST /api/cast<br/>playlist command + topicID
    Relayer->>Controld: WebSocket Message<br/>Command forwarding
    
    Controld->>Controld: Command Processing<br/>Parse, rate-limit, route
    Controld->>Player: CDP Request<br/>Runtime.evaluate (displayPlaylist)
    Player->>Player: Load Artwork<br/>from the DP-1 playlist
    
    Player-->>User: Display Artwork<br/>Visual output on screen
```

---

## Detailed Flow Steps

### 1. User Command Input

Users type natural language commands like "show me unsupervised art" or "display Refik Anadol's work" into a text input field in the mobile app. The interface provides immediate feedback and command history.

### 2. AI Command Processing

The Command API receives user commands and processes them in two stages:

#### Intent Parsing
Natural language commands are analyzed to extract:
- **Artist names** - Fuzzy matching handles variations in spelling and aliases
- **Artwork titles** - Semantic understanding of partial or informal references  
- **Action types** - Commands like "show", "display", "play", "find"
- **Duration preferences** - Time-based filtering for playlist length
- **Style attributes** - Genre, medium, or aesthetic descriptors

#### Entity Resolution
Parsed entities are matched against databases of artworks, artists, and collections using vector similarity and fuzzy string matching.

### 3. DP-1 Playlist Generation

Based on the processed command, the system generates standardized DP-1 playlists:

#### Playlist Structure
Each playlist follows the DP-1 specification with:
- **Metadata** - Title, creation timestamp, unique identifiers
- **Display Defaults** - Scaling mode, background color, margin settings
- **Playlist Items** - Ordered list of artworks with individual timing and display preferences
- **Digital Signatures** - Ed25519 cryptographic signatures for authenticity
- **Licensing Information** - Access control (open, token-gated, subscription)

#### Content Curation
The AI system selects appropriate artworks based on:
- User intent and preferences
- Artwork compatibility with display devices
- Licensing and access permissions
- Optimal viewing sequences and timing

### 4. Playlist Storage and Distribution

Generated playlists are stored in the DP-1 Feed Server system:

#### Secure Storage
- **Cryptographic Signing** - Playlists are signed with Ed25519 for tamper-proof authenticity
- **Persistent storage** - PostgreSQL holds playlist data and related metadata behind the feed API
- **Schema Validation** - Strict adherence to DP-1 specification ensures compatibility

#### Access Control
- **Signature-Authorized Writes** - A write carries no API key and no `Authorization` header. The feed verifies the signatures in the request body, and only an owner key can replace or delete a stored document (see [Hosted Feed](../../hosted-feed/index.md))
- **Content Addressing** - Immutable playlist URLs enable reliable sharing and caching
- **Licensing Enforcement** - Token-gated and subscription-based access controls

### 5. Command Relay to Device

The mobile app communicates with the Art Computer through a secure relay system:

#### Secure Routing
- **HMAC Authentication** - Topic-based message routing with cryptographic verification
- **Device Targeting** - Commands are routed to specific devices using unique identifiers
- **Real-time Communication** - WebSocket connections provide immediate command delivery

#### Message Structure
Commands include:
- **Device identification** - Target device selection and verification
- **Playlist references** - URLs or identifiers for DP-1 playlists to display
- **Display preferences** - User-specified overrides for scaling, timing, or interaction

### 6. Device Command Processing

The Art Computer receives and processes commands through the `feral-controld` daemon. The same command envelope also arrives over the local network: [ff-cli](../../ff-cli/index.md), agents, and the app on the same LAN `POST` it to `/api/cast` on port 1111, and both paths share one command router.

#### Command Routing
- **Device commands** (pairing, power, rotation, volume, sleep schedule, display control, updates, factory reset) run in the device executor
- **Player commands** (`displayPlaylist`, `refreshArtwork`, …) are forwarded to the bundled player over the Chrome DevTools Protocol
- **Storm protection** - high-cost commands are rate-limited and bounded by a global concurrency budget; a rejected command comes back as `rate_limited` over the relayer, or as HTTP `429` on the LAN API

#### Playlist Preparation
- **Direct fetch** - The player fetches the DP-1 playlist and the artworks it references directly from their URLs; playlist content never travels through the relayer
- **Offline cache** - Items can be downloaded ahead of time (`downloadPlaylist`) so playback survives a network outage
- **Display Configuration** - Apply user preferences and device-specific settings

### 7. Artwork Rendering and Display

The final step involves rendering artworks through the device's display engine:

#### Multi-Format Support
The Chromium-based player handles diverse artwork types:
- **Static Images** - PNG, JPEG, SVG with proper scaling and positioning
- **Video Content** - MP4, WebM with streaming and loop support
- **Interactive Code** - HTML5, WebGL, WebAssembly for generative and interactive art
- **Audio Elements** - Background soundtracks and audio-reactive components

#### Display Modes
- **Fit Mode** - Preserves aspect ratio with letterboxing if needed
- **Fill Mode** - Fills entire screen, cropping content if necessary  
- **Custom Scaling** - Artist-specified display preferences and interaction settings

#### Real-time Features
- **Automatic Progression** - Timed transitions between playlist items
- **Interactive Elements** - Keyboard and mouse input for interactive artworks
- **Performance Monitoring** - Frame rate and resource usage tracking

---

## Data Flow Summary

1. **User Input** → Mobile app captures text commands
2. **AI Processing** → Command API parses intent and generates playlists  
3. **Storage** → DP1 Feed Server stores signed playlists
4. **Routing** → Relayer forwards commands to specific Art Computers
5. **Device Processing** → `feral-controld` routes commands to the device executor or the player via CDP
6. **Display** → Chromium player renders artwork on screen

---

## Security & Authentication

### API Security
- **API Keys** - Bearer token authentication for Command API
- **HMAC Signatures** - Topic-based routing with cryptographic verification
- **Ed25519 Signatures** - Playlist authenticity and tamper protection

### Device Security  
- **Device Pairing** - Wi-Fi setup over the device's own hotspot and captive portal; the device is bound to an account through the cloud via a QR code or app discovery on the local network
- **Factory Reset** - Clears the pairing and Wi-Fi settings and boots the pristine factory snapshot; a reset device belongs to whoever pairs with it next
- **Snapshot System** - Btrfs snapshots with a one-shot trial boot for OTA updates and a factory snapshot kept on disk for recovery

---

## Error Handling & Recovery

### Command Processing Errors
- **Invalid Intent** - Fallback to clarification requests
- **Network Issues** - Retry logic with exponential backoff
- **Playlist Validation** - Signature verification before display

### Device Communication Errors  
- **Connection Loss** - `feral-controld` reconnects to the relayer with randomized retry delays (1–5 s after a dropped connection, 10–60 s when the endpoint is busy); if reconnection keeps failing the daemon exits and systemd restarts it. The LAN API on port 1111 stays up throughout
- **Kiosk Hangs** - `feral-watchdog` polls Chromium over the DevTools Protocol and restarts the kiosk on a hang, rebooting after repeated restarts
- **Dead Player Page** - After boot, `feral-controld` checks the player's own status and re-navigates it if the page came up before Wi-Fi did

---

## Performance Optimizations

### Streaming Response
- **NDJSON Streaming** - Progressive command processing feedback
- **Parallel Processing** - Concurrent playlist generation and validation


### Caching and persistence
- **Feed storage** - PostgreSQL in the feed service for playlist persistence and queries ([`dp1-feed-v2`](https://github.com/display-protocol/dp1-feed-v2))
- **Device Caching** - Local artwork caching on the Art Computer