Skip to content
feralfile.com

Art Computer End-to-End Architecture

How a natural language command becomes artwork on the screen

Overview

Users can create and play artwork playlists through natural language text commands in the mobile app. The system parses these commands to generate DP-1 compliant playlists, stores them in a feed server, and delivers them to the Art Computer for playback.

Architecture Components

Core Services

  • Mobile App - Art Computer and Digital Art System controller
  • DP1 Feed Server (dp1-feed-v2) — playlist storage and distribution (open-source reference; see Hosted Feed (Feral File) for the managed option)
  • Relayer - WebSocket bridge between mobile and device
  • Device OS (ffos) - FF OS, the Art Computer's runtime and system services
  • Device daemon (ffos-user) - feral-controld, the Go service that holds the relayer connection, serves the LAN API on port 1111, routes commands, and owns setup
  • Display Engine - Chromium kiosk running the bundled player, driven over the Chrome DevTools Protocol

Complete Flow Diagram

sequenceDiagram
    participant User
    participant Mobile as Mobile App
    participant CommandAPI as Command API<br/>(Cloudflare Workers)
    participant DP1Feed as DP1 Feed Server<br/>(Go/Gin + PostgreSQL)
    participant Relayer as Relayer<br/>(WebSocket Bridge)
    participant Controld as feral-controld<br/>(Go Daemon)
    participant Player as Chromium Player<br/>(HTML/JS)

    User->>Mobile: Text Command<br/>"Show me Unsupervised art"

    Mobile->>CommandAPI: POST /api/v1/text<br/>command + device_names
    CommandAPI->>CommandAPI: AI Intent Parsing<br/>NLP + Entity Recognition
    CommandAPI->>DP1Feed: Generate DP1 Playlist<br/>JSON with signatures
    DP1Feed-->>CommandAPI: Playlist URL + ID
    CommandAPI-->>Mobile: Stream Response<br/>{dp1_call, intent, response}

    Mobile->>Relayer: POST /api/cast<br/>playlist command + topicID
    Relayer->>Controld: WebSocket Message<br/>Command forwarding

    Controld->>Controld: Command Processing<br/>Parse, rate-limit, route
    Controld->>Player: CDP Request<br/>Runtime.evaluate (displayPlaylist)
    Player->>Player: Load Artwork<br/>from the DP-1 playlist

    Player-->>User: Display Artwork<br/>Visual output on screen

Detailed Flow Steps

1. User Command Input

Users type natural language commands like "show me unsupervised art" or "display Refik Anadol's work" into a text input field in the mobile app. The interface provides immediate feedback and command history.

2. AI Command Processing

The Command API receives user commands and processes them in two stages:

Intent Parsing

Natural language commands are analyzed to extract: - Artist names - Fuzzy matching handles variations in spelling and aliases - Artwork titles - Semantic understanding of partial or informal references
- Action types - Commands like "show", "display", "play", "find" - Duration preferences - Time-based filtering for playlist length - Style attributes - Genre, medium, or aesthetic descriptors

Entity Resolution

Parsed entities are matched against databases of artworks, artists, and collections using vector similarity and fuzzy string matching.

3. DP-1 Playlist Generation

Based on the processed command, the system generates standardized DP-1 playlists:

Playlist Structure

Each playlist follows the DP-1 specification with: - Metadata - Title, creation timestamp, unique identifiers - Display Defaults - Scaling mode, background color, margin settings - Playlist Items - Ordered list of artworks with individual timing and display preferences - Digital Signatures - Ed25519 cryptographic signatures for authenticity - Licensing Information - Access control (open, token-gated, subscription)

Content Curation

The AI system selects appropriate artworks based on: - User intent and preferences - Artwork compatibility with display devices - Licensing and access permissions - Optimal viewing sequences and timing

4. Playlist Storage and Distribution

Generated playlists are stored in the DP-1 Feed Server system:

Secure Storage

  • Cryptographic Signing - Playlists are signed with Ed25519 for tamper-proof authenticity
  • Persistent storage - PostgreSQL holds playlist data and related metadata behind the feed API
  • Schema Validation - Strict adherence to DP-1 specification ensures compatibility

Access Control

  • API Authentication - Bearer token authentication for write operations
  • Content Addressing - Immutable playlist URLs enable reliable sharing and caching
  • Licensing Enforcement - Token-gated and subscription-based access controls

5. Command Relay to Device

The mobile app communicates with the Art Computer through a secure relay system:

Secure Routing

  • HMAC Authentication - Topic-based message routing with cryptographic verification
  • Device Targeting - Commands are routed to specific devices using unique identifiers
  • Real-time Communication - WebSocket connections provide immediate command delivery

Message Structure

Commands include: - Device identification - Target device selection and verification - Playlist references - URLs or identifiers for DP-1 playlists to display - Display preferences - User-specified overrides for scaling, timing, or interaction

6. Device Command Processing

The Art Computer receives and processes commands through the feral-controld daemon. The same command envelope also arrives over the local network: ff-cli, agents, and the app on the same LAN POST it to /api/cast on port 1111, and both paths share one command router.

Command Routing

  • Device commands (pairing, power, rotation, volume, sleep schedule, panel control, updates, factory reset) run in the device executor
  • Player commands (displayPlaylist, refreshArtwork, …) are forwarded to the bundled player over the Chrome DevTools Protocol
  • Storm protection - high-cost commands are rate-limited and bounded by a global concurrency budget; a rejected command comes back as rate_limited over the relayer, or as HTTP 429 on the LAN API

Playlist Preparation

  • Direct fetch - The player fetches the DP-1 playlist and the artworks it references directly from their URLs; playlist content never travels through the relayer
  • Offline cache - Items can be downloaded ahead of time (downloadPlaylist) so playback survives a network outage
  • Display Configuration - Apply user preferences and device-specific settings

7. Artwork Rendering and Display

The final step involves rendering artworks through the device's display engine:

Multi-Format Support

The Chromium-based player handles diverse artwork types: - Static Images - PNG, JPEG, SVG with proper scaling and positioning - Video Content - MP4, WebM with streaming and loop support - Interactive Code - HTML5, WebGL, WebAssembly for generative and interactive art - Audio Elements - Background soundtracks and audio-reactive components

Display Modes

  • Fit Mode - Preserves aspect ratio with letterboxing if needed
  • Fill Mode - Fills entire screen, cropping content if necessary
  • Custom Scaling - Artist-specified display preferences and interaction settings

Real-time Features

  • Automatic Progression - Timed transitions between playlist items
  • Interactive Elements - Keyboard and mouse input for interactive artworks
  • Performance Monitoring - Frame rate and resource usage tracking

Data Flow Summary

  1. User Input → Mobile app captures text commands
  2. AI Processing → Command API parses intent and generates playlists
  3. Storage → DP1 Feed Server stores signed playlists
  4. Routing → Relayer forwards commands to specific Art Computers
  5. Device Processing → feral-controld routes commands to the device executor or the player via CDP
  6. Display → Chromium player renders artwork on screen

Security & Authentication

API Security

  • API Keys - Bearer token authentication for Command API
  • HMAC Signatures - Topic-based routing with cryptographic verification
  • Ed25519 Signatures - Playlist authenticity and tamper protection

Device Security

  • Device Pairing - Wi-Fi setup over the device's own hotspot and captive portal; the device is bound to an account through the cloud via a QR code or app discovery on the local network
  • Factory Reset - Clears the pairing and Wi-Fi settings and boots the pristine factory snapshot; a reset device belongs to whoever pairs with it next
  • Snapshot System - Btrfs snapshots with a one-shot trial boot for OTA updates and a factory snapshot kept on disk for recovery

Error Handling & Recovery

Command Processing Errors

  • Invalid Intent - Fallback to clarification requests
  • Network Issues - Retry logic with exponential backoff
  • Playlist Validation - Signature verification before display

Device Communication Errors

  • Connection Loss - feral-controld reconnects to the relayer with randomized retry delays (1–5 s after a dropped connection, 10–60 s when the endpoint is busy); if reconnection keeps failing the daemon exits and systemd restarts it. The LAN API on port 1111 stays up throughout
  • Kiosk Hangs - feral-watchdog polls Chromium over the DevTools Protocol and restarts the kiosk on a hang, rebooting after repeated restarts
  • Dead Player Page - After boot, feral-controld checks the player's own status and re-navigates it if the page came up before Wi-Fi did

Performance Optimizations

Streaming Response

  • NDJSON Streaming - Progressive command processing feedback
  • Parallel Processing - Concurrent playlist generation and validation

Caching and persistence

  • Feed storage - PostgreSQL in the feed service for playlist persistence and queries (dp1-feed-v2)
  • Device Caching - Local artwork caching on the Art Computer