Page cover

Data Storage Architecture

JuliaOS utilizes several mechanisms for storing different types of data, ranging from runtime state to persistent configurations and blockchain transaction records.

1. Runtime State (In-Memory)

  • Location: Held within the running Julia backend process.

  • Data:

    • AgentSystem.ACTIVE_AGENTS: A global Dict storing the AgentState (including configuration, current memory, skill status, connections, message queue) for all currently active agents.

    • AgentSystem.ACTIVE_SWARMS: A global Dict storing the SwarmState (including configuration, algorithm state, member agent IDs, metrics, communication logs) for active swarms.

  • Persistence: None. This data is lost when the Julia backend process stops or restarts.

  • Purpose: Enables fast access to the current operational state of agents and swarms during execution.

2. Persistent Configuration & Core Data (SQLite)

  • Location: Local SQLite database file (~/.juliaos/juliaos.sqlite by default).

  • Module: Managed by Storage.jl.

  • Data Stored:

    • Agent configurations (agents table).

    • Swarm configurations (swarms table).

    • Agent-Swarm relationships (swarm_agents table).

    • Blockchain transaction records (transactions table - hash, status, chain, etc.).

    • API Keys for external services (api_keys table - potential security concern if not encrypted).

    • General settings (settings table - key/value).

    • Document/Vector data for LangChain (DocumentStorage.jl tables).

  • Persistence: Yes. Data survives backend restarts.

  • Purpose: Stores the fundamental setup and persistent records of agents, swarms, and system operations.

3. Decentralized Storage (Arweave - Optional)

  • Location: Arweave blockchain.

  • Module: Managed by Storage.jl via the ArweaveStorage.jl submodule.

  • Data Stored: Can be configured to store agent data, swarm data, or arbitrary data blobs, typically identified by Arweave transaction IDs and tags.

  • Persistence: Yes. Permanent and decentralized storage.

  • Purpose: Provides an option for immutable, censorship-resistant storage of agent/swarm definitions or results, potentially for sharing or long-term archiving.

4. Configuration Files

  • Location: /julia/config/ directory (e.g., config.toml, tokens.json).

  • Module: Loaded by config.jl and specific modules like Bridge.jl.

  • Data Stored: Backend server settings (port, host, log level), token address mappings (bridge_token_map).

  • Persistence: Yes. Stored as files in the repository or deployment.

  • Purpose: Static configuration required for the backend to start and operate correctly.

5. Log Files

  • Location: /julia/logs/ directory (e.g., server.log, server_run.log).

  • Module: Written to by Julia's Logging system, configured potentially in julia_server.jl or startup scripts.

  • Data Stored: Runtime operational logs, debug messages, errors, warnings.

  • Persistence: Yes, as files, but typically rotated or managed by external logging systems in production.

  • Purpose: Debugging, monitoring, and auditing system behavior.

Data Flow Summary

  • Agent/Swarm configurations are loaded from SQLite (Storage.jl) into in-memory AgentState/SwarmState when created or activated.

  • Runtime operations update the in-memory state (ACTIVE_AGENTS/ACTIVE_SWARMS).

  • Persistent changes (like creating/deleting agents/swarms, updating config via API) should be written back to the SQLite database via Storage.jl.

  • Blockchain transaction submissions via Bridge.jl are recorded in the SQLite transactions table by Storage.jl.

  • Arweave storage is used optionally via explicit calls to ArweaveStorage.jl functions within Storage.jl.

  • Backend settings and token maps are read from config files at startup.

  • Logs are continuously written to log files.