Data Storage Architecture

JuliaOS utilizes several mechanisms for storing different types of data, ranging from runtime state to persistent configurations and blockchain transaction records.

1. Runtime State (In-Memory)

• Location: Held within the running Julia backend process.

• Data:

  • AgentSystem.ACTIVE_AGENTS: A global Dict storing the AgentState (including configuration, current memory, skill status, connections, message queue) for all currently active agents.

  • AgentSystem.ACTIVE_SWARMS: A global Dict storing the SwarmState (including configuration, algorithm state, member agent IDs, metrics, communication logs) for active swarms.

• Persistence: None. This data is lost when the Julia backend process stops or restarts.

• Purpose: Enables fast access to the current operational state of agents and swarms during execution.

2. Persistent Configuration & Core Data (SQLite)

• Location: Local SQLite database file (~/.juliaos/juliaos.sqlite by default).

• Module: Managed by Storage.jl.

• Data Stored:

  • Agent configurations (agents table).

  • Swarm configurations (swarms table).

  • Agent-Swarm relationships (swarm_agents table).

  • Blockchain transaction records (transactions table - hash, status, chain, etc.).

  • API Keys for external services (api_keys table - potential security concern if not encrypted).

  • General settings (settings table - key/value).

  • Document/Vector data for LangChain (DocumentStorage.jl tables).

• Persistence: Yes. Data survives backend restarts.

• Purpose: Stores the fundamental setup and persistent records of agents, swarms, and system operations.

3. Decentralized Storage (Arweave - Optional)

• Location: Arweave blockchain.

• Module: Managed by Storage.jl via the ArweaveStorage.jl submodule.

• Data Stored: Can be configured to store agent data, swarm data, or arbitrary data blobs, typically identified by Arweave transaction IDs and tags.

• Persistence: Yes. Permanent and decentralized storage.

• Purpose: Provides an option for immutable, censorship-resistant storage of agent/swarm definitions or results, potentially for sharing or long-term archiving.

4. Configuration Files

• Location: /julia/config/ directory (e.g., config.toml, tokens.json).

• Module: Loaded by config.jl and specific modules like Bridge.jl.

• Data Stored: Backend server settings (port, host, log level), token address mappings (bridge_token_map).

• Persistence: Yes. Stored as files in the repository or deployment.

• Purpose: Static configuration required for the backend to start and operate correctly.

5. Log Files

• Location: /julia/logs/ directory (e.g., server.log, server_run.log).

• Module: Written to by Julia's Logging system, configured potentially in julia_server.jl or startup scripts.

• Data Stored: Runtime operational logs, debug messages, errors, warnings.

• Persistence: Yes, as files, but typically rotated or managed by external logging systems in production.

• Purpose: Debugging, monitoring, and auditing system behavior.

Data Flow Summary

• Agent/Swarm configurations are loaded from SQLite (Storage.jl) into in-memory AgentState/SwarmState when created or activated.

• Runtime operations update the in-memory state (ACTIVE_AGENTS/ACTIVE_SWARMS).

• Persistent changes (like creating/deleting agents/swarms, updating config via API) should be written back to the SQLite database via Storage.jl.

• Blockchain transaction submissions via Bridge.jl are recorded in the SQLite transactions table by Storage.jl.

• Arweave storage is used optionally via explicit calls to ArweaveStorage.jl functions within Storage.jl.

• Backend settings and token maps are read from config files at startup.

• Logs are continuously written to log files.