Page cover

Storage

JuliaOS utilizes different storage mechanisms for various types of data, including configuration, runtime state, persistent records, and decentralized options.

Overview

JuliaOS implements a comprehensive storage system with multiple layers to meet different persistence, performance, and decentralization requirements:

  1. In-Memory Runtime State: Fast access for active agents/swarms, but volatile

  2. SQLite Database: Persistent local storage for configurations, agent/swarm definitions, and transaction history

  3. Arweave: Decentralized, permanent storage for archiving or sharing data

  4. IPFS: Distributed file storage for content-addressable data

  5. Configuration Files: Static backend settings and mappings

  6. Log Files: Operational logs for debugging and monitoring

Local Storage: SQLite (Storage.jl)

  • Location: ~/.juliaos/juliaos.sqlite

  • Purpose: Primary persistent storage for structured data

  • Schema Includes:

    • agents: Agent definitions, configurations, and states

    • swarms: Swarm definitions, algorithms, and configurations

    • swarm_agents: Mapping between swarms and their member agents

    • transactions: Blockchain transaction history and status

    • api_keys: Securely stored API keys for external services

    • settings: System-wide and user preferences

    • documents: Text documents for LangChain and RAG applications

    • vectors: Vector embeddings for semantic search

    • tasks: Agent and swarm task history and results

    • memory: Agent memory storage

  • Key Features:

    • CRUD operations for all major entities

    • JSON handling for complex configuration fields

    • Transaction tracking and history

    • Settings management

    • Document and vector storage for LangChain integration

    • Efficient querying and indexing

Decentralized Storage: Arweave (ArweaveStorage.jl)

  • Purpose: Permanent, decentralized storage for long-term data preservation

  • Implementation: Fully implemented in ArweaveStorage.jl with integration through Storage.jl

  • Key Features:

    • Store and retrieve agent/swarm data on the Arweave network

    • Upload and download arbitrary files

    • Permanent storage with one-time payment

    • Content addressing for data integrity

    • Immutable storage for audit trails and provenance

  • Configuration: Set up via Storage.configure_arweave(gateway_url, wallet_path)

  • Use Cases:

    • Archiving important agent configurations and results

    • Storing training data and model weights

    • Creating permanent records of transactions and decisions

    • Sharing agent configurations and swarm algorithms publicly

Distributed Storage: IPFS (IPFSStorage.jl)

  • Purpose: Content-addressable, distributed file storage

  • Implementation: Implemented in IPFSStorage.jl with integration through Storage.jl

  • Key Features:

    • Store and retrieve data using content-based addressing

    • Distributed storage across the IPFS network

    • Efficient for larger files and datasets

    • Content verification through cryptographic hashing

  • Configuration: Set up via Storage.configure_ipfs(gateway_url, api_key)

  • Use Cases:

    • Storing larger datasets for agent training

    • Sharing data between agents and swarms

    • Distributing agent configurations and algorithms

    • Storing intermediate results from swarm optimizations

Document/Vector Storage (DocumentStorage.jl)

  • Purpose: Storage for text documents and vector embeddings to support LangChain applications

  • Implementation: Fully implemented in DocumentStorage.jl with SQLite backend

  • Key Features:

    • Add, search, and delete documents

    • Store and retrieve vector embeddings

    • Support for semantic search using vector similarity

    • Integration with LangChain for RAG applications

    • Efficient indexing for fast retrieval

  • Use Cases:

    • Building knowledge bases for agents

    • Implementing Retrieval-Augmented Generation (RAG)

    • Creating semantic search capabilities

    • Storing agent-generated content with metadata

Memory Storage (MemoryStorage.jl)

  • Purpose: Persistent memory for agents to store and retrieve information

  • Implementation: Implemented in MemoryStorage.jl with SQLite backend

  • Key Features:

    • Key-value storage for agent memory

    • Support for structured and unstructured data

    • Efficient retrieval and updating

    • Memory persistence across agent restarts

  • Use Cases:

    • Storing agent state and learned information

    • Maintaining context between agent tasks

    • Implementing agent learning and adaptation

    • Sharing information between agents

Configuration Files (/julia/config/)

  • Purpose: Store static configuration needed at startup

  • Examples:

    • config.toml: Server settings, ports, and global configurations

    • tokens.json: Token address mappings for different chains

    • dex.json: DEX configurations and contract addresses

    • chains.json: Blockchain network configurations

Log Files (/julia/logs/)

  • Purpose: Record operational events, errors, and debug information

  • Files: server.log, server_run.log, agents.log, swarms.log, etc.

  • Features: Configurable log levels, rotation, and formatting

Data Flow Summary

  1. Initialization: Configurations are loaded from SQLite into memory at startup/activation

  2. Runtime: Changes happen in memory for performance

  3. Persistence: Updates are saved back to SQLite for local persistence

  4. Decentralization: Critical data can be stored on Arweave or IPFS for permanence or distribution

  5. Transactions: Blockchain transactions are recorded in SQLite for history and tracking

  6. Documents/Vectors: Added to SQLite when needed for LangChain and RAG applications

Storage Selection Guidelines

  • SQLite: Use for most agent and swarm data, configurations, and transaction history

  • Arweave: Use for permanent storage of critical data and public sharing with permanence

  • IPFS: Use for larger datasets, content-addressable storage, and distributed access

  • In-Memory: Use for high-performance, temporary data during agent/swarm execution

See Data Storage Architecture for more technical details on implementation.