go-indexus-core is the core application of the Indexus protocol, enabling decentralized data indexing and retrieval through a peer-to-peer network. It allows communities to index data and make it available in a decentralized way, focusing on efficient data availability and search capabilities.

• Key Features
• Getting Started
• Configuration
• Architecture
• API Reference
• Contributing
• License

• Decentralized Data Storage: Distributed storage of collections and items across the network
• Write-Ahead Logging: Ensures data durability through operation logging
• Automatic Data Replication: Delegates data to other nodes when collections grow too large
• Binary-Space Partitioning: Efficient data organization using BST for routing and data management
• Persistent Storage: Automatic saving of collections to disk with backup functionality
• Monitoring Interface: HTTP endpoints for monitoring node status and network health

• Go 1.22.5 or later
• Git

1. Clone the repository:

git clone https://github.com/indexus/go-indexus-core.git cd go-indexus-core

2. Install dependencies:

go mod tidy

The project includes several launch configurations:

1. Run Simulation:

go run app/simulation/main.go -port 2100

2. Run Node 1 (Bootstrap):

go run app/node/main.go -p2pPort 21001 -monitoringPort 19001

3. Run Node 2 (Peer):

go run app/node/main.go -bootstrap "127.0.0.1|21001" -p2pPort 21002 -monitoringPort 19002

1. Collections

   • Manages sets of items with their locations
   • Implements Write-Ahead Logging for durability
   • Automatic data delegation when size thresholds are reached

2. Node

   • Handles peer-to-peer communication
   • Manages routing table and network topology
   • Implements Kademlia-like routing algorithm

3. Worker

   • Performs periodic tasks:
     • Collection saving (every 5 minutes)
     • Network observation
     • Data refresh and updates
     • Operation replay on startup

4. Storage

   • Binary file format for collections
   • Write-Ahead Log for operation durability
   • Automatic backup system

1. Binary Search Tree (BST)

   • Used for routing table management
   • Efficient peer lookup and management
   • Thread-safe implementation

2. Collection

   • Manages sets of items
   • Handles data delegation
   • Implements efficient binary encoding

3. Queue

   • Thread-safe implementation
   • Used for asynchronous operation processing

• POST /ping - Node discovery and health check
• GET /neighbors - Retrieve neighboring nodes
• GET /random - Get a random peer
• POST /transfer - Transfer items between nodes
• GET /set - Retrieve items from a collection
• POST /item - Add new items to a collection

• GET /acknowledged - List acknowledged nodes
• GET /registered - List registered nodes
• GET /routing - View routing table
• GET /ownership - View owned collections
• GET /queue - Check operation queue status

The system implements several persistence mechanisms:

1. Collection Storage

   • Binary format for efficient storage
   • Automatic periodic saving
   • Backup creation on successful saves

2. Write-Ahead Log

   • Records all operations before execution
   • Enables recovery after crashes
   • Cleared after successful collection saves

3. Backup System

   • Creates timestamped backups
   • Handles corrupted file recovery
   • Maintains data integrity

1. Fork the repository
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

This project is licensed under the MIT License - see the LICENSE file for details.