Module memory_backend 

Copyright © 2025-2026 Wenze Wei. All Rights Reserved.

This file is part of DMSC. The DMSC project belongs to the Dunimd Team.

Licensed under the Apache License, Version 2.0 (the “License”); You may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

In-Memory Queue Implementation

This file implements an in-memory queue backend for the DMSC queue system. The in-memory queue provides a lightweight, fast queue implementation suitable for testing, development, and scenarios where durability is not a strict requirement. It also supports optional persistence to disk for basic durability.

Key Components

• DMSCMemoryQueue: Main in-memory queue implementation
• MemoryQueueState: Internal state management for the queue
• MemoryQueueProducer: Producer implementation for sending messages
• MemoryQueueConsumer: Consumer implementation for receiving messages

Design Principles

1. Lightweight: Minimal dependencies and overhead
2. Fast Performance: In-memory operations for low latency
3. Optional Persistence: Can be configured to persist messages to disk
4. Consumer Groups: Supports multiple consumer groups with message distribution
5. Async-First: All operations are asynchronous
6. Thread-safe: Uses Arc and RwLock for safe concurrent access
7. Durable Option: Optional disk persistence for message durability
8. Simple API: Implements the standard DMSCQueue interfaces
9. Non-blocking: Uses tokio’s spawn_blocking for file I/O operations
10. Message Retry: Supports message requeueing with retry count increment

Usage

use dmsc::queue::{DMSCQueue, DMSCQueueMessage, DMSCQueueProducer, DMSCQueueConsumer};
use dmsc::queue::backends::DMSCMemoryQueue;
use dmsc::core::DMSCResult;
use serde_json::json;

async fn example() -> DMSCResult<()> {
    // Create a basic in-memory queue
    let queue = DMSCMemoryQueue::new("example_queue");
     
    // Or create a queue with disk persistence
    // let queue = DMSCMemoryQueue::with_persistence("example_queue", "/tmp/queue_persistence");
     
    // Create a producer
    let producer = queue.create_producer().await?;
     
    // Create a message
    let payload = json!({ "key": "value" }).to_string().into_bytes();
    let message = DMSCQueueMessage::new(payload);
     
    // Send the message
    producer.send(message).await?;
     
    // Create a consumer
    let consumer = queue.create_consumer("consumer_group_1").await?;
     
    // Receive a message
    if let Some(message) = consumer.receive().await? {
        // Process the message
        let payload = String::from_utf8_lossy(&message.payload);
        println!("Received message: {}", payload);
         
        // Acknowledge the message
        consumer.ack(&message.id).await?;
    }
     
    Ok(())
}

Structs

DMSCMemoryQueue

In-memory queue implementation.