Data Management

Overview

ALOS DB provides a complete data management toolkit built directly into the database. No external tools, no third-party backup agents, no cron scripts. Everything is a single API call.

Export — stream any collection or entire database to ALOS binary, JSONL, JSON, or MongoDB-compatible formats
Import — load data from any supported format with automatic format detection, streaming parsing, and batch insertion
Automatic Backups — schedule recurring backups with configurable intervals, retention limits, and custom storage paths
Manual Backups — trigger an instant backup at any time with a single call
Restore — restore from any backup with automatic safety snapshots and manifest verification
ZIP Support — export collections as compressed ZIP archives and import from ZIP files

Safety first. Every restore operation automatically creates a safety snapshot of your current data before overwriting anything. If something goes wrong, you can always recover from the pre-restore snapshot.

Export Formats

ALOS DB supports four export formats. Each has different tradeoffs for size, compatibility, and readability.

Format	Constant	Structure	Multi-Collection	Best For
ALOS Binary	`FormatALOS`	Header + JSONL records with collection tags	Yes	Full database backup & restore
JSONL	`FormatJSONL`	One JSON object per line	Yes	Streaming, large datasets, pipelines
JSON	`FormatJSON`	Single JSON array or object	Yes	Human-readable, small exports
MongoJL	`FormatMongoJL`	MongoDB-compatible JSONL	Yes	Migration from/to MongoDB

ALOS Binary Format

The ALOS format is the recommended format for backup and restore operations. It uses a header line with database metadata followed by one JSONL record per document, each tagged with its collection name. This format preserves the full database structure including all collections and their document IDs.

ALOS Format Structure

// Line 1: Header
{"version": 1, "database": "myapp", "created_at": "2026-01-15T10:30:00Z", "collections": ["users", "orders"]}

// Lines 2+: One document per line, tagged with collection
{"collection": "users", "doc": {"_id": "abc123", "name": "Alice", "email": "alice@example.com"}}
{"collection": "orders", "doc": {"_id": "ord456", "user_id": "abc123", "total": 99.99}}

JSONL Format

JSONL (JSON Lines) writes one JSON object per line. For database-level exports, the first line is a metadata record with __alos_export: true and each subsequent document includes a __collection tag. For single-collection exports, each line is a raw document with no metadata.

JSON Format

Standard JSON with optional pretty-printing. For single collections, this is a JSON array of documents. For database-level exports, this is a JSON object with database metadata and a collections map of collection-name to document-array. Best for small exports that need to be human-readable or consumed by tools that don't support streaming.

MongoJL Format

MongoDB-compatible JSONL. Identical in wire format to JSONL but uses the FormatMongoJL constant for semantic clarity when migrating data between ALOS DB and MongoDB. Documents are written one-per-line in standard JSON.

Export a Collection

Export a single collection to any supported format. The export streams documents one at a time — it never loads all documents into memory, so it works efficiently even for collections with millions of documents.

Go

// Export as JSONL (streaming, memory-efficient)
f, _ := os.Create("users_export.jsonl")
defer f.Close()

err := db.ExportCollection(f, "users", core.FormatJSONL, false)
if err != nil {
    log.Fatal(err)
}

// Export as pretty JSON (human-readable)
f2, _ := os.Create("users_export.json")
defer f2.Close()

err = db.ExportCollection(f2, "users", core.FormatJSON, true)

// Export as ALOS binary (for backup/restore)
f3, _ := os.Create("users_export.alos")
defer f3.Close()

err = db.ExportCollection(f3, "users", core.FormatALOS, false)

Method Signature

Go

func (db *Database) ExportCollection(
    w    io.Writer,       // destination (file, HTTP response, buffer, etc.)
    coll string,          // collection name
    fmt  ExportFormat,    // FormatALOS | FormatJSONL | FormatJSON | FormatMongoJL
    pretty bool,         // indent JSON output (FormatJSON only)
) error

The writer can be anything that implements io.Writer: a file, an HTTP response body, a network connection, a buffer, or even os.Stdout. The export uses a 256KB buffered writer internally for optimal I/O throughput.

Export Entire Database

Export all collections in a database (or a specific subset) in a single call. The output includes metadata headers so the data can be imported back into ALOS DB with full structure preservation.

Go

// Export entire database as ALOS binary
f, _ := os.Create("full_backup.alos")
defer f.Close()

err := db.ExportDatabase(f, core.ExportOptions{
    Format: core.FormatALOS,
})

// Export only specific collections as JSONL
f2, _ := os.Create("partial_backup.jsonl")
defer f2.Close()

err = db.ExportDatabase(f2, core.ExportOptions{
    Format:      core.FormatJSONL,
    Collections: []string{"users", "orders"},
})

// Export as pretty JSON for inspection
f3, _ := os.Create("readable_backup.json")
defer f3.Close()

err = db.ExportDatabase(f3, core.ExportOptions{
    Format: core.FormatJSON,
    Pretty: true,
})

ExportOptions

Go

type ExportOptions struct {
    Format      ExportFormat  // output format (required)
    Collections []string      // specific collections (empty = all)
    Pretty      bool          // indent JSON output
}

When Collections is empty, all collections in the database are exported in alphabetical order. Each collection is streamed sequentially — the entire export uses constant memory regardless of database size.

ZIP Export

Export a collection as a compressed ZIP archive. Each document is stored as an individual JSON file inside the ZIP, organized by document ID. This is useful for archival, S3 upload, or file-based processing pipelines.

Go

// Export collection as ZIP
f, _ := os.Create("users_archive.zip")
defer f.Close()

err := db.ExportCollectionZip(f, "users")
if err != nil {
    log.Fatal(err)
}
// Result: users_archive.zip containing:
//   users/abc123.json
//   users/def456.json
//   users/ghi789.json
//   ...

ZIP export uses flate.BestSpeed compression for fast throughput. Each document is pretty-printed with 2-space indentation for readability. The ZIP writer streams documents directly from disk — no full collection materialization in memory.

Importing Data

ALOS DB can import data from any of its supported formats. The import system uses streaming parsers that process documents one at a time, so you can import datasets that are larger than available memory.

Import Options

Go

type ImportOptions struct {
    Format        ExportFormat  // hint format (auto-detected if empty)
    Collection    string        // target collection (single-collection import)
    KeepIDs       bool          // preserve original _id values
    BatchSize     int           // documents per batch insert (default: 1000)
    FlushInterval int           // flush to disk every N documents (default: 50000)
}

Key Options Explained

KeepIDs — when true, the original _id field from each document is preserved. When false (default), ALOS DB generates new unique IDs for each imported document. Use true when restoring from a backup, false when importing data from external sources.
BatchSize — documents are accumulated into batches and inserted via InsertMany() for throughput. Larger batches = faster import, more memory. Default is 1000 documents per batch.
FlushInterval — controls how often the database flushes data to disk during import. Range: 5,000 to 600,000 documents. Default: 50,000. Lower values reduce memory usage at the cost of import speed.

Auto-detection. If you don't specify a Format, ALOS DB automatically detects the format by examining the first line of the input. It can distinguish between ALOS binary, JSONL, JSON arrays, and MongoDB exports without any configuration.

Import a Collection

Import documents into a single collection from a reader (file, HTTP request body, etc.).

Go

// Import JSONL file into "users" collection
f, _ := os.Open("users_export.jsonl")
defer f.Close()

result, err := db.ImportCollectionFromReader(f, core.ImportOptions{
    Collection: "users",
    Format:     core.FormatJSONL,
    KeepIDs:    true,
    BatchSize:  5000,
})

fmt.Printf("Imported %d docs into %s (%d errors)\n",
    result.Inserted, result.Collection, result.Errors)

ImportCollectionResult

Go

type ImportCollectionResult struct {
    Collection string // target collection name
    Inserted   int64  // number of documents successfully inserted
    Errors     int64  // number of documents that failed to insert
}

Import a Database

Import an entire database from an ALOS binary, JSONL, or JSON export. The importer automatically routes documents to their correct collections based on collection tags in the export file.

Go

// Import full database from ALOS binary backup
f, _ := os.Open("full_backup.alos")
defer f.Close()

result, err := db.ImportDatabaseFromReader(f, core.ImportOptions{
    KeepIDs:   true,
    BatchSize: 10000,
})

fmt.Printf("Total imported: %d docs across %d collections\n",
    result.TotalInserted, len(result.Collections))

// Per-collection breakdown
for name, cr := range result.Collections {
    fmt.Printf("  %s: %d inserted, %d errors\n", name, cr.Inserted, cr.Errors)
}

ImportDatabaseResult

Go

type ImportDatabaseResult struct {
    TotalInserted int64                                      // total across all collections
    TotalErrors   int64                                      // total errors across all collections
    Collections   map[string]*ImportCollectionResult     // per-collection results
    Format        string                                     // detected format name
}

The importer batches documents per collection and flushes them in bulk. This is significantly faster than inserting one document at a time. Collections are created automatically if they don't exist.

ZIP Import

Import data from ZIP archives. ALOS DB supports both database-level ZIP import (multiple collections) and single-collection ZIP import.

Go

// Import entire database from ZIP
err := db.ImportDatabaseZip("full_backup.zip")

// Import a single collection from ZIP
err = db.ImportCollectionZip("users_archive.zip", "users")

The ZIP importer reads each .json file from the archive, parses the document, and inserts it into the target collection. Files are processed in batches for throughput. The ZIP file is read from disk (not streamed), so it must fit on the filesystem.

Automatic Backups

ALOS DB has a built-in backup scheduler that runs inside the database process. No external cron jobs, no backup agents, no infrastructure. Configure it once and it runs forever.

Configuration

Go

// Enable automatic backups every 60 minutes, keep last 24 copies
err := backupManager.Configure(core.BackupConfig{
    Enabled:         true,
    IntervalMinutes: 60,         // backup every hour
    MaxCopies:       24,         // keep last 24 backups (1 day rolling)
    BackupPath:      "",         // use default path (dataPath/_backups/dbname)
})

// Or use a custom backup directory
err = backupManager.Configure(core.BackupConfig{
    Enabled:         true,
    IntervalMinutes: 30,
    MaxCopies:       48,
    BackupPath:      "/mnt/backup-drive/alosdb",
})

BackupConfig

Go

type BackupConfig struct {
    Enabled         bool    // enable/disable auto-backup
    IntervalMinutes int     // minutes between backups (minimum: 1)
    MaxCopies       int     // max backups to retain (0 = unlimited)
    BackupPath      string  // custom backup directory (empty = default)
}

How Auto-Backup Works

Flush — all pending writes are flushed to disk before the backup starts
Copy — the entire database directory is copied to the backup location using a recursive directory copy
Manifest — a _manifest.json file is written inside the backup with file count, total size, and file list for verification
Rotate — if MaxCopies is set, the oldest backups are deleted to stay within the limit
Log — backup completion is logged with name and size

Persistent configuration. Backup configuration is saved to _backup_config.json inside the backup directory. On server restart, the configuration is automatically loaded and the backup scheduler resumes where it left off. No manual re-configuration needed.

Backup Status

Go

// Check current backup status
status := backupManager.Status()

fmt.Printf("Enabled: %v\n", status.Enabled)
fmt.Printf("Interval: %d min\n", status.IntervalMinutes)
fmt.Printf("Max copies: %d\n", status.MaxCopies)
fmt.Printf("Backup path: %s\n", status.BackupPath)
fmt.Printf("Total backups: %d\n", status.TotalBackups)
fmt.Printf("Currently running: %v\n", status.Running)

if status.LastBackup != nil {
    fmt.Printf("Last: %s (%.2f MB)\n",
        status.LastBackup.Name,
        float64(status.LastBackup.SizeBytes)/(1024*1024))
}
if status.NextBackupAt != nil {
    fmt.Printf("Next backup at: %s\n", status.NextBackupAt.Format(time.RFC3339))
}

BackupStatus

Go

type BackupStatus struct {
    Enabled         bool         // is auto-backup active
    IntervalMinutes int          // configured interval
    MaxCopies       int          // configured retention limit
    BackupPath      string       // resolved backup directory
    LastBackup      *BackupInfo  // most recent backup (nil if none)
    NextBackupAt    *time.Time   // next scheduled backup time
    TotalBackups    int          // current backup count on disk
    Running         bool         // backup in progress right now
}

Manual Backup

Trigger an instant backup at any time, independent of the auto-backup schedule. The backup is created immediately and follows the same process as automatic backups: flush, copy, manifest, rotate.

Go

// Create an instant backup
info, err := backupManager.BackupNow()
if err != nil {
    log.Fatal(err)
}

fmt.Printf("Backup created: %s\n", info.Name)
fmt.Printf("Path: %s\n", info.Path)
fmt.Printf("Size: %.2f MB\n", float64(info.SizeBytes)/(1024*1024))
fmt.Printf("Created at: %s\n", info.CreatedAt.Format(time.RFC3339))

BackupInfo

Go

type BackupInfo struct {
    Name      string     // "backup_20260415_120000"
    Path      string     // full filesystem path
    CreatedAt time.Time  // when the backup was created
    SizeBytes int64      // total size of all files
}

Backup names follow the pattern backup_YYYYMMDD_HHMMSS using UTC timestamps. The backup directory is a complete copy of the database directory, including all data shards, index snapshots, and metadata files. It can be used for restore or inspected manually.

Backup Manifest

Every backup includes a _manifest.json file that records the exact state of the backup:

Go

type BackupManifest struct {
    Version   int       // manifest format version
    DBName    string    // database name
    CreatedAt string    // RFC3339 timestamp
    FileCount int       // total files in backup
    TotalSize int64     // total bytes
    Files     []string  // list of all backed-up files
}

During restore, the manifest is verified to ensure the backup is complete and uncorrupted before any data is overwritten.

Restore from Backup

Restore a database from any previous backup. The restore process is designed to be safe and reversible.

Go

// List available backups
backups := backupManager.ListBackups()
for _, b := range backups {
    fmt.Printf("%s  %.2f MB  %s\n",
        b.Name,
        float64(b.SizeBytes)/(1024*1024),
        b.CreatedAt.Format(time.RFC3339))
}

// Restore from a specific backup
err := backupManager.RestoreFromBackup("backup_20260415_120000")
if err != nil {
    log.Fatal(err)
}
// Database is now restored and reloaded

Restore Safety Process

Every restore follows this exact sequence to prevent data loss:

Verify — the backup manifest is checked to confirm the backup exists and is complete
Flush — all pending writes in the current database are flushed to disk
Safety snapshot — the current database directory is copied to _pre_restore_YYYYMMDD_HHMMSS inside the backup directory. This is your escape hatch if the restore goes wrong.
Clear — the current database directory is removed
Copy — the backup is copied into the database directory
Reload — the database is reloaded from the restored files, rebuilding indexes and re-opening collections

Safety snapshots are never deleted automatically. The _pre_restore_* directories are excluded from normal backup listing and rotation. They persist until you manually delete them, giving you an unlimited window to recover from a bad restore.

Manage Backups

List All Backups

List all available backups, sorted by creation time (newest first). Pre-restore safety snapshots are excluded from this list.

Go

backups := backupManager.ListBackups()
// Returns []BackupInfo sorted by CreatedAt descending

Delete a Backup

Delete a specific backup by name. The backup name is validated to prevent path traversal attacks — names containing .., /, or \ are rejected.

Go

err := backupManager.DeleteBackup("backup_20260415_120000")

Stop Auto-Backups

Disable the automatic backup scheduler. Existing backups are not deleted.

Go

// Disable auto-backups
backupManager.Configure(core.BackupConfig{
    Enabled: false,
})

// Or stop the scheduler directly
backupManager.Stop()

Automatic Rotation

When MaxCopies is set, ALOS DB automatically deletes the oldest backups after each new backup is created. For example, with MaxCopies: 24 and IntervalMinutes: 60, you get a rolling 24-hour backup window. Old backups are deleted oldest-first.

Set MaxCopies: 0 for unlimited retention — backups will accumulate until you manually delete them or run out of disk space.

Data Management

Overview

Export Formats

ALOS Binary Format

JSONL Format

JSON Format

MongoJL Format

Export a Collection

Method Signature

Export Entire Database

ExportOptions

ZIP Export

Importing Data

Import Options

Key Options Explained

Import a Collection

ImportCollectionResult

Import a Database

ImportDatabaseResult

ZIP Import

Automatic Backups

Configuration

BackupConfig

How Auto-Backup Works

Backup Status

BackupStatus

Manual Backup

BackupInfo

Backup Manifest

Restore from Backup

Restore Safety Process

Manage Backups

List All Backups

Delete a Backup

Stop Auto-Backups

Automatic Rotation

Explore More

Indexing

Configuration

API Reference