The wal (Write-Ahead Logging) project aims to provide an efficient log management system. This system ensures high reliability by using a write-ahead logging mechanism.
- Disk based
- Append only write
The Index struct is encoded into a 20-byte format as follows:
+-------------------+--------------------+-------------------+ | Index (8B) | MetadataOffset (8B)| MetadataSize (4B) | +-------------------+--------------------+-------------------+ The metadata struct is encoded into format as follows:
// Layout of Data struct: +----------+-------------+----- ... ----+ | Size (4B) | Index (8B) | LogMetadata | +----------+-------------+----- ... ----+ // Layout of LogMetadata struct: +----------------+-------------+----------------+------------+-------------+ | SegmentID (4B) | Size (4B) | Sequence (4B) | CRC (4B) | Offset (8B) | +----------------+-------------+----------------+------------+-------------+ The metadata struct is encoded into format as follows:
+----- ... ----+ | Payload | +----- ... ----+ To install the wal package, use the following command:
go get github.com/ISSuh/walTo create a new storage instance, use the NewStorage function:
package main import ( "fmt" "github.com/ISSuh/wal" ) func main() { options := wal.Options{ Path: "/path/to/log/storage", SegmentFileSize: 1024 * 1024, // default segment size is 1 GB SyncAfterWrite: true, // sync file when after wrtie } storage, err := wal.NewStorage(options) if err != nil { fmt.Printf("failed to create storage: %v", err) return } defer storage.Close() }To write data to the storage, use the Write method:
data := []byte("example log data") index, err := storage.Write(data) if err != nil { log.Fatalf("failed to write data: %v", err) } log.Printf("data written at index: %d", index)To read data from the storage, use the Read method:
readData, err := storage.Read(index) if err != nil { log.Fatalf("failed to read data: %v", err) } log.Printf("read data: %s", string(readData))To ensure all data is flushed to disk, use the Sync method:
if err := storage.Sync(); err != nil { log.Fatalf("failed to sync data: %v", err) }If SyncAfterWrite option is true, don't need to call Sync method
To properly close the storage and release resources, use the Close method:
if err := storage.Close(); err != nil { log.Fatalf("failed to close storage: %v", err) }package main import ( "github.com/ISSuh/wal" ) func main() { options := wal.Options{ Path: "/path/to/log", } log, err := wal.NewLog(options) if err != nil { panic(err) } defer log.Close() // Append a log entry entry := []byte("log entry") err = log.Append(entry) if err != nil { panic(err) } }BenchmarkWrite-11 10000 2501022 ns/op 144 B/op 5 allocs/op BenchmarkWriteWithSyncAfterWrite-11 10000 7084083 ns/op 144 B/op 5 allocs/op BenchmarkRead-11 10000 2290 ns/op 160 B/op 6 allocs/opThis project is licensed under the MIT License. See the LICENSE file for details.