What's new at a glance

"v1 non-goals" was an aspirational list. Then I read it back.

M1 — OR

M2 — logfmt and plain

M3 — --follow

M4 — prune and LOGDIVE_DB

M5 — GET /version and configurable CORS

M6 — Multi-arch Docker

The updated query grammar, in one place

Tradeoffs, honestly

Performance

Upgrading from v0.1.0

Call for contributions

A short field guide to shipping v0.2 of anything Two weeks after I shipped logdive v0.1.0, I tried using it.

Within an hour I wanted to write level=error OR level=warn. The README said no.Within a day I wanted to tail a growing log file. The README said no.Within a week I had a 1.8 GB index from a CI pipeline and no way to trim it. The README didn't say anything, because I hadn't thought about it.

logdive v0.2.0 ships everything the v0.1 README told you not to ask for. Twenty seconds: ingest a couple of structured lines, then query them with OR. No daemon. No config file. One binary. Six numbered milestones. 330 tests passing. Both binaries still under 10 MB (logdive 3.9 MB, logdive-api 4.2 MB). The compiled Docker image lands at 97 MB. The v0.1 README had a tidy little section calling six things explicit non-goals. Some of them genuinely should be non-goals (multi-machine clustering, log shipping daemons). Three of them were just "things I didn't want to write yet, in a costume." OR queries. I argued in the v0.1 post that AND covered the dominant query pattern and OR would roughly double the parser. Both were true. Both were also irrelevant the first time I needed level=error OR level=warn during a real incident and couldn't write it. "I don't want to grow the parser" is a feeling, not an engineering position. Follow mode. v0.1 was a batch tool. Pipe a file in, query it, walk away. The first time I wanted to watch a container that was actively misbehaving, the missing --follow flag turned a five-second loop into "re-ingest every thirty seconds." That's not a tool — that's a chore the tool is now causing. Non-JSON ingestion. "v1 is JSON-only" sounded principled. It stopped sounding principled the first time someone tried logdive against nginx access logs or a legacy Java service emitting Apache-style plaintext with a [ERROR] prefix. Pruning. This one I just didn't think about. Then I ran logdive against a CI pipeline for two weeks and the index hit 1.8 GB. The fix in v0.1 was rm ~/.logdive/index.db && start over. The kind of UX I'd mock in someone else's tool. So: v0.2.0 ships all four. Plus a Docker image, plus a versioned API, because if you're going to break your own scope rules you may as well do it once. The query language now has a real two-level grammar: AND binds tighter than OR, the way it does in SQL and the way your fingers expect: Parenthesised groups ((a OR b) AND c) are still out — that's the headline v0.3 milestone. AND+OR covers ~95% of real queries; the remaining 5% you can reshape with De Morgan's laws and a small sigh. Implementation note for anyone curious: it's still a hand-written recursive-descent parser, no parser-combinator library, ~340 lines of pure Rust in crates/core/src/query.rs. Two new methods (parse_or_expr, parse_and_expr), one new AST variant (AndGroup { clauses: Vec<Clause> }), and a SQL generator that always parenthesises each AND-group so WHERE clauses come out unambiguous: One breaking change for library users: QueryNode::And(Vec<Clause>) is now QueryNode::Or(Vec<AndGroup>). Even single-clause queries wrap in the two-level structure — uniformity for the executor, slight clumsiness if you were pattern-matching. CLI users see nothing. logdive ingest gained a --format flag with three values: The logfmt parser is hand-written: roughly 200 lines, no nom, no pest. It handles escaped quotes (\", \\), bareword booleans (debug → debug=true, the Heroku convention), hyphenated and dotted keys (request-id, user.id), and the last-write-wins rule on duplicate keys that go-kit/log uses. A malformed token inside an otherwise-fine line is skipped to the next whitespace boundary; only a truly fatal condition — empty input, no parseable pairs, an unterminated quote — drops the whole line. Real-world logfmt is messy; the parser tries to be the more polite party. Plain is what it sounds like. The entire line becomes LogEntry::message. No timestamp parsing, no level extraction, no heuristics — because every plaintext format encodes those differently and a wrong guess silently corrupts your last 2h queries. For formats without their own timestamps, a new --timestamp-now flag stamps each entry with the current ingestion time: Opt-in. v0.1's skip-if-no-timestamp policy is still the default, because fabricating timestamps without explicit consent is the kind of foot-gun you discover at 3am. The implementation lives in crates/core/src/follow.rs, gated behind #[cfg(unix)]. It's built on the notify crate for inotify/FSEvents/kqueue plus a FileTailer struct that tracks (dev, ino, offset) and reacts to two conditions: Both checks run on every read. The tailer starts at EOF (matches tail -f), buffers partial lines until the newline arrives, and strips \r\n for the Windows-line-ending crowd. Fifteen unit tests in follow.rs cover the boring edges: 20 KiB single line spanning three read buffers, unicode, blank lines, mid-rotation gap where the file briefly doesn't exist. The CLI loop wires it together with ctrlc for clean SIGINT/SIGTERM handling and std::sync::mpsc for the watcher → ingest channel. The CLI stays fully synchronous — no tokio dependency, because watching a file and reading lines does not need an async runtime, and adding one for the sake of fashion was tempting and I resisted. One real limitation: Windows. The (dev, ino) trick uses std::os::unix::fs::MetadataExt. The module is gated; on Windows the flag is rejected with an error. Cross-platform rotation detection is a clean v0.3 contribution. Two mutually-exclusive flags, one of which is required (clap's ArgGroup enforces this). --older-than accepts a count plus a unit (m/h/d); --before accepts RFC 3339, naive UTC datetime, or a bare date. By default prune counts the doomed rows, shows you the number, and asks for confirmation. --yes skips for scripts. Under the hood it's DELETE FROM log_entries WHERE timestamp < ?1 followed by VACUUM. The VACUUM is a separate statement because SQLite refuses to run it inside an explicit transaction — a thing I learned by writing the obvious code first, hitting the error, and reading the SQLite docs second. Comparison is strict <, so a row whose timestamp exactly equals the cutoff is kept. Surprising? Yes, slightly. Documented? Now it is. Also in M4: the API used to honour LOGDIVE_DB as an environment-variable fallback for --db. The CLI did not. v0.2 fixes the asymmetry — both binaries now respect it, with the command-line flag taking precedence when both are set. The HTTP API got a third endpoint: Three fields, all compile-time constants. version is env!("CARGO_PKG_VERSION"). formats is LogFormat::ALL.iter().map(|f| f.name()).collect() — adding a new format to core automatically propagates here, no manual maintenance. capabilities is hard-coded today, sorted alphabetically with a test pinning that ordering so future additions can't accidentally break a client that did assert_eq!. The endpoint never touches the database. It's reachable before the first ingest, after a prune --yes, during a fresh Docker volume's first millisecond — which is exactly what makes it a good HEALTHCHECK target (see M6). CORS is now configurable via --cors-origins or LOGDIVE_API_CORS_ORIGINS. Disabled by default (same-origin only). Pass * to allow any origin, or a comma-separated list for specific ones. Mixing * with explicit origins is rejected at startup because it's meaningless and almost certainly a typo: The implementation is tower-http's CorsLayer, restricted to GET only because the API is and stays read-only. The router wiring is the only place the methods list lives, so adding write endpoints would force a deliberate code change rather than a one-line config fix. This is the kind of friction you want. One trap I hit during M5 worth flagging: tower-http = "0.6" checks axum version compatibility at the trait-bound level. My first cargo add tower-http --features cors got axum 0.7.5 from cache where 0.7.6+ is needed. The error said Service trait not satisfied. cargo update -p axum fixed it but the error message gave me zero hints about the root cause. Multi-stage build with cargo-chef so dependency compilation is cached across source-only changes. debian:bookworm-slim runtime. Non-root user (logdive, UID 1000). /data volume for index persistence. EXPOSE 4000. HEALTHCHECK curls /version every 30 seconds — no DB access, no false negatives during a long query. Both binaries ship in one image. Default ENTRYPOINT is logdive-api; the CLI is reachable through --entrypoint logdive: Two image-level environment defaults that don't exist on host installs: The second one matters: on a host install the API binds 127.0.0.1 for security. In a container, that would make the published port unreachable. The override is the right default for the container. It is also why the README is explicit that exposing port 4000 with no reverse proxy puts a read-only API on the public internet. Read-only is defence in depth, not access control. GitHub Actions builds for linux/amd64 and linux/arm64 via docker buildx + QEMU, pushes to GHCR with the workflow's built-in GITHUB_TOKEN (no PAT to rotate). Builds run on every push to main, every push to a release/v* branch, every v* tag, and on PRs without pushing. One CI gotcha I'll save you the debug session for: cache-to: type=gha, mode=max reliably 502s on cache export for this workspace. The cache backend disagrees with how large the intermediate-layer export is. mode=min works. PR builds skip cache write entirely. The current .github/workflows/docker.yml has the working config. While I was in there, I also fixed a v0.1 UX paper-cut: logdive-api used to refuse to start if the database file was missing. That made sense for --db /home/me/typo.db; it was hostile for docker run -v fresh-volume:/data. v0.2 auto-creates an empty index with the right schema on first run, prints a one-line note to stderr explaining what just happened, and otherwise lets the server come up. Genuinely bad paths (non-existent parent, permission denied) still fail fast. AND, OR, CONTAINS, last, since, true, and false are case-insensitive. Known fields (timestamp, level, message, tag) hit indexed columns. Everything else routes through json_extract(fields, '$.<key>') — slower than a real index, but works against arbitrary JSON shapes without forcing a schema. SQL output always parenthesises each AND-group so the WHERE clause is unambiguous regardless of how many disjuncts you stack. A few v0.1 tradeoffs are resolved. New ones replaced them. Multi-format ingestion adds a small dispatch cost. Every line goes through a LogFormat match arm before hitting its parser. ~1–2% overhead on the benchmark suite, which I'd characterise as "not a thing" but felt obliged to mention. Follow mode is Unix-only. The rotation check uses (dev, ino) from std::os::unix::fs::MetadataExt. Windows compiles fine but rejects --follow at runtime. A notify-based fallback for the rotation half is a real contribution opportunity. The Docker image is 97 MB. debian:bookworm-slim + two binaries + curl + ca-certificates. A musl-static build against a distroless runtime would cut this to ~10 MB but adds linker complexity with bundled rusqlite and blake3. Deferred. LOGDIVE_API_HOST=0.0.0.0 in the container is correct but dangerous. The README repeats the warning twice; the API stays read-only specifically so a misconfigured deployment leaks data, not the world. OR without parens covers ~95%. (level=error OR level=warn) AND service=payments requires duplicating the service clause as level=error AND service=payments OR level=warn AND service=payments. Parenthesised expressions are the v0.3 headline. Fresh cargo bench numbers from v0.2.0. Ingest and query paths weren't touched in v0.2, so these track v0.1 to within run-to-run variance: Reproducible from crates/core/benches/. cargo bench runs the suite; criterion writes HTML reports to target/criterion. The 25%-match number is the most variance-sensitive (it walks the result set rather than short-circuiting on a count); your hardware will differ. Release binary sizes: logdive 3.9 MB, logdive-api 4.2 MB. The "<10 MB" budget from v0.1 still holds, with room. No CLI flags were removed or renamed. Every v0.1 command still works. Library users (logdive-core directly): two breaking changes. Both are mechanical migrations. v0.1's contribution list had seven items. Four shipped in v0.2 (OR, non-JSON formats, follow mode, Docker image). Three remain, plus the work v0.2 exposed: The repo has CI, integration tests against tempfile-backed SQLite, conventional-commit history, and a release process that's a documented sequence of one-line commands. Bug reports and PRs at github.com/Aryagorjipour/logdive. Five things v0.2 taught me that the v0.1 release didn't: Each of these took a CI run or a stack trace to learn. None of them are in any "publishing a Rust workspace" tutorial I can find. They're in this article now, which is the only reason I'm slightly less annoyed about having learned them. Arya Gorjipour — backend engineer, logdive maintainer. If you ship a v0.3 contribution from the list above, I want to hear about it. If you use logdive to debug a real incident, I want to hear about that too. The most interesting bug reports get a hat tip in the next release post. Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse