data engineer
I build data and ML systems, and then I keep them running.
- Currently
- Data engineer at Sepal AI, since December 2025.
Observability and DevOps. Evaluation pipelines for language models, on ClickHouse, Grafana, and Prometheus.
- Previously
- EncryptMyWork (2025) · Kenexai (2021 to 2023) · ZF Group (2020, intern)
Kafka, Flink, Spark, Iceberg, SageMaker. Streaming and ML platform work throughout.
- Degree
- MS, Computer Software Engineering Systems. Northeastern University, May 2025.
- On the side
- Coconut Labs, a two-person lab I run with a co-founder, Jay.
Rented GPUs, published benchmarks, open-source releases. It is not my employer.
production data engineering
4 yearsacross three companies
Kafka, Flink, Spark, Iceberg. Billions of events a day at Kenexai, at 99.9% availability.
shared gpu, starved tenant
26xless waiting for a first token
1,585 ms under plain first-come-first-served, 61.5 ms with the scheduler I wrote. One A100, Llama-3.1-8B, vLLM 0.19.1, 311 requests. the bench →
mistakes in my own results
7found, numbered, published
A file of corrections against my own benchmarks, C1 to C7, in the public repo. read them →
running now · on a schedule · public record
Things I run
These are services, not simulations. They run on a cron when I am
not looking at them, and each one publishes its own record, including the parts
that failed. The counters live on the service. Nothing on this page can go stale
because nothing on this page is a copy of them.
The Loop
nightly cron · github actions · public repo
A release pipeline for a model, run by a scheduled workflow with no human in
it. Each night it registers a candidate version, scores it against a quality
gate, sends a slice of traffic to it as a canary, then either promotes it or
rolls back to the version already serving. Every night commits its own
artifacts: the gate scores, the canary comparison, and a decision file naming
the action and the reason for it.
So far it has rolled back every candidate it has been handed. That is the gate
doing its job against a threshold set above what the serving model itself
scores, not a regression caught in the wild. The decision files say so, which is
the point of committing them.
the live view →
· source
· every run it has ever done
Pulse
continuous ingestion · public gap ledger
An ingestion pipeline that pulls live vehicle positions from the MBTA feed on
a roughly 66 second cycle and lands them. When a poll is missed, it does not
quietly heal. It writes the outage into a public ledger with a start time, a
duration, the number of cycles lost, and a cause in plain words, such as
upstream unreachable. Any pipeline can look healthy on a dashboard. This one has
to name its holes.
It has had real outages, which is the whole reason the ledger exists. Freshness,
volume, null-rate and disk gates run against it continuously. Their current
state, the row count, the measured cadence, and every gap it has ever recorded
live on the status page. None of it is copied onto this page, so none of it can
be stale on this page.
the live view →
· source
status.json
machine readable
The same state as JSON, for anyone who would rather curl it than read a page.
status.json →
· both services, one page
how i work
Four decisions, and what they cost
My rate limiter did nothing for three weeks
I had written a cap of 16 concurrent requests in front of the inference
server. It was never holding anything back. The code released a request's slot
the moment the response started streaming, not when it ended, so for streaming
traffic the cap was effectively infinite. A poller sampled the in-flight gauge
149 times while 32 requests were running, and read zero every time. All of my
benchmarks were streaming benchmarks.
I wrote it up, marked which published runs it invalidated, said which
conclusions still stood and why, and fixed the release path. It is entry C5 in
the corrections file, next to six others. I would rather someone find that file
than find the bug.
C5 · 149 samples, cap 16, 0 in flight · results/CORRECTIONS.md
Then I ran the experiment that killed the idea anyway
Once that cap actually worked, I had to find out whether it was worth having.
The bet was that holding traffic at the door would smooth the latency cliff
under overload. It did not. The capped arm and the uncapped arm landed within 4
percent of each other on slowest-1-percent latency, against a threshold I had
written down in advance saying the cap had to be four times better to be worth
keeping. At one concurrency level the cap made things worse.
Then I checked my own bench with Little's Law and found the run was too short
to have put the cap under real pressure. The result flattered my rigor and I had
not earned it, so I downgraded my own verdict from FAIL to AMBIGUOUS, rented an
H100, and ran it again at 16,000 requests per arm. Same answer, within 4
percent. The idea was wrong, and now it was wrong properly.
16,000 requests per arm · H100 SXM5 · about $1.30 of GPU time ·
PROGRESS.md ·
tuning_guide.md
A number does not travel without its regime
The 26x on this page is one specific thing: how much less a starved tenant
waits for its first token when it shares a GPU with a noisy one. It holds on one
A100 at saturation, running Llama-3.1-8B under vLLM 0.19.1, over 311 requests.
On an H100 at the same offered load the starvation never happens, so the number
does not transfer. An earlier run on vLLM 0.8.5 measured 523x, which looks
twenty times better and was real output from a real machine. It was a weaker
batcher handling cold-start backpressure badly, so 26x is what gets cited and
the reason 523x does not is published next to it.
1,585 ms → 61.5 ms · solo baseline 53.9 ms · what this does not show
The name got ahead of the code
I called a project kvwarden, and then the code did not do what the name said.
Version 0.1.x ships tenant-aware admission control. It does not manage the KV
cache. Rather than let someone install it and find that out, I put the objection
into the public FAQ as a heading, in the words a sceptical reader would use: the
name says KV warden but you're not touching the KV cache, what gives. The answer
underneath says exactly that, and the release that shipped it reframed the docs
to match the code instead of the other way round.
v0.1.5, doc reframe to what the code does ·
the FAQ entry ·
pypi
shipped · 2026
In the order it happened
- aug 2026
Put two services on a schedule and left them there
The Loop and Pulse, above. One is a nightly model release pipeline that has to
justify every promote or roll back in a committed file. The other is an
ingestion job that has to write down its own outages. Both cost nothing to run
and both are readable by a stranger.
what they are · the live view
- aug 2026
kvwarden 0.1.6 on PyPI
A maintenance release. Removing dependencies the package never imported took a
clean install from 144 MB to 33 MB and cleared four security advisories that CI
had been installing for months.
pypi · source
- aug 2026
Six guardrails, open-sourced
Each one is a single failure that type checks and green dashboards do not
catch: a schema-quiet data corruption, a feature lookup that silently returns
stale values, a query that stops pruning partitions, a training set that leaks
the future into itself. Small repos, one bug each, source public.
silent data regression · point-in-time correctness · the lifecycle map
- jun 2026
A real-time analytics pipeline, costed on AWS
Seven runnable pieces, each with its measured throughput and its measured
dollar cost on the instance it ran on. No slide deck, no estimates.
private repo, walkthrough on request
- apr 2026
The fairness result, and a rename
Established the tenant-fairness measurement on a rented A100. The same month,
a trademark audit found a prior user of the old project name, so I renamed it in
public and attached the reasoning instead of quietly rewriting history.
kvwarden.org · benchmarks
- feb 2026
Rented the first GPU
First commit of the inference middleware that became kvwarden, and the rule
the lab has run on since: a claim without its hardware, its sample count, and
its version is not a result.
github.com/coconut-labs
Older work sits on the GitHub profile:
an RDMA NVMe offload stack, a C++20 latency workspace, a latent diffusion model
trained from scratch on 10 H100 nodes, and an archived data platform kept public
because the review that stopped it is worth more than pretending it still runs.
writing
Lab records, cleaned up
A model in the room
Three jobs a model can do inside a creative tool, and the fence that keeps it useful.
Longer form: Systems That Don't Lie,
five chapters on tenant fairness, single-machine inference, nanosecond signals,
agents, and observability that cannot flatter itself.