KitharaThe layer beneath your product
← All work
Case 01 · Seed-stage AI data platform · Oct 2025 — present

The platform layer of an agent-native data platform

The client builds an agent-native data platform for science and private data. Kithara was brought in at seed stage to turn a contractor-built proof of concept into a platform a team could build a company on, and has led its technical direction since.

Role
Platform lead · technical direction
Stack
GoPythonTypeScriptAWSPulumiECS FargateSQSEventBridgeRedisPostgresMCPGitHub
  • ~10sFirst scale-up in staging load tests, down from 360s measured
  • 8 moProof of concept to production, June 2026
  • 64Repositories on one security and release standard
  • <6 wkEmpty repo to public CLI beta
The short version
Where it started

A contractor-built monolith that predated the engineering team, with no developer platform, infrastructure standards or service boundaries. One long-lived executor ran all of the product’s compute.

What we own

The platform layer and its technical direction: developer platform, shared infrastructure, service decomposition, the event-driven compute and control planes, and the security, release and operability standards the team builds against.

Where it is now

In production since June 2026, with a measured scaling model to 25k users and a public CLI in beta. A team of eight engineers builds and runs it.

The architecture

Three planes on one event fabric

One rule shapes the design. Large bytes move between the browser, object storage and compute workers without passing through chat servers or long-lived executors, so no single machine’s CPU, RAM, disk or network becomes the platform’s ceiling.

Control plane
product-api
User-facing API. Publishes root run + cancel.
event fabric
Shared EventBridge + SQS. Work routing, cancel delivery.
run-ledger
Durable run ledger. Consume-only read model.
control-plane
Reconciler framework. Policy plugins: tiering, timeouts.
↓ work routing↑ lifecycle · progress · terminal
Compute plane
compute-runtime
Planners · shard workers · reducers · finalizers. Executes work and emits events. Holds no authoritative state.
↕ manifests · inputs · outputs
Data plane
artifacts-api
Manifests, output promotion, analysis-ready metadata.
object storage
Large-byte inputs, partials, durable worker-readable data.
Not shown: the web client above product-api, and control-plane reading the run ledger over HTTP for policy triggers, the one deliberate edge off the bus. Large uploads go browser-direct to object storage.

Designed for a 25k-user, 250 GB scaling goal after aggressive load testing showed what real-time chat workloads cost. compute-runtime owns every lever that changes how much capacity exists. control-plane owns only where a run lands.

In depth

Compute that has to answer while someone is waiting

Two controls put a machine under a user’s question fast enough that it still feels answered: an autoscaler that sets capacity, and a routing policy that picks a tier when one is backed up. We built both, and kept them separate on purpose.

360s
Measured reaction, managed autoscaling
~10s
Observed first scale-up, replacement
5s
Autoscaler decision cadence
0.16s
Warm per-job overhead
12ms
Worker boot inside the container
360s is measured. ~10s is the first scale-up seen in a staging load test, asserted at ≤15s against a ≤30s milestone target. 5s is the decision cadence only.
How it was done
The constraint

A user asks a question in chat. Code the agent generated seconds ago must run on a worker that does not exist yet. Every second of scheduling is latency the user sees.

The platform scaled on AWS Application Auto Scaling target tracking for ECS, driven by a CloudWatch alarm on queue depth per running task. A controlled in-VPC campaign measured its reaction to a burst at 360 seconds. The fleet design had been justified against an estimate of 30 to 90 seconds, four to twelve times optimistic.

Most of that time went to deciding to scale. A container starts in 38 to 50 seconds and its worker boots in about twelve milliseconds. Target tracking waits for several sustained one-minute CloudWatch datapoints, and a minute is the finest resolution the alarm has, so no tuning fixes it.

The mechanism
01

Segment the work so one shape cannot starve another

Four worker pools, each with its own queue, dead-letter queue, routing rule, service and task size, sized from measured job duration: 3.7 seconds on average for platform tools inside a chat turn, 5.7 seconds for generated compute, 108.9 seconds for heavy analysis. A burst of long analyses can no longer block interactive requests.

The fourth pool is a security boundary. Credentialed shell execution has its own task role, queue and runtime-capability set, and the ordinary and credentialed pools each refuse the other’s work.

02

Size the warm floor from measured traffic

An interactive path cannot wait out a cold start. The warm floor comes from measured per-tier duration and traffic share, run through Little’s law and projected from 5,000 to 25,000 users.

The interactive tool tier needs about one warm worker even at 25,000 users, because 3.7-second jobs barely overlap. Generated compute drives the floor, from two workers to eight. Heavy analysis keeps a floor of zero and a wide burst ceiling, since a job that runs for minutes can wait for a worker.

03

Designed and built a replacement autoscaler

We replaced the managed policy outright with a 256 MB Go Lambda that reads each tier’s queue depth and running count, computes the desired count, and writes it to the service.

The control law: divide depth by jobs per worker, clamp to the tier’s floor and ceiling, then clamp to within five of the running count. Depth includes in-flight messages, or the fleet would drop to its floor as soon as workers picked up work. Running workers are capacity and never count as demand.

An earlier version added running count to depth and used it as the lower clamp. During a rolling deploy, with old and new tasks overlapping, the desired count ratcheted up one task per tick against an empty queue until 432 tasks had exhausted the subnet’s free addresses. A test now asserts that a fleet above its ceiling drains.

The five-task clamp is the only damping, so one number bounds how fast a burst is answered and how fast an idle fleet drains. The scheduler’s minimum interval is a minute, so each invocation loops eleven times at five-second spacing. Every tick publishes the shortfall between demanded and permitted capacity, so a tier pinned at its ceiling shows how far demand overshot.

04

Designed the traffic shift that covers what scaling cannot

Even a five-second decision cannot produce a worker in under forty seconds. To cover that gap, the routing service polls the same queues every twenty seconds, rates each tier’s saturation by backlog per running worker, and sends new runs to a tier with room.

A run moves up at most one tier, never down, because a downgrade can run a job out of memory. A misordered tier table fails config parsing, since it would turn an upgrade into a downgrade. When several tiers stall together, runs stay put, because rerouting would only feed the hottest backlog. Nothing is rejected: an oversized request goes to the largest available tier and raises a flag.

We did the arithmetic first. The autoscaler reacts to about four outstanding messages within five seconds. The router would not shift traffic until about eighteen, after a twenty-second poll, so it engages only during a cold start or when a tier is pinned at its ceiling. That makes it a bridge, and keeps two controllers from fighting over one signal.

05

Make scale-in safe enough that scaling hard is safe

Load testing showed scale-in could destroy work. Across two staging runs it killed fourteen in-flight jobs, each seventy to ninety seconds into a ninety-second job.

A task now holds reference-counted scale-in protection while a job runs, expiring just above the tier’s job ceiling so a crashed worker is still reclaimed quickly. Because a serial worker drops protection between jobs, shutdown also lets the running job finish and acknowledge its own message.

Protected tasks will not stop for a deployment either, so rolling deploys let a tier run at twice its desired count. At the default setting, deploys would have deadlocked.

control-plane cannot start a worker. When it reroutes, it buys the forty seconds compute-runtime needs to bring one up.

Key decisions

Eleven domains, run concurrently

One line each. Open any card for the engineering detail.

Developer platform (DevEx)8 golden templates

Built the developer platform from zero

A Go CLI scaffolds a production-ready service from eight templates, on a shared IaC library used across a 64-repo org.

Engineering detail

Wrote a Go CLI that scaffolds a production-ready service (repo, GitHub environments, infrastructure stacks, CI/CD, OCI image pipeline) from eight golden templates, on a shared IaC library consumed as versioned releases across a 64-repo org. Engineers add security scanning, release cutting or telemetry with one Makefile target, and new services get deployed without days of hand-wiring.

Decomposition8 phases

Drove the monolith toward a service platform

Extracted the web app in eight phases, then split compute out of the API monolith into two new services.

Engineering detail

Led the eight-phase extraction of the web app from the monorepo, replacing monorepo coupling with a build-time OpenAPI spec registry in a dedicated platform account, with no public exposure, so the exact staging image promotes to production unchanged. Then split compute execution out of the API monolith into two new services and set up the platform’s C4, standards, ADR and break-glass docs.

Compute planeuntrusted code

Designed and shipped the event-driven execution plane

Events feed per-tier queues and segmented Fargate pools, with workers built to run untrusted code.

Engineering detail

compute-runtime is worker-only: events → per-tier queues → segmented Fargate pools. Workers are built for untrusted code, with panic recovery, constant-memory output streaming, process-group kill on timeout, a non-root child uid, an unconditional environment scrub, ambient-credential probe tests, and per-job tokens for script-local storage.

Measured, then built360s measured

Measured the thing everyone had estimated

Cold start had been estimated at 30 to 90 seconds. Measured, it was 360. The same pass revived a tier carrying 0% of traffic.

Engineering detail

The cold-start figure had been asserted as “30 to 90 seconds plausible” and used as an architectural input. Measured, it was 360 seconds.

The same pass found a dead tier. Its rule required a timeout under 30 seconds, but none was configured below 120, so it carried 0% of traffic while looking healthy. Redefined on code provenance, it now carries about 14% of runs in staging and production.

Both are the same move: measure the quantity a design depends on before the design depends on it.

Control planepoll → apply → publish

Built a reconciler framework the routing policy lives in

A tick-loop control plane hosts the routing policy, checks it against real capacity, and retires stranded runs.

Engineering detail

control-plane runs poll durable state → apply policy → publish event on a tick loop and is named for that shape, so it can host policies beyond scheduling. The compute classifier is its first tenant.

A drift guard checks that every published policy ceiling fits the physical capacity compute-runtime reports, and flags drift without adapting to it. It fails open on unpublished fields, since a missing number and a real zero look the same on the wire. A run supervisor declares stranded runs dead. Nothing else could, because the ledger is passive and the chat tier times out only its own rows.

Closing the loopcapability APIs

Told the model what it was actually running on

Both services publish their real capabilities, so the agent sizes each job to actual tier limits.

Engineering detail

With no authoritative description of its runtime, the agent guessed: unloadable dependencies, output-contract violations, and a tool prompt hardcoding “estimate runtime on a 16 GB machine”, wrong for two of the three tiers.

Both services now publish capability APIs: compute-runtime for physical and image facts, control-plane for policy. The API tier polls both into Redis with no TTL (the cache is the last-known fallback) and renders them into the compute tool’s model-facing description, so the agent sizes jobs to real tier ceilings and they land on the right tier. Routing posture and denylists stay out, and a stable order keeps the provider’s prompt cache hitting.

Security~1,900 lines of IAM

Hardened the security posture org-wide

SSO, federated CI, least-privilege IAM as code, and builds that fail when a policy drifts, across 64 repositories.

Engineering detail

We own security and access across a multi-account cloud organization (sandbox, dev, staging, production, platform): identity-center SSO, cross-account deployment roles, CI OIDC federation, client VPN with SAML and certificate auth, and the public/private network boundary of VPC endpoints, PrivateLink and internal load balancers.

Least-privilege IAM is ~1,900 lines of policy as code. A resource registry fails CI when a policy lacks an action a stack’s resources require, so permission drift surfaces at build time. Across 64 repositories: rulesets, required checks, break-glass bypass, enforced job timeouts, scoped App tokens in place of long-lived personal tokens, and platform-native dependency and code scanning.

The connections service has write-time SSRF guards and authenticated probing, so a passing connection test proves the credentials work.

Release engineering6+ repos

Automated release and change management across the fleet

Generated changelogs, a CalVer release line, and a self-hosted CI runner fleet with its own health checks.

Engineering detail

Commit-generated changelogs in six-plus repos, pushed by a scoped App composite. A CalVer release line for the platform API. A self-hosted CI runner fleet with an ephemeral ops pool, job-aware drain, and a health check across the cloud, CI and instance planes. Telemetry on the build tooling’s own latency and failures.

Local development11 services

Owned the local-development and testing platform

A developer sidecar runs eleven services locally, with health, logs, search and synthetic tests.

Engineering detail

A developer sidecar runs eleven services across switchable workspaces, with a dependency-graph UI, live health, log streaming and search, and synthetic browser and k6 suites. Secrets come from a password manager when needed and never sit at rest. Every service’s local stack builds on a shared local cloud-emulation and Postgres fabric.

Public surface<6 weeks

Shipped the public CLI

One Go binary that is both a CLI and an MCP server, from empty repo to public beta in under six weeks.

Engineering detail

Empty repo to public beta in under six weeks: one Go binary that is both a CLI and a stdio MCP server, so any coding agent can drive platform workspaces. Runs on macOS, Linux and Windows (native and WSL), with a checksum-verifying installer.

Cross-service delivery3 services

Delivered bring-your-own-MCP across three services

Customers connect their own MCP servers. Designed the contract and delivered it across three services to GA.

Engineering detail

Customers connect their own MCP servers. The connections service got the source type, SSRF guard and credential brokering. The API tier got the auth bridge, fan-out registration and tool auto-attach. The web app got the connect wizard and management views. Designed the contract, wrote each repo’s handoff doc, and removed the feature gate for GA.

Systems owned

Primary or sole author

Counts as of 2026-08-20.

Show all 6 repositories
scaffolding CLI
Developer platform CLI · Go
95 commits · 45 PRs
sole primary author

Owns the lifecycle of every service in the org. init scaffolds a repo from golden templates through to first push. init-infra retrofits repos that predate the automation. pkg and release build and push OCI images. common sync keeps consumers on a pinned shared-IaC release.

shared-iac
Shared infrastructure & CI/CD · Pulumi, Bash, Actions
194 commits · 114 PRs
~70% of repo history

The org’s shared infrastructure: 8 stacks, 13 reusable workflows, 30 composite actions and 80 shared scripts across 15 domains, plus runbooks, incident forensics and managed agent skills. All versioned and pinned, so consumers upgrade deliberately.

  • Golden templates regenerated from real repos and drift-checked in CI.
  • Intent-scoped migrations, with plan and preview-diff modes, replace three-way-merge template re-application, which diverged repos cannot take safely.
  • Runner fleet with a canary tier and per-job workspace hygiene. Found and fixed a disk-prune race that deleted images mid-job.
  • Spec registry: one artifact bucket behind an internal load balancer and PrivateLink, replacing a per-environment CDN design.
compute-runtime
Event-driven code execution · Go
133 commits · 73 PRs
primary author

Runs untrusted Python for the analyst agent. Milestone five removed the public HTTP API and MCP submission shim, so jobs enter only through the event fabric.

  • Tiers: small (in-chat platform tools, trusted provenance, under 300s), standard (generated compute, catch-all), large (heavy analysis, beyond 1800s) and credentialed CLI. The catch-all rule derives from the tier list, so a new tier cannot double-dispatch.
  • Raising the warm floor from 3 to 8 eliminated cold start for a typical 20-job burst.
  • SQS visibility heartbeats bounded by a maximum in-flight duration. Per-job HMAC tokens for script-local storage, and security probes runnable against any environment.
  • An investigation into orphaned running jobs first blamed this service. The correction proved the ledger row is owned upstream, where nothing timed it out.
control-plane
Reconciliation framework · Go
29 commits · 23 PRs
sole author

Applies each policy as an event the owning service consumes, and dispatches downstream work onto the event bus.

  • Two-lane classification: requests the agent marks as intentional go to the smallest enabled tier whose admission ceiling fits, and the rest use the timeout-and-provenance lane. Activation is traffic-driven, so there was no flag day.
  • Saturation posture (open / throttled / upgrading) comes from the same function the classifier reroutes with, so the two cannot disagree. Staleness is judged per tier, so one failed poll does not flap the busiest tier to open.
  • Run-supervisor recovery sweeps, and a per-tick heartbeat that never reads green over a partial failure.
dev sidecar
Developer sidecar & test platform · TypeScript
118 commits · co-lead
1,200+ tests

Covers what CI can’t: expensive tests that don’t belong in every PR pipeline. Its releases dashboard shows what is deployed in each environment, when, by whom and from which branch. Also root-caused a recurring editor lockup that had been costing days.

public CLI + distribution
Public CLI & MCP server · Go
57 commits · 16 PRs
empty repo → beta in 6 weeks

Serves MCP over stdio from the CLI binary, with no separately hosted service. Keyring credentials with an encrypted-file fallback and env-var override for headless use, refresh-token revocation on logout, and silent 401 recovery. Ships for Windows, macOS and Linux on both architectures.

Your own systems would get this level of detail, measured and written down for your team.

Start a conversation