Hemant Kumar

Hemant Kumar

Applied AI engineer at BNY · India

I build AI systems end to end: inference, agents, retrieval, evals, and the backend that keeps them running. I've been an applied AI engineer at BNY since July 2024.

I spend my own attention on the spec and the system design and let agents do the grinding. Then I prototype, ship, test, and improve, as fast as the work allows.

On the side I train small models, take big ones apart, and publish the benchmarks, including the ones that didn't go my way.

Now · October 2026

Building
FractalMem, memory for coding agents that run for hours.
Reading
Looped transformers, recursive self-improvement, and the economics of inference.
Training
Lifting.

Built and measured

openJev Verdict

2026 · 405 stars

A 151M model that makes typed decisions in about 20 ms, and beat models up to 230 times its size on JevBench.

Most automation asks a chat model for JSON, then parses it and retries when it breaks. Verdict skips generation. One forward pass over ModernBERT returns a typed choice, score, or yes/no with a calibrated confidence, so plain code can decide when to act and when to hand off to a person. Version 1.4 fixed three inference bugs without retraining and moved the JevBench score from 66.2 to 74.9. On the 20 September leaderboard that put it first among open models, ahead of models with 9B to 35B parameters, GPT-5.6 Luna, and Gemini 3.1 Flash-Lite. Verdict 2.0, a separate fine-tune, beats TypeSafe's Jev and the 421M Laya on typed-decisions.

JevBench scores, 20 September 2026 leaderboard Horizontal bars of JevBench scores on an axis from 50 to 80. Jev 1.13.0 (TypeSafe AI, closed) scores 75.4. openJev Verdict v1.4* (151M, open) scores 74.9. Semlf (Qwen3.5-4B) scores 74.7. djev (diffusion-gemma) scores 74.3. Laya (ModernBERT-large, 421M) scores 70.1. OpenJev (DiffusionGemma 26B-A4B) scores 67.7. openjev-sglang (Qwen3.6-35B-A3B) scores 66.3. GPT-5.6 Luna (OpenAI, low reasoning) scores 66.2. Bespoke Nimble 9B (Bespoke Labs) scores 63.7. Gemini 3.1 Flash-Lite (Google) scores 60.9. DeepSeek V4.1 Flash (thinking default) scores 57.8. openJev Verdict v1.4, a 151M model, is highlighted. Jev 1.13.0 TypeSafe AI, closed 75.4 openJev Verdict v1.4* 151M, open 74.9 Semlf Qwen3.5-4B 74.7 djev diffusion-gemma 74.3 Laya ModernBERT-large, 421M 70.1 OpenJev DiffusionGemma 26B-A4B 67.7 openjev-sglang Qwen3.6-35B-A3B 66.3 GPT-5.6 Luna OpenAI, low reasoning 66.2 Bespoke Nimble 9B Bespoke Labs 63.7 Gemini 3.1 Flash-Lite Google 60.9 DeepSeek V4.1 Flash thinking default 57.8 50 60 70 80
JevBench scores, 20 September 2026 leaderboard Horizontal bars of JevBench scores on an axis from 50 to 80. Jev 1.13.0 (TypeSafe AI, closed) scores 75.4. openJev Verdict v1.4* (151M, open) scores 74.9. Semlf (Qwen3.5-4B) scores 74.7. djev (diffusion-gemma) scores 74.3. Laya (ModernBERT-large, 421M) scores 70.1. OpenJev (DiffusionGemma 26B-A4B) scores 67.7. openjev-sglang (Qwen3.6-35B-A3B) scores 66.3. GPT-5.6 Luna (OpenAI, low reasoning) scores 66.2. Bespoke Nimble 9B (Bespoke Labs) scores 63.7. Gemini 3.1 Flash-Lite (Google) scores 60.9. DeepSeek V4.1 Flash (thinking default) scores 57.8. openJev Verdict v1.4, a 151M model, is highlighted. Jev 1.13.0TypeSafe AI, closed 75.4 openJev Verdict v1.4*151M, open 74.9 SemlfQwen3.5-4B 74.7 djevdiffusion-gemma 74.3 LayaModernBERT-large, 421M 70.1 OpenJevDiffusionGemma 26B-A4B 67.7 openjev-sglangQwen3.6-35B-A3B 66.3 GPT-5.6 LunaOpenAI, low reasoning 66.2 Bespoke Nimble 9BBespoke Labs 63.7 Gemini 3.1 Flash-LiteGoogle 60.9 DeepSeek V4.1 Flashthinking default 57.8 50 60 70 80
JevBench score from the 20 September 2026 leaderboard: intelligence, calibration, speed, and cost, 25% each. *Verdict v1.4 was scored on the 231 public tasks; the other rows on all 534.
74.9
JevBench score, #1 open model
77.10%
accuracy on 2,000 held-out decisions (Laya 76.60%, Jev 72.70%)
1.44%
calibration error (ECE) on the confidence head
8.8 h
to fine-tune, on a consumer laptop GPU

expert-stream

2026 · C++ / llama.cpp

A 284B model generating text on a 16 GB laptop. Its published memory floor is 96 GB.

DeepSeek V4-Flash is 82.5 GB on disk, but each token only uses 6 of 256 experts per layer. So the engine leaves the experts on the NVMe SSD and streams in the ones each token picks. Stock llama.cpp died after two tokens on this machine; capping the working set lets it run to completion, and the greedy output is byte-identical. It's slow, under one token a second, and the repo shows where every second goes. Before writing any code, I tested about twenty ideas borrowed from biology and physics, from immune memory to the way DNA packs a body into a small genome. Most turned out to be a cache under a new name. The compression idea lasted an hour: real expert weights measured barely more compressible than random noise.

An 82.5 GB model streamed from SSD into 16 GB of RAM Three bars on one 0 to 100 GB scale: published memory floor of 96 GB, model on disk of 82.5 GB shown as a dashed outline, and this laptop's RAM of 16 GB, highlighted. Below, a flow: the SSD holds 256 experts across 43 layers, each token uses the top 6 per layer, and 1.649 GiB is streamed into RAM. Published memory floor 96 GB Model on disk (NVMe SSD) 82.5 GB This laptop's RAM 16 GB 0 50 100 GB SSD: 256 experts× 43 layers Per token:top 6 per layer RAM: 1.649 GiBstreamed in
An 82.5 GB model streamed from SSD into 16 GB of RAM Three bars on one 0 to 100 GB scale: published memory floor of 96 GB, model on disk of 82.5 GB shown as a dashed outline, and this laptop's RAM of 16 GB, highlighted. Below, a flow: the SSD holds 256 experts across 43 layers, each token uses the top 6 per layer, and 1.649 GiB is streamed into RAM. Published memory floor 96 GB Model on disk (NVMe SSD) 82.5 GB This laptop's RAM 16 GB 0 50 100 GB SSD: 256 experts× 43 layers Per token:top 6 per layer RAM: 1.649 GiBstreamed in
Every token reads 1.649 GiB of expert weights from disk to produce about 700 KB of activations.
82.5 GB
model, on a laptop with 16 GB of RAM
identical
greedy output to stock llama.cpp
124 → 95 s
same prompt, after swapping in the streaming reader
0.71 tok/s
I/O-only ceiling at 0% cache hits

FractalMem

2026 · Python

Memory for long-running coding agents. Compact the context, keep the evidence.

When an agent compacts its context, the summary drops the details it needs an hour later, like the exact name of a function. FractalMem stores redacted events in SQLite, builds a hierarchical summary that only appends, so the prompt prefix stays stable for caching, and lets the agent search exact symbols and text over MCP.

Fractalmem mechanism Three agent sessions and an agent hook feed the fractald daemon, which owns all writes. The daemon appends to a summary tree and to a SQLite database with FTS5 full-text search that stores redacted events. The MCP tool search_symbols_and_text reads from that database. agent session 1 agent session 2 agent hook fractald daemon owns all writes summary tree (append-only) SQLite + FTS5 redacted events MCP: search_symbols_and_text
Fractalmem mechanism Three agent sessions and an agent hook feed the fractald daemon, which owns all writes. The daemon appends to a summary tree and to a SQLite database with FTS5 full-text search that stores redacted events. The MCP tool search_symbols_and_text reads from that database. agent session 1 agent session 2 agent hook fractald daemon owns all writes summary tree (append-only) SQLite + FTS5 redacted events MCP: search_symbols_and_text
Hooks send events to one daemon. The daemon owns writes, builds the summary tree, and serves search over MCP.
97.83%
session recall@10 on LongMemEval (cleaned oracle, 470 questions)
1,000/1,000
exact-symbol top matches, p99 query 0.03 ms
99.47%
of 30,000 turns kept the byte prefix, zero violations
0
API keys needed; it runs offline

nanoserve

2026 · Python / MLX

A small inference engine for Apple Silicon with continuous batching, prefix caching, and OpenAI-compatible streaming.

Running one prompt locally is easy. Serving overlapping requests is where the real problems are: who owns the KV cache, when to reuse a prefix, and how to schedule. nanoserve is my answer in MLX, measured on an M4 with Qwen2.5-0.5B.

Time to first token, cold prefix and warm prefix Two horizontal bars share a 0 to 250 ms axis. The cold prefix takes 240.50 ms to first token. The warm prefix with cache takes 77.76 ms, which is 67.7% faster. Cold prefix 240.50 ms Warm prefix (cached) 77.76 ms 67.7% faster 0 50 100 150 200 250 ms to first token
Time to first token, cold prefix and warm prefix Two horizontal bars share a 0 to 250 ms axis. The cold prefix takes 240.50 ms to first token. The warm prefix with cache takes 77.76 ms, which is 67.7% faster. Cold prefix 240.50 ms Warm prefix (cached) 77.76 ms 67.7% faster 0 50 100 150 200 250 ms to first token
Median time to first token with a 576-token prefix, cold versus warm cache, on an M4.
63.94 ms
median time to first token
67.7%
faster first token on a warm prefix
178.79 tok/s
median, vs 173.50 for mlx_lm.generate
80.76 ms
p95 first token at 8 concurrent requests

ThreadGraph

2026 · Go

Finds goroutine leaks and deadlocks from execution traces. It caught 64 of 68 real bugs.

Go concurrency bugs depend on scheduling, so tests pass and reviewers miss them. ThreadGraph reads a runtime trace, rebuilds which goroutine spawned and blocked which, and runs leak, lock-cycle, and channel analyses. It needs no instrumentation.

Detection of 68 GoBench GoKer blocking bugs: 64 detected, 4 missed A grid of 68 dots, 17 columns by 4 rows, one dot per blocking bug. The 64 filled dots are detected. The 4 outlined dots at the end of the last row are missed. detected (64) missed (4)
Detection of 68 GoBench GoKer blocking bugs: 64 detected, 4 missed A grid of 68 dots, 10 columns by 7 rows, one dot per blocking bug. The 64 filled dots are detected. The 4 outlined dots at the end of the last row are missed. detected (64) missed (4)
GoBench GoKer: 68 blocking bugs taken from production Go projects. Filled dots were detected.
64 / 68
GoBench GoKer bugs detected (94%)
0
false positives on the net/http/httptest control (224 goroutines)

scout-deterministic

2026 · Python

Reward-hacking scanners for agent transcripts that finish in 0.4 seconds. The LLM judge takes 560.

Coding agents can pass pytest without solving the task. I wrote four rule-based scanners for Inspect Scout and benchmarked them against its LLM judge on 26 live episodes. They are much faster and cost nothing, with lower precision, and I published the case none of them caught: an agent that made assert 1 + 1 == 3 pass.

Wall time on a log scale: rule-based scanners and the LLM judge Two bars on a log axis from 0.1 s to 1000 s. Rule-based scanners take 0.4 s. The LLM judge (llm_scanner) takes 560 s. Both were timed on 26 transcripts on the same machine. 0.4 s Rule-based scanners 560 s LLM judge (llm_scanner) 0.1 s 1 s 10 s 100 s 1000 s
Wall time on a log scale: rule-based scanners and the LLM judge Two bars on a log axis from 0.1 s to 1000 s. Rule-based scanners take 0.4 s. The LLM judge (llm_scanner) takes 560 s. Both were timed on 26 transcripts on the same machine. Rule-based scanners 0.4 s LLM judge (llm_scanner) 560 s 0.1 s 1 s 10 s 100 s 1000 s
Wall time to scan the same 26 transcripts. Bars on a log scale.
0.4 s
vs 560 s for the LLM judge
$0
token cost
100% / 54%
recall / precision (LLM judge: 79% / 85%)
0.79
F1 for the best single scanner

Also built

  • Agent BoardPlanner, executors, and a review gate. Nothing commits until the verifier passes.
  • PalimpA local context graph for agents with provenance, temporal recall, and contradiction tracking.
  • Hallucination SentinelA single-pass risk score for LLM output, calibrated into allow, warn, or block.
  • Still MiniA learned 8× KV-cache compactor lost to a simple baseline, 0.12 to 0.54. The writeup explains why.
  • DAG-MoE reproductionThe ICML 2026 routing mechanism on an M4 Air. No loss advantage at that scale, reported as is.
  • NexusCacheDistributed cache in Go with consistent hashing and etcd discovery. 23.4K ops/s at 713 µs p50.
  • TitanQueueA Redis task queue in Go. When a worker dies, its lease expires and the task comes back.
  • ContextRAGLocal hybrid retrieval: dense vectors plus BM25, rank fusion, and a reranker, served over MCP.
  • GitHub Trending ReportDaily GitHub Trending, sorted into categories and scored by repeat appearances.
  • UnslopAn agent skill that strips AI-writing tells from prose without touching the facts.

How I work

I study agents closely: how they plan, where they cut corners, and how they fail without telling you. Knowing that is what lets me hand them the repetitive work while I keep the parts that need judgment, like the spec, the architecture, and the review.

I think evals matter more than anything else in an AI system. If I can't measure whether a change helped, I don't trust it. That's why the projects above ship with their benchmarks, including the results that went against me.

The problems I want to spend the next few years on: memory for AI systems, architectures beyond the autoregressive transformer decoder, models whose structure borrows from the brain's topology, small calibrated models that make decisions, and better chip design. If you're working on one of these, reach out on X or LinkedIn.

Experience

BNY Applied AI Engineer

Jul 2024 to now

RAG assistants, multi-agent systems, MCP servers, eval harnesses, and backend services for financial platforms that move about $100B a day.

Stack

Languages
Python, TypeScript, Go, Java, SQL
AI
PyTorch, MLX, llama.cpp, Hugging Face, fine-tuning, RAG, evals, MCP
Backend
FastAPI, Spring Boot, Node.js, gRPC, Kafka, Redis, PostgreSQL, SQLite
Infra
Docker, Prometheus, GitHub Actions
Agents
Claude Code, Codex, Cursor

Reading

I read across a lot of fields, mostly to understand how AI fits together from the chip up to the market.

Silicon
How GPUs and accelerators are built, and why memory bandwidth often matters more than raw compute for LLM inference.
Infrastructure
Distributed systems and the infrastructure that serves AI at scale: schedulers, caches, queues, and what breaks when traffic grows.
Models
Transformer architectures and their variants, including looped transformers, which run the same layers several times per token. I read a few research papers most weeks.
Agents
Multi-agent orchestration, software factories where agents write and review most of the code, and recursive self-improvement: whether AI systems can improve the systems that build them.
Agent behavior
Research on how agents actually behave: how they write code, where they take shortcuts, and how they reward-hack their way past tests. scout-deterministic is my own work on that last one.
Markets
The economics of LLMs, data centers, and startups. I read VC reports to see where the money is going, and I follow the new AI labs forming outside the big ones.
Products
Consumer AI apps, and what separates the ones people keep using from the ones they try once.
Mind
Neuroscience and neuroplasticity, how the brain rewires as it learns, and some philosophy.

Recent reads

Papers I've reproduced or built on

About

I have more interests than hours, and my attention jumps between them. I'm fairly sure I'm neurodivergent, and AI changed what that means for me. With agents I can learn a field in a week, prototype in it the next, and find out fast whether an idea holds up. I think that's one of the biggest things AI does for people whose brains work like mine.

When I'm away from the keyboard, I'm traveling somewhere new with friends, out for the evening, or in the gym lifting. Otherwise I'm working with my agents.

I believe that if a problem doesn't break the laws of physics, it can be solved, however hard it is. That's the bar I use when deciding what to work on.