Research compute, infrastructure and reliability at quant firms

Two workloads, one estate. That is the shape of infrastructure work at a systematic trading firm, and it is why the function does not map cleanly onto either a cloud SRE role or a traditional datacentre role.

On one side: research compute. Large numbers of batch jobs, submitted in bursts, from teams who all believe their work is urgent. On the other: the live trading path, where the interesting number is the worst case rather than the average, and where an unplanned restart is not a retry.

Firm-specific detail below is attributed to the firm that published it.

The research side: scheduling is a fairness problem

The hard part of research compute is rarely raw capacity. It is what happens when demand exceeds it - which, on a fixed on-premise pool, is most of the time.

G-Research published a description of exactly this problem and its answer, Armada. Its account describes migrating workloads to containers on Linux, and notes that in the batch model they were used to, you can submit far more jobs than the infrastructure can process at once, with the excess queued externally and prioritised by fair share. Kubernetes, the post says, was best of breed for container orchestration but lacked the ability to queue jobs or schedule them fairly when over-provisioned. Their stated goals included adding queuing and fair share without altering Kubernetes itself, supporting multiple clusters to scale past the limit of a single cluster, and using a pull-based model, with the aim of running a fleet of thousands of servers (G-Research, Armada, 10 December 2020).

Read that as one firm's design, published at a point in time, not as the industry's standard architecture. What generalises is the question it answers, and that question is worth being able to discuss in an interview: when the queue is longer than the cluster, who runs next, and who decides?

Comparison: three ways firms answer it

A useful interview question is not which of these a firm chose but what happened the last time the choice was wrong.

The live side: determinism, not throughput

The trading path inverts the objective. Average performance is close to irrelevant; the number that matters is the worst case you can demonstrate, and the property you are protecting is that the system behaves the same way under load as it does in a quiet market.

Practically, that changes the reliability vocabulary:

• Capacity planning is about headroom at the peak second, not the peak hour.
• A restart is a business event with a cost, not a remediation step.
• Monitoring has to distinguish "slow" from "stopped" quickly, because on this path the two require different responses.
• Change management is stricter, and deployment windows are dictated by market hours rather than by the team's calendar.

Observability as a product problem

The published examples from these firms share a pattern: alerting and monitoring are treated as engineering products, not configuration. Hudson River Trading's systems team published HeraclesQL, a Python DSL its systems developers wrote after thousands of hand-written alert queries became repetitive and hard to express, and describes a systems team that provisions and monitors the firm's servers and network equipment across both trading and research (HRT, HeraclesQL; HRT, Engineering and Interviewing at HRT).

Again: one firm, published by that firm. The transferable point is the expectation. If you arrive from an environment where monitoring meant maintaining dashboards, the gap to close is treating the alerting surface as software with users, tests and a design.

A worked scenario: the queue that starved a team

Illustrative, not an account of any real incident.

A research group reports that their jobs have been waiting for hours while another group's work runs. The estate is healthy: no node failures, utilisation near 100 per cent. Nothing is broken; the system is doing exactly what it was configured to do.

The useful diagnosis separates four things that all look identical from the researcher's seat:

1. Fair share is working and the complaining team has already had its share. The answer is a conversation about allocations, not an engineering change. 2. Fair share is being defeated by job shape. Thousands of tiny jobs from one team can occupy scheduling capacity out of proportion to the compute they consume. 3. Resource requests are wrong. Jobs requesting far more memory than they use will fragment the pool and starve everyone, including the requester. 4. A queue is not draining because execution is failing quietly - image pulls, mount failures, a full disk on a subset of nodes - and retries look like demand.

Only the first is a policy question. The other three are the platform team's, and you can tell them apart in minutes if the platform publishes per-queue wait time, requested-versus-used resources, and retry counts as first-class metrics. If it does not, the team will argue about fairness with no evidence, which is the actual failure.

Readiness checks

• Explain fair-share scheduling to someone who has only used priority queues, and say where it breaks down.
• Describe an alert you designed that was actionable, and one you deleted because it was not.
• Walk through an incident you owned end to end, including the change you made afterwards.
• Explain what you would measure to tell contention apart from failure in a batch estate.
• For a low-latency seat: describe how you would demonstrate a worst case rather than an average, and what you would rule out first.

Questions to ask the team

• Is this seat on the research estate, the trading path, or both? Who decides when they conflict?
• How is compute allocated between teams, and who owns the allocation policy?
• What is in scope for on-call, and what was the last thing that paged out of hours?
• How much of this team's output is tooling it builds versus systems it operates?
• What is the relationship between this team and the engineers who write the jobs it runs?

What this guide does not claim

Armada and HeraclesQL are two published projects from two firms. They are not evidence about how most firms schedule work or alert on it, and this guide makes no claim about hours, team sizes or pay premiums for infrastructure work. Security engineering overlaps with this function but is a separate career with its own path.