Research data and ML platform engineering at quant firms

A research organisation is only as good as the data layer underneath it. That layer is a career, and it is not the same career as finding signals. This guide describes the work, the technical problem that defines it, how to evaluate a team, and how to prepare - for people arriving from data engineering, ML infrastructure or backend platform work.

Firm-specific detail below is attributed to the firm or project that published it. None of it describes how every firm works.

What the function owns

Four responsibilities recur. How they are split varies: one team may hold all four, while elsewhere ingestion and platform sit with different groups.

• Acquisition. Market data, reference data, vendor feeds and alternative datasets, each with its own corrections, gaps and redelivery behaviour.
• Storage and access. The library or service a researcher actually calls. Read shape matters more than write shape: the same dataset is read thousands of times in slightly different slices.
• Reproducibility. A result is only a result if the exact data version behind it can be recovered. This is where research data work stops resembling ordinary analytics engineering.
• Feature and model platform. Feature production, experiment tracking, training and serving infrastructure, and the scheduling that keeps a thousand experiments from starving each other.

The problem that defines the work: point-in-time correctness

A recurring class of failure in a research data platform is a version problem wearing another costume. A vendor restates yesterday's prices. A reference file is corrected on Thursday for a Monday effective date. A researcher reruns last quarter's notebook and gets a better answer than they did at the time - because the data improved underneath them, not because the idea did.

A storage layer that only holds "the current truth" cannot answer the question that matters, which is *what did we know, and when did we know it*. That is why versioned time-series stores exist in this corner of the industry rather than off-the-shelf warehousing alone.

A published example worth reading

Man Group publishes ArcticDB, which its documentation describes as a high-performance, serverless DataFrame database built for the Python data science ecosystem: Pandas in and Pandas out, reads and writes to S3 or LMDB, a C++ data-processing engine, schemaless append and update, and time travel - reading previous versions of the data and taking snapshots (Man Group, ArcticDB; man-group/ArcticDB on GitHub).

Two things are worth taking from it, and one thing is worth being careful about.

Worth taking: the API boundary is the researcher's own idiom. Nothing about the design asks a researcher to leave Python or learn a query dialect to read a slice of history. And versioning is a first-class property of the store rather than a convention layered on top of file names - which is exactly the difference between "we try to keep old data" and "a result is reproducible".

Worth being careful about: the licence. ArcticDB's repository states it is released under a Business Source License 1.1. The source is available and the features are free to use, but the repository states users may not use it for production use or for a Database Service without an agreement with the named Man Group entity (man-group/ArcticDB, README). It is source-available under stated restrictions, not unrestricted open source, and if you cite it in an interview as an option for a production platform, know the licence position before you do. Check the current licence text yourself; licences change and the repository is the authority.

A worked scenario: the late correction

This scenario is written for illustration. It is not an account of any firm's incident.

A researcher reports that a signal that backtested well last month now backtests badly, with no code change. The platform team has four candidate explanations and one hour.

1. The data changed. A vendor reissued a month of history with corrections. Under a store that overwrites, this is invisible. Under a versioned store, it is a two-line diff between versions. 2. The read changed. The researcher's slice now spans a boundary where a new ingestion path took over, and the two paths disagree about how a missing value is represented. 3. The clock changed. The backtest reads data as of the query time rather than as of the simulated time, so a correction published on Thursday is visible to a Monday simulation. This is leakage, and it usually flatters the earlier run rather than the later one - so a *worse* result after a correction is a clue the earlier run was the wrong one. 4. Nothing changed and the signal was noise. The least popular explanation and often the right one.

The platform's job is to make the first three answerable in minutes, with evidence, and to leave the fourth to the researcher. Concretely that means: every read is resolvable to a version; every dataset carries both an effective date and a knowledge date; and the ingestion path that produced a row is recorded on the row, not in a runbook.

A platform team that can settle this question quickly is worth more to a research group than one that shaves latency off reads.

Comparison: three places this work sits

Ask which of these three the seat actually is. The job title rarely says.

What transfers, and what needs work

Readiness checks

Stop preparing when you can do these, not after a fixed number of weeks.

• Explain, without notes, the difference between an effective date and a knowledge date, and give an example where confusing them changes a result.
• Describe a storage layout decision you made and what it cost on the read path.
• Talk through how you would detect that a dataset was silently restated.
• Explain one leakage mechanism that lives in the data layer rather than in the model.
• Describe a reprocessing run you owned: what it touched, how you knew it was correct, and how you would have rolled it back.

Questions to ask the team

• Can a researcher here reproduce a result from a year ago exactly? Walk me through what they type.
• Where does the corrections handling live - in ingestion, in the store, or in each researcher's notebook?
• Who arbitrates when two groups disagree about the same dataset?
• What proportion of this team's week is new platform work versus data incidents?
• Is the platform a product with users, or a service ticket queue? What evidence would show me which?

What this guide does not claim

It does not say that every quant firm uses a versioned store, that ArcticDB is representative of the sector, or that research data pay carries a fixed premium over platform work elsewhere. Pay varies by firm, function, seniority and market, and the salary index states what is advertised separately from what is modelled.