Signals and Threads

Will Wilson is the founder and CEO of Antithesis, which is trying to change how people test software. The idea is that you run your application inside a special hypervisor environment that intelligently (and deterministically) explores the program’s state space, allowing you to pinpoint and replay the events leading to crashes, bugs, and violations of invariants. In this episode, he and Ron take a broad view of testing, considering not just “the unreasonable effectiveness of example-based tests” but also property-based testing, fuzzing, chaos testing, type systems, and formal methods. How do you blend these techniques to find the subtle, show-stopper bugs that will otherwise wake you up at 3am? As Will has discovered, making testing less painful is actually a tour of some of computer science’s most vexing and interesting problems. You can find the transcript for this episode on our website. Some links to topics that came up in the discussion: Antithesis, Will’s company FoundationDB’s deterministic simulation framework QuickCheck — the original Haskell property-based testing library, by Koen Claessen and John Hughes Hypothesis — property-based testing for Python, created by David MacIver QuviQ — John Hughes’ company commercializing QuickCheck, including automotive testing work Netflix Chaos Monkey Goodhart’s law — “When a measure becomes a target, it ceases to be a good measure” CAP theorem — the impossibility result for distributed systems that FoundationDB claims to have in some sense violated. Paxos — the consensus algorithm FoundationDB reimplemented from scratch Large cardinals, an area Will studied before abandoning mathematics Lyapunov exponent — measure of chaotic divergence Chesterton’s fence The Story of the Flash Fill Feature in Excel Building a C compiler with a team of parallel Claudes Barak Richman, “How Community Institutions Create Economic Advantage: Jewish Diamond Merchants in New York”

Chris Lattner is the creator of LLVM and led the development of the Swift language at Apple. With Mojo, he’s taking another big swing: How do you make the process of getting the full power out of modern GPUs productive and fun? In this episode, Ron and Chris discuss how to design a language that’s easy to use while still providing the level of control required to write state of the art kernels. A key idea is to ask programmers to fully reckon with the details of the hardware, but making that work manageable and shareable via a form of type-safe metaprogramming. The aim is to support both specialization to the computation in question as well as to the hardware platform. “Somebody has to do this work,” Chris says, “if we ever want to get to an ecosystem where one vendor doesn’t control everything.” You can find the transcript for this episode on our website. Some links to topics that came up in the discussion: Democratizing AI compute (an 11-part series) Modular AI Mojo MLIR Swift

Daniel Pontecorvo runs the “physical engineering” team at Jane Street. This group blends architecture, mechanical engineering, electrical engineering, and construction management to build functional physical spaces. In this episode, Ron and Dan go deep on the challenge of heat exchange in a datacenter, especially in the face of increasingly dense power demands—and the analogous problem of keeping traders cool at their desks. Along the way they discuss the way ML is changing the physical constraints of computing; the benefits of having physical engineering expertise in-house; the importance of monitoring; and whether you really need Apollo-style CO2 scrubbers to ensure your office gets fresh air. You can find the transcript for this episode on our website. Some links to topics that came up in the discussion: ASHRAE (American Society of Heating, Refrigerating and Air-Conditioning Engineers) Some research on CO2’s effects on human performance, which motivated us to look into CO2 Scrubbers The Open Compute Project Rail-Optimized and Rail-only network topologies. Immersion cooling, where you submerge a machine in a dielectric fluid!

Ian Henry started his career at Warby Parker and Trello, building consumer apps for millions of users. Now he writes high-performance tools for a small set of experts on Jane Street’s options desk. In this episode, Ron and Ian explore what it’s like writing code at a company that has been “on its own parallel universe software adventure for the last twenty years.” Along the way, they go on a tour of Ian’s whimsical and sophisticated side projects—like Bauble, a playground for rendering trippy 3D shapes using signed distance functions—that have gone on to inform his work: writing typesafe frontend code for users who measure time in microseconds and prefer their UIs to be “six pixels high.” You can find the transcript for this episode on our website. Some links to topics that came up in the discussion: Bauble studio Janet for Mortals, by Ian Henry What if writing tests was a joyful experience?

In Young Cho thought she was going to be a doctor but fell into a trading internship at Jane Street. Now she helps lead the research group’s efforts in machine learning. In this episode, In Young and Ron touch on the porous boundaries between trading, research, and software engineering, which require different sensibilities but are often blended in a single person. They discuss the tension between flexible research tools and robust production systems; the challenges of ML in a low-data, high-noise environment subject to frequent regime changes; and the shift from simple linear models to deep neural networks. You can find the transcript for this episode on our website.

Sylvain Gugger is a former math teacher who fell into machine learning via a MOOC and became an expert in the low-level performance details of neural networks. He’s now on the ML infrastructure team at Jane Street, where he helps traders speed up their models. In this episode, Sylvain and Ron go deep on learning rate schedules; the subtle performance bugs PyTorch lets you write; how to keep a hungry GPU well-fed; and lots more, including the foremost importance of reproducibility in training runs. They also discuss some of the unique challenges of doing ML in the world of trading, like the unusual size and shape of market data and the need to do inference at shockingly low latencies. You can find the transcript for this episode on our website. Some links to topics that came up in the discussion: “Practical Deep Learning for Coders,” a FastAI MOOC by Jeremy Howard, and the book, of which Sylvain is a co-author. The Stanford DAWNBench competition that Sylvain participated in. HuggingFace, and the Accelerate library that Sylvain wrote there. Some of the languages/systems for expression ML models that were discussed: PyTorch, TensorFlow, Jax, Mojo, and Triton CUDA graphs and streams Hogwild concurrency

Liora Friedberg is a Production Engineer at Jane Street with a background in economics and computer science. In this episode, Liora and Ron discuss how production engineering blends high-stakes puzzle solving with thoughtful software engineering, as the people doing support build tools to make that support less necessary. They also discuss how Jane Street uses both tabletop simulation and hands-on exercises to train Production Engineers; what skills effective Production Engineers have in common; and how to create a culture where people aren’t blamed for making costly mistakes. You can find the transcript for this episode on our website. Some links to topics that came up in the discussion: More about production engineering at Jane Street, including how to apply. Notes on Site reliability engineering in the wider world. Alarm fatigue and desensitization. Jane Street’s 1950’s era serialization-format of choice, Some games that Streeters have used for training people to respond to incidents.

Erin Murphy is Jane Street’s first UX designer, and before that, she worked at NASA’s Jet Propulsion Laboratory building user interfaces for space missions. She’s also an illustrator with her own quarterly journal. In this episode, Erin and Ron discuss the challenge of doing user-centered design in an organization where experts are used to building tools for themselves. How do you bring a command-line interface to the web without making it worse for power users? They also discuss how beauty in design is more about utility than aesthetics; what Jane Street looks for in UX candidates; and how to help engineers discover what their users really want. You can find the transcript for this episode on our website. Some links to topics that came up in the discussion: Erin’s website that shows off her work. Her quarterly journal of sketches and observations. An article about Erin’s design work with NASA JPL. A paper that among other things talks about the user study work that Erin did at JPL. Jane Street’s current UX job opening.

Andrew Hunter makes code really, really fast. Before joining Jane Street, he worked for seven years at Google on multithreaded architecture, and was a tech lead for tcmalloc, Google’s world-class scalable malloc implementation. In this episode, Andrew and Ron discuss how, paradoxically, it can be easier to optimize systems at hyperscale because of the impact that even miniscule changes can have. Finding performance wins in trading systems—which operate at a smaller scale, but which have bursty, low-latency workloads—is often trickier. Andrew explains how he approaches the problem, including his favorite profiling techniques and tools for visualizing traces; the unique challenges of optimizing OCaml versus C++; and when you should and shouldn’t care about nanoseconds. They also touch on the joys of musical theater, and how to pass an interview when you’re sleep-deprived. You can find the transcript for this episode on our website. Some links to topics that came up in the discussion: “Profiling a warehouse-scale computer” Magic-trace OODA loop

Peter Bogart-Johnson was one of Jane Street’s first program managers, and helped bring the art of PMing—where that “P” variously stands for “project,” “product,” or some blend of the two—to the company at large. He’s also a poet and the editor of a literary magazine. In this episode, Peter and Ron discuss the challenge of gaining trust as an outsider: how do you teach teams a new way of doing things while preserving what’s already working? The key, Peter says, is you listen; a good PM is an anthropologist. They also discuss how paying down technical debt isn’t something you do instead of serving customers; what Jane Street looks for in PM candidates; and how to help teams coordinate in times of great change. You can find the transcript for this episode on our website. Some links to topics that came up in the discussion: LIT Magazine (more recently here ) How to be a PM that engineers don’t hate and How to be an engineer that PMs don’t hate

Richard Eisenberg is one of the core maintainers of Haskell. He recently joined Jane Street’s Tools and Compilers team, where he hacks on the OCaml compiler. He and Ron discuss the powerful language feature that got him into PL design in the first place—dependent types—and its role in a world where AIs can (somewhat) competently write your code for you. They also discuss the differences between Haskell and OCaml; the perils of trying to make a language that works for everybody; and how best a company like Jane Street can collaborate with the open source community. You can find the transcript for this episode on our website. Some links to topics that came up in the discussion: Dependent types GHC Unboxed types in OCaml Language extensions in Haskell

Ella Ehrlich has been a developer at Jane Street for close to a decade. During much of that time, she’s worked on Gord, one of Jane Street’s oldest and most critical systems, which is responsible for normalizing and distributing the firm’s trading data. Ella and Ron talk about how to grow and modernize a legacy system without compromising uptime, why game developers are the “musicians of software,” and some of the work Jane Street has done to try to hire a more diverse set of software engineers. You can find the transcript for this episode on our website. Some links to topics that came up in the discussion: EG, The League of Legends team that Ella is a huge fan of. Apache Kafka, the message bus that Gord migrated to. Some of the various sources of symbology you have to deal with when normalizing trading data. (Really, there are too many sources to list here!) A list of Jane Street’s recruiting Programs and Events, including INSIGHT, which focuses on women, and IN FOCUS, which focuses on historically underrepresented ethnic or racial minorities.

Doug Patti is a developer in Jane Street’s Client-Facing Tech team, where he works on a system called Concord that undergirds Jane Street’s client offerings. In this episode, Doug and Ron discuss how Concord, which has state-machine replication as its core abstraction, helps Jane Street achieve the reliability, scalability, and speed that the client business demands. They’ll also discuss Doug’s involvement in building a successor system called Aria, which is designed to deliver those same benefits to a much wider audience. You can find the transcript for this episode on our website. Some links to topics that came up in the discussion: Jane Street’s client-facing trading platforms A Signals and Threads episode on market data and multicast which discusses some of the history of state-machine replication in the markets. The FIX protocol UDP multicast Reliable multicast Kafka

Stephen Dolan works on Jane Street’s Tools and Compilers team where he focuses on the OCaml compiler. In this episode, Stephen and Ron take a trip down memory lane, discussing how to manage computer memory efficiently and safely. They consider trade-offs between reference counting and garbage collection, the surprising gains achieved by prefetching, and how new language features like local allocation and unboxed types could give OCaml users more control over their memory. You can find the transcript for this episode on our website. Some links to topics that came up in the discussion: Stephen’s command-line JSON processor, jq Stephen’s Cambridge dissertation, “Algebraic Subtyping”, and a protoype implementation of mlsub, a language based on those ideas. A post from Stephen on how to benchmark different memory allocators. A Jane Street tech talk on “Unboxed Types for OCaml”, and an RFC in the OCaml RFC repo. A paper from Stephen and KC Sivaramakrishnan called “Bounding Data Races in Space and Time”, which is all about a new and better memory model for Multicore OCaml. Another paper describing the design of OCaml’s multicore GC. The Rust RFC for Higher-ranked trait bounds.

Anil Madhavapeddy is an academic, author, engineer, entrepreneur, and OCaml aficionado. In this episode, Anil and Ron consider the evolving role of operating systems, security on the internet, and the pending arrival (at last!) of OCaml 5.0. They also discuss using Raspberry Pis to fight climate change; the programming inspiration found in British pubs and on Moroccan beaches; and the time Anil went to a party, got drunk, and woke up with a job working on the Mars Polar Lander. You can find the transcript for this episode on our website. Some links to topics that came up in the discussion: Ron, Anil, and Jason Hickey’s book, “Real World OCaml” Anil’s personal website and Google Scholar page The MirageOS library operating system Cambridge University’s OCaml Labs NASA’s Mars Polar Lander The Xen Project, home to the hypervisor The Tezos proof-of-stake blockchain The Coq Proof Assistant system

Ty Overby is a programmer in Jane Street’s web platform group where he works on Bonsai, our OCaml library for building interactive browser-based UI. In this episode, Ty and Ron consider the functional approach to building user interfaces. They also discuss Ty’s programming roots in Neopets, what development features they crave on the web, the unfairly maligned CSS, and why Excel is “arguably the greatest programming language ever developed.” You can find the transcript for this episode on our website. Some links to topics that came up in the discussion: Jane Street’s Bonsai library The 3D design system OpenSCAD Matt Keeter’s libfive design tools Try.NET in-browser repl Jane Street’s Incr_dom library The Elm Architecture “pattern for architecting interactive programs” React JavaScript library The Houdini proposal Svelte UI toolkit

James Somers is Jane Street’s writer-in-residence, splitting his time between English and OCaml, and helping to push forward all sorts of efforts around knowledge-sharing at Jane Street. In this episode, James and Ron talk about the role of technical writing in an organization like Jane Street, and how engineering software relates to editing prose. You can find the transcript for this episode on our website. Some links to topics that came up in the discussion: mdx, the modified Markdown format that supports executing OCaml code blocks More on the 4 types of technical writing that James references Donald Knuth’s original book on Literate Programming More on John McPhee’s use of KEDIT Peter Seibel’s Coders at Work David Goodsell’s The Machinery of Life Scott Huler’s Defining the Wind Some of James’s writing on our tech blog

Signals & Threads is back, and we have a fun season of topics lined up, including: Building better abstractions for design and user interfaces, the role of writing in a technical organization, the approach that different languages take to memory management...and more. We hope you’ll join us. The first episode drops September 1st.

In this week's episode, the season 1 finale, Ron speaks with Jeanne, Matt, and Grace, three former tech interns at Jane Street who have returned as full-timers. They talk about the experience of being an intern at Jane Street, the types of projects that interns work on, and how they've found the transition to full-time work. You can find the transcript for this episode along with links to things we discussed on our website.

Despite a steady trickle of newcomers, email still reigns supreme as the chief communication mechanism for the Information Age. At Jane Street, it’s just as critical as anywhere, but there’s one difference: the system at the heart of our email infrastructure is homegrown. This week, Ron talks to Dominick LoBraico, an engineer working on Jane Street’s technology infrastructure, about how and why we built Mailcore, an email server written and configured in OCaml. They delve into questions around how best to represent the configuration of a complex system, when you should build your own and when you shouldn’t, and the benefits of bringing a code-focused approach to solving systems problems. You can find the transcript for this episode along with links to things we discussed on our website.