Principal Machine Learning Infrastructure Engineer

About us PhysicsX is a deep-tech company with roots in numerical physics and Formula One, dedicated to accelerating hardware innovation at the speed of software. We are building an AI-driven simulation software stack for engineering and manufacturing across advanced industries. By enabling high-fidelity, multi-physics simulation through AI inference across the entire engineering lifecycle, PhysicsX unlocks new levels of optimization and automation in design, manufacturing, and operations — empowering engineers to push the boundaries of possibility. Our customers include leading innovators in Aerospace & Defense, Materials, Energy, Semiconductors, and Automotive.Note: We are currently recruiting for multiple positions, however please only apply for the role that best aligns with your skillset and career goals. The Role The Principal ML Infrastructure Engineer will extend and operate the infrastructure that powers our research model training, fine-tuning, and serving pipelines. You will be embedded within our Research function, partnering directly with ML engineers and research scientists to ensure they can train Large Physics Models efficiently and reliably at scale. Team Context In this role, you will be vertically embedded in Research, working daily with: Research Scientists who determine the model architectures and methods ML Engineers who implement and develop the models Simulation Data Engineers who are accountable for upstream data pipelines You will have end-to-end responsibilities over the research infrastructure, with the autonomy to make architectural decisions and the responsibility to keep data flowing reliably. Horizontally, you will be part of an infrastructure engineering group responsible for infrastructure across the company. What you will do Training Infrastructure Design and operate distributed training infrastructure for neural operator architectures (Transolver, Point Cloud Transformer, etc.) on our large NVIDIA DGX B200 platform. Optimize training pipelines for throughput, fault tolerance, and cost efficiency, including checkpointing strategies, gradient accumulation, and multi-node synchronization. Build and maintain experiment tracking and observability systems that give researchers clear visibility into training runs, hyperparameter sweeps, and model performance. Data I/O and Performance Solve data loading bottlenecks for large-scale mesh datasets. Optimize data pipelines for efficient I/O from cloud storage, including prefetching, caching, and format optimization. Work with heterogeneous data sources of varying formats and resolutions. Model Serving and Deployment Build serving infrastructure for pre-trained LPMs, supporting both zero-shot inference and uncertainty quantification (Monte Carlo Dropout). Design and implement model packaging pipelines for customer deployment. Models must run reliably in customer environments with fine-tuning capabilities. Ensure reproducibility: any model checkpoint should be dep...