Skip to main content

Project case study

Maxwell — ML Data Pipelines

Data Engineering

GPU-accelerated pipelines processing 20+ years of market data with Dask, RAPIDS, and Apache Arrow

PythonPandasNumPyNumbaRAPIDSDaskApache ArrowApache ParquetScikit-learnNVIDIA CUDA

Custom libraries for acquiring, preparing, and storing financial market data. They provide WebSockets/REST connectors, anomaly detection and cleansing across hundreds of millions of data points, and domain-specific feature engineering to produce high-integrity modeling data.

Maxwell processed 20+ years of trading and order-book market data at multiple time resolutions from high-volume real-time market APIs, accelerating storage and retrieval with Apache Arrow in-memory columnar formats and Apache Parquet persistent on-disk storage.

Key capabilities:

  • Automated labeling of time-series data with algorithmic trading attributes such as stop losses, long/short price targets, ATR, and VWAP
  • Domain-driven feature engineering from technical indicators and statistical markers, with automated binning, one-hot encoding, scaling, and normalization tuned for market microstructure data
  • ML pipeline orchestration tools managing data flow across modeling and algorithmic-trading workflows with fluid feature extraction
  • GPU-accelerated (CUDA) processing with Dask and RAPIDS for hundreds of millions of data points