跳到主要内容

Tardis

Tardis provides granular 数据 for cryptocurrency markets including tick-by-tick order book snapshots & updates, trades, open interest, funding rates, options chains and liquidations 数据 for leading crypto exchanges.

NautilusTrader provides an integration with the Tardis API and 数据 formats, enabling seamless access. The capabilities of this adapter include:

  • TardisCSVDataLoader: Reads Tardis-format CSV files and converts them into Nautilus data, with support for both bulk loading and memory-efficient streaming.
  • TardisMachineClient: Supports live streaming and historical replay of data from the Tardis Machine WebSocket server - converting messages into Nautilus data.
  • TardisHttpClient: Requests instrument definition metadata from the Tardis HTTP API, parsing it into Nautilus instrument definitions.
  • TardisDataClient: Provides a live data client for subscribing to data streams from a Tardis Machine WebSocket server.
  • TardisInstrumentProvider: Provides instrument definitions from Tardis through the HTTP instrument metadata API.
  • Data pipeline functions: Enables replay of historical data from Tardis Machine and writes it to the Nautilus Parquet format, including direct catalog integration for streamlined data management (see below).
信息

A Tardis API key is required for the adapter to operate correctly. See also environment variables.

概览​

This adapter is implemented in Rust, with optional Python bindings for ease of use in Python-based workflows. It does not require any external Tardis 客户端 库 依赖.

信息

There is no need for additional 安装 steps for Tardis. The core components of the adapter are compiled as static libraries and automatically linked during the build process.

Tardis 文档​

Tardis provides extensive user documentation. We recommend also referring to the Tardis 文档 in conjunction with this NautilusTrader integration 指南.

Supported formats​

Tardis provides normalized market 数据—a unified format consistent across all supported exchanges. This normalization is highly valuable because it allows a single parser to handle data from any Tardis-supported exchange, reducing development time and complexity. As a result, NautilusTrader will not 支持 exchange-native market 数据 formats, as it would be inefficient to implement separate parsers for each exchange at this stage.

The following normalized Tardis formats are supported by NautilusTrader:

Tardis formatNautilus 数据 type
book_changeOrderBookDelta
book_snapshot_*OrderBookDepth10
quoteQuoteTick
quote_10sQuoteTick
tradeTrade
trade_bar_*Bar
instrumentCurrencyPair, CryptoFuture, CryptoPerpetual, OptionContract
derivative_tickerNot yet supported
disconnectNot applicable

Notes:

信息

See also the Tardis normalized market data APIs.

Bars​

The adapter will automatically convert Tardis trade bar interval and suffix to Nautilus BarTypes. This includes the following:

Tardis suffixNautilus bar aggregation
ms - millisecondsMILLISECOND
s - secondsSECOND
m - minutesMINUTE
ticks - number of ticksTICK
vol - volume sizeVOLUME

Symbology and normalization​

The Tardis integration ensures seamless 兼容性 with NautilusTrader’s crypto exchange 适配器 by consistently normalizing symbols. Typically, NautilusTrader uses the native exchange naming conventions provided by Tardis. However, for certain exchanges, raw symbols are adjusted to adhere to the Nautilus symbology normalization, as outlined below:

Common rules​

  • All symbols are converted to uppercase.
  • Market type suffixes are appended with a hyphen for some exchanges (see exchange-specific normalizations).
  • Original exchange symbols are preserved in the Nautilus instrument definitions raw_symbol field.

Exchange-specific normalizations​

  • Binance: Nautilus appends the suffix -PERP to all perpetual symbols.
  • Bybit: Nautilus uses specific product category suffixes, including -SPOT, -LINEAR, -INVERSE, -OPTION.
  • dYdX: Nautilus appends the suffix -PERP to all perpetual symbols.
  • Gate.io: Nautilus appends the suffix -PERP to all perpetual symbols.

For detailed symbology 文档 per exchange:

Venues​

Some exchanges on Tardis are partitioned into multiple venues. The table below outlines the mappings between Nautilus venues and corresponding Tardis exchanges, as well as the exchanges that Tardis supports:

Nautilus venueTardis exchange(s)
ASCENDEXascendex
BinanceBinance, Binance-dex, Binance-european-options, Binance-futures, Binance-jersey, Binance-options
BINANCE_DELIVERYBinance-delivery (COIN-margined contracts)
BINANCE_USBinance-us
BITFINEXbitfinex, bitfinex-derivatives
BITFLYERbitflyer
BITGETbitget, bitget-futures
BitMEXBitMEX
BITNOMIALbitnomial
BITSTAMPbitstamp
BLOCKCHAIN_COMblockchain-com
BybitBybit, Bybit-options, Bybit-spot
CoinbaseCoinbase
COINBASE_INTXCoinbase-international
COINFLEXcoinflex (for historical research)
CRYPTO_COMcrypto-com, crypto-com-derivatives
CRYPTOFACILITIEScryptofacilities
DELTAdelta
DERIBITderibit
dYdXdYdX
DYDX_V4dYdX-v4
FTXftx, ftx-us (historical research)
GATE_IOgate-io, gate-io-futures
GEMINIgemini
HITBTChitbtc
HUOBIhuobi, huobi-dm, huobi-dm-linear-swap, huobi-dm-options
HUOBI_DELIVERYhuobi-dm-swap
HyperliquidHyperliquid
KRAKENkraken
KUCOINkucoin, kucoin-futures
MANGOmango
OKCOINokcoin
OKEXokex, okex-futures, okex-options, okex-spreads, okex-swap
PHEMEXphemex
POLONIEXpoloniex
SERUMserum (historical research)
STAR_ATLASstar-atlas
UPBITupbit
WOO_Xwoo-x

环境 variables​

The following 环境 variables are used by Tardis and NautilusTrader.

  • TM_API_KEY: API key for the Tardis Machine.
  • TARDIS_API_KEY: API key for NautilusTrader Tardis clients.
  • TARDIS_MACHINE_WS_URL (optional): WebSocket URL for the TardisMachineClient in NautilusTrader.
  • TARDIS_BASE_URL (optional): Base URL for the TardisHttpClient in NautilusTrader.
  • NAUTILUS_PATH (optional): Parent directory containing the catalog/ subdirectory for writing replay data in the Nautilus catalog format.

Running Tardis Machine historical replays​

The Tardis Machine Server is a locally runnable server with built-in 数据 caching, providing both tick-level historical and consolidated 实时 cryptocurrency market 数据 through HTTP and WebSocket APIs.

You can perform complete Tardis Machine WebSocket replays of historical 数据 and output the results in Nautilus Parquet format, using either Python or Rust. Since the function is implemented in Rust, 性能 is consistent whether run from Python or Rust, letting you choose based on your preferred workflow.

The end-to-end run_tardis_machine_replay data pipeline function utilizes a specified configuration to execute the following steps:

  • Connect to the Tardis Machine 服务器.
  • Request and parse all necessary instrument definitions from the Tardis instruments metadata HTTP API.
  • Stream all requested 金融工具 and 数据 types for the specified time ranges from the Tardis Machine 服务器.
  • For each instrument, 数据 type and date (UTC), generate a .parquet file in the Nautilus format.
  • Disconnect from the Tardis Marchine 服务器, and terminate the program.
备注

You can request 数据 for the first day of each month without an API key. For all other dates, a Tardis Machine API key is required.

This process is optimized for direct output to a Nautilus Parquet 数据 catalog. Ensure that the NAUTILUS_PATH 环境 variable is set to the parent directory containing the catalog/ subdirectory. Parquet files will then be organized under <NAUTILUS_PATH>/catalog/数据/ in the expected subdirectories corresponding to 数据 type and instrument.

If no output_path is specified in the 配置 file and the NAUTILUS_PATH 环境 variable is unset, the system will default to the current working directory.

Procedure​

First, ensure the Tardis-machine docker 容器 is running. Use the following command:

docker run -p 8000:8000 -p 8001:8001 -e "TM_API_KEY=YOUR_API_KEY" -d tardisdev/tardis-machine

This command starts the Tardis-machine 服务器 without a persistent local 缓存, which may affect 性能. For improved performance, consider running the server with a persistent volume. Refer to the Tardis Docker documentation for details.

配置​

Next, ensure you have a 配置 JSON file available.

配置 JSON format

FieldTypeDescriptionDefault
tardis_ws_urlstring (optional)The Tardis Machine WebSocket URL.If null then will use the TARDIS_MACHINE_WS_URL env var.
normalize_symbolsbool (optional)If Nautilus symbol normalization should be applied.If null then will default to true.
output_pathstring (optional)The output directory path to write Nautilus Parquet 数据 to.If null then will use the NAUTILUS_PATH env var, otherwise the current working directory.
optionsJSON[]An array of ReplayNormalizedRequestOptions objects.

An example configuration file, example_config.json, is available here:

{
  "tardis_ws_url": "ws://localhost:8001",
  "output_path": null,
  "options": [
    {
      "exchange": "bitmex",
      "symbols": [
        "xbtusd",
        "ethusd"
      ],
      "data_types": [
        "trade"
      ],
      "from": "2019-10-01",
      "to": "2019-10-02"
    }
  ]
}

Python Replays​

To run a replay in Python, create a script similar to the following:

import asyncio

from nautilus_trader.core import nautilus_pyo3


async def run():
    config_filepath = Path("YOUR_CONFIG_FILEPATH")
    await nautilus_pyo3.run_tardis_machine_replay(str(config_filepath.resolve()))


if __name__ == "__main__":
    asyncio.run(run())

Rust Replays​

To run a replay in Rust, create a binary similar to the following:

use std::{env, path::PathBuf};

use nautilus_adapters::tardis::replay::run_tardis_machine_replay_from_config;

#[tokio::main]
async fn main() {
    tracing_subscriber::fmt()
        .with_max_level(tracing::Level::DEBUG)
        .init();

    let config_filepath = PathBuf::from("YOUR_CONFIG_FILEPATH");
    run_tardis_machine_replay_from_config(&config_filepath).await;
}

Make sure to enable Rust 日志记录 by exporting the following 环境 variable:

export RUST_LOG=debug

A working example binary can be found here.

This can also be run using cargo:

cargo run --bin tardis-replay <path_to_your_config>

Loading Tardis CSV 数据​

Tardis-format CSV 数据 can be loaded using either Python or Rust. The loader reads the CSV text 数据 from disk and parses it into Nautilus 数据. Since the loader is implemented in Rust, 性能 remains consistent regardless of whether you run it from Python or Rust, allowing you to choose based on your preferred workflow.

You can also optionally specify a limit parameter for the load_* functions/methods to control the maximum number of rows loaded.

备注

Loading mixed-instrument CSV files is challenging due to precision 要求 and is not recommended. Use single-instrument CSV files instead (see below).

Loading CSV 数据 in Python​

You can load Tardis-format CSV 数据 in Python using the TardisCSVDataLoader. When loading 数据, you can optionally specify the instrument ID but must specify both the price precision, and size precision. Providing the instrument ID improves loading 性能, while specifying the precisions is required, as they cannot be inferred from the text 数据 alone.

To load the 数据, create a script similar to the following:

from nautilus_trader.adapters.tardis import TardisCSVDataLoader
from nautilus_trader.model import InstrumentId


instrument_id = InstrumentId.from_str("BTC-PERPETUAL.DERIBIT")
loader = TardisCSVDataLoader(
    price_precision=1,
    size_precision=0,
    instrument_id=instrument_id,
)

filepath = Path("YOUR_CSV_DATA_PATH")
limit = None

deltas = loader.load_deltas(filepath, limit)

Loading CSV 数据 in Rust​

You can load Tardis-format CSV data in Rust using the loading functions found here. When loading 数据, you can optionally specify the instrument ID but must specify both the price precision and size precision. Providing the instrument ID improves loading 性能, while specifying the precisions is required, as they cannot be inferred from the text 数据 alone.

For a complete example, see the example binary here.

To load the 数据, you can use code similar to the following:

use std::path::Path;

use nautilus_adapters::tardis;
use nautilus_model::identifiers::InstrumentId;

#[tokio::main]
async fn main() {
    // You must specify precisions and the CSV filepath
    let price_precision = 1;
    let size_precision = 0;
    let filepath = Path::new("YOUR_CSV_DATA_PATH");

    // Optionally specify an instrument ID and/or limit
    let instrument_id = InstrumentId::from("BTC-PERPETUAL.DERIBIT");
    let limit = None;

    // Consider propagating any parsing error depending on your workflow
    let _deltas = tardis::csv::load_deltas(
        filepath,
        price_precision,
        size_precision,
        Some(instrument_id),
        limit,
    )
    .unwrap();
}

Streaming Tardis CSV 数据​

For 内存-efficient processing of large CSV files, the Tardis integration provides streaming capabilities that load and process 数据 in configurable chunks rather than loading entire files into 内存 at once. This is particularly useful for processing multi-gigabyte CSV files without exhausting system 内存.

The streaming functionality is available for all supported Tardis 数据 types:

  • Order book deltas (stream_deltas).
  • Quote ticks (stream_quotes).
  • Trade ticks (stream_trades).
  • Order book depth snapshots (stream_depth10).

Streaming CSV 数据 in Python​

The TardisCSVDataLoader provides streaming methods that yield chunks of 数据 as iterators. Each method accepts a chunk_size parameter that controls how many records are read from the CSV file per chunk:

from nautilus_trader.adapters.tardis import TardisCSVDataLoader
from nautilus_trader.model import InstrumentId

instrument_id = InstrumentId.from_str("BTC-PERPETUAL.DERIBIT")
loader = TardisCSVDataLoader(
    price_precision=1,
    size_precision=0,
    instrument_id=instrument_id,
)

filepath = Path("large_trades_file.csv")
chunk_size = 100_000  # Process 100,000 records per chunk (default)

# Stream trade ticks in chunks
for chunk in loader.stream_trades(filepath, chunk_size):
    print(f"Processing chunk with {len(chunk)} trades")
    # Process each chunk - only this chunk is in memory
    for trade in chunk:
        # Your processing logic here
        pass

Streaming Order Book 数据​

For order book 数据, streaming is available for both deltas and depth snapshots:

# Stream order book deltas
for chunk in loader.stream_deltas(filepath):
    print(f"Processing {len(chunk)} deltas")
    # Process delta chunk

# Stream depth10 snapshots (specify levels: 5 or 25)
for chunk in loader.stream_depth10(filepath, levels=5):
    print(f"Processing {len(chunk)} depth snapshots")
    # Process depth chunk

Streaming Quote 数据​

Quote 数据 can be streamed similarly:

# Stream quote ticks
for chunk in loader.stream_quotes(filepath):
    print(f"Processing {len(chunk)} quotes")
    # Process quote chunk

内存 Efficiency Benefits​

The streaming approach provides significant 内存 efficiency advantages:

  • Controlled Memory Usage: Only one chunk is loaded in memory at a time.
  • Scalable Processing: Can process files larger than available RAM.
  • Configurable Chunk Sizes: Tune chunk_size based on your system's memory and performance requirements (default 100,000).
注意

When using streaming with precision inference (not providing explicit precisions), the inferred precision may differ from bulk loading the entire file. This is because precision inference works within chunk boundaries, and different chunks may contain values with different precision 要求. For deterministic precision behavior, provide explicit price_precision and size_precision parameters when calling streaming methods.

Streaming CSV 数据 in Rust​

The underlying streaming functionality is implemented in Rust and can be used directly:

use std::path::Path;
use nautilus_adapters::tardis::csv::{stream_trades, stream_deltas};
use nautilus_model::identifiers::InstrumentId;

#[tokio::main]
async fn main() {
    let filepath = Path::new("large_trades_file.csv");
    let chunk_size = 100_000;
    let price_precision = Some(1);
    let size_precision = Some(0);
    let instrument_id = Some(InstrumentId::from("BTC-PERPETUAL.DERIBIT"));

    // Stream trades in chunks
    let stream = stream_trades(
        filepath,
        chunk_size,
        price_precision,
        size_precision,
        instrument_id,
    ).unwrap();

    for chunk_result in stream {
        match chunk_result {
            Ok(chunk) => {
                println!("Processing chunk with {} trades", chunk.len());
                // Process chunk
            }
            Err(e) => {
                eprintln!("Error processing chunk: {}", e);
                break;
            }
        }
    }
}

Requesting instrument definitions​

You can request instrument definitions in both Python and Rust using the TardisHttpClient. This client interacts with the Tardis instruments metadata API to request and parse instrument metadata into Nautilus instruments.

The TardisHttpClient constructor accepts optional parameters for api_key, base_url, and timeout_secs (default is 60 seconds).

The 客户端 provides methods to retrieve either a specific instrument, or all 金融工具 available on a particular exchange. Ensure that you use Tardis’s lower-kebab casing convention when referring to a Tardis-supported exchange.

备注

A Tardis API key is required to access the 金融工具 metadata API.

Requesting 金融工具 in Python​

To request instrument definitions in Python, create a script similar to the following:

import asyncio

from nautilus_trader.core import nautilus_pyo3


async def run():
    http_client = nautilus_pyo3.TardisHttpClient()

    instrument = await http_client.instrument("bitmex", "xbtusd")
    print(f"Received: {instrument}")

    instruments = await http_client.instruments("bitmex")
    print(f"Received: {len(instruments)} instruments")


if __name__ == "__main__":
    asyncio.run(run())

Requesting 金融工具 in Rust​

To request instrument definitions in Rust, use code similar to the following. For a complete example, see the example binary here.

use nautilus_adapters::tardis::{enums::Exchange, http::client::TardisHttpClient};

#[tokio::main]
async fn main() {
    tracing_subscriber::fmt()
        .with_max_level(tracing::Level::DEBUG)
        .init();

    let client = TardisHttpClient::new(None, None, None).unwrap();

    // Nautilus instrument definitions
    let resp = client.instruments(Exchange::Bitmex).await;
    println!("Received: {resp:?}");

    let resp = client.instrument(Exchange::Bitmex, "ETHUSDT").await;
    println!("Received: {resp:?}");
}

Instrument provider​

The TardisInstrumentProvider requests and parses instrument definitions from Tardis through the HTTP instrument metadata API. Since there are multiple Tardis-supported exchanges, when loading all instruments, you must filter for the desired venues using an InstrumentProviderConfig:

from nautilus_trader.config import InstrumentProviderConfig

# See supported venues https://nautilustrader.io/docs/nightly/integrations/tardis#venues
venues = {"BINANCE", "BYBIT"}
filters = {"venues": frozenset(venues)}
instrument_provider_config = InstrumentProviderConfig(load_all=True, filters=filters)

You can also load specific instrument definitions in the usual way:

from nautilus_trader.config import InstrumentProviderConfig

instrument_ids = [
    InstrumentId.from_str("BTCUSDT-PERP.BINANCE"),  # Will use the 'binance-futures' exchange
    InstrumentId.from_str("BTCUSDT.BINANCE"),  # Will use the 'binance' exchange
]
instrument_provider_config = InstrumentProviderConfig(load_ids=instrument_ids)
备注

金融工具 must be available in the 缓存 for all subscriptions. For simplicity, it’s recommended to load all 金融工具 for the venues you intend to subscribe to.

Live 数据 客户端​

The TardisDataClient enables integration of a Tardis Machine with a running NautilusTrader system. It supports subscriptions to the following 数据 types:

  • OrderBookDelta (L2 granularity from Tardis, includes all changes or full-depth snapshots)
  • OrderBookDepth10 (L2 granularity from Tardis, provides snapshots up to 10 levels)
  • QuoteTick
  • TradeTick
  • Bar (trade bars with Tardis-supported bar aggregations)

数据 WebSockets​

The main TardisMachineClient 数据 WebSocket manages all stream subscriptions received during the initial connection phase, up to the duration specified by ws_connection_delay_secs. For any additional subscriptions made after this period, a new TardisMachineClient is created. This approach optimizes 性能 by allowing the main WebSocket to handle potentially hundreds of subscriptions in a single stream if they are provided at startup.

When an initial subscription delay is set with ws_connection_delay_secs, unsubscribing from any of these streams will not actually remove the subscription from the Tardis Machine stream, as selective unsubscription is not supported by Tardis. However, the 组件 will still unsubscribe from message bus publishing as expected.

All subscriptions made after any initial delay will behave normally, fully unsubscribing from the Tardis Machine stream when requested.

提示

If you anticipate frequent subscription and unsubscription of 数据, it is recommended to set ws_connection_delay_secs to zero. This will create a new 客户端 for each initial subscription, allowing them to be later closed individually upon unsubscription.

Limitations and considerations​

The following limitations and considerations are currently known:

  • Historical 数据 requests are not supported, as each would require a minimum one-day replay from the Tardis Machine, potentially with a filter. This approach is neither practical nor efficient.
最后由 huojiabin 于 更新