Skip to main content

Data Download

Getting Data for Backtesting and Hyperopt​

To download data (candles / OHLCV) needed for backtesting and hyperoptimization use the freqtrade download-data command.

If no additional parameter is specified, freqtrade will download data for "1m" and "5m" timeframes for the last 30 days.

Exchange and pairs will come from config.json (if specified using -c/--config). Without provided configuration, --exchange becomes mandatory.

You can use a relative timerange (--days 20) or an absolute starting point (--timerange 20200101-). For incremental downloads, the relative approach should be used.

Updating Existing Data

If you already have backtesting data available in your data-directory and would like to refresh this data up to today, freqtrade will automatically calculate the missing timerange for the existing pairs and the download will occur from the latest available point until "now", neither --days or --timerange parameters are required. Freqtrade will keep the available data and only download the missing data.

If you are updating existing data after inserting new pairs that you have no data for, use the --new-pairs-days xx parameter. Specified number of days will be downloaded for new pairs while old pairs will be updated with missing data only.

Basic Usage​

Simple Download Commands​

# Download data using config.json settings
freqtrade download-data --exchange binance

# Download specific pairs
freqtrade download-data --exchange binance --pairs ETH/USDT XRP/USDT BTC/USDT

# Download all USDT pairs (regex)
freqtrade download-data --exchange binance --pairs ".*/USDT"

# Download specific timeframes
freqtrade download-data --exchange binance --timeframes 1m 5m 1h

# Download specific number of days
freqtrade download-data --exchange binance --days 10

# Download from specific date
freqtrade download-data --exchange binance --timerange 20230101-

Advanced Download Options​

# Download to custom directory
freqtrade download-data --exchange binance --datadir user_data/data/custom

# Download including inactive pairs
freqtrade download-data --exchange binance --pairs ".*/USDT" --include-inactive-pairs

# Download with specific data format
freqtrade download-data --exchange binance --data-format-ohlcv feather

# Download new pairs with specific days
freqtrade download-data --exchange binance --new-pairs-days 100
Downloading All Data for One Quote Currency

Often, you'll want to download data for all pairs of a specific quote-currency. In such cases, you can use the following shorthand:

freqtrade download-data --exchange binance --pairs ".*/USDT" <...>

The provided "pairs" string will be expanded to contain all active pairs on the exchange.

To also download data for inactive (delisted) pairs, add --include-inactive-pairs to the command.

Startup Period

download-data is a strategy-independent command. The idea is to download a big chunk of data once, and then iteratively increase the amount of data stored.

For that reason, download-data does not care about the "startup-period" defined in a strategy. It's up to the user to download additional days if the backtest should start at a specific point in time (while respecting startup period).

Download Options​

Time Range Options​

# Download last 30 days (default)
freqtrade download-data --exchange binance

# Download last 10 days
freqtrade download-data --exchange binance --days 10

# Download from specific date to now
freqtrade download-data --exchange binance --timerange 20230101-

# Download specific date range
freqtrade download-data --exchange binance --timerange 20230101-20230601

Timeframe Options​

# Download default timeframes (1m, 5m)
freqtrade download-data --exchange binance

# Download specific timeframes
freqtrade download-data --exchange binance --timeframes 1m 5m 15m 1h 4h 1d

# Download only 1-hour data
freqtrade download-data --exchange binance --timeframes 1h

Pair Selection​

# Use pairs from config.json
freqtrade download-data --config config.json

# Specify pairs directly
freqtrade download-data --exchange binance --pairs BTC/USDT ETH/USDT

# Use regex patterns
freqtrade download-data --exchange binance --pairs ".*/USDT" ".*/BTC"

# Use pairs file
freqtrade download-data --exchange binance --pairs-file user_data/data/binance/pairs.json

Download Additional Historical Data​

Prepend Data to Existing Dataset​

Assuming you downloaded all data from 2022 (--timerange 20220101-) - but you'd now like to also backtest with earlier data.

You can do so by using the --prepend flag, combined with --timerange - specifying an end-date.

freqtrade download-data --exchange binance --pairs ETH/USDT XRP/USDT BTC/USDT --prepend --timerange 20210101-20220101
Prepend Mode

Freqtrade will ignore the end-date in this mode if data is available, updating the end-date to the existing data start point.

Data Formats​

Freqtrade currently supports the following data formats:

  • feather - A dataformat based on Apache Arrow (default, recommended)
  • json - Plain "text" JSON files
  • jsongz - A gzip-compressed version of JSON files
  • parquet - Columnar datastore (OHLCV only)

By default, both OHLCV data and trades data are stored in the feather format.

Specify Data Format​

# Download in feather format (default)
freqtrade download-data --exchange binance --data-format-ohlcv feather

# Download in JSON format
freqtrade download-data --exchange binance --data-format-ohlcv json

# Download in compressed JSON format
freqtrade download-data --exchange binance --data-format-ohlcv jsongz

# Download in Parquet format
freqtrade download-data --exchange binance --data-format-ohlcv parquet

Configure Default Data Format​

To persist this change, add the following to your configuration:

{
    "dataformat_ohlcv": "feather",
    "dataformat_trades": "feather"
}

Data Format Comparison​

Performance comparison with sample data:

FormatSizeLoad TimeNotes
feather72MB3.5sRecommended - Best performance/size balance
json149MB25.6sHuman readable, large files
jsongz39MB27sCompressed, slow to load
parquet83MB3.8sGood for analytics, OHLCV only
Performance Recommendation

For the best performance/size mix, we recommend using the default feather format, or parquet for analytical workloads.

Pairs File​

In alternative to the whitelist from config.json, a pairs.json file can be used.

Create Pairs File​

# Create directory structure
mkdir -p user_data/data/binance

# Create pairs file
touch user_data/data/binance/pairs.json

Pairs File Format​

The format of the pairs.json file is a simple JSON list:

[
    "ETH/BTC",
    "ETH/USDT",
    "BTC/USDT",
    "XRP/ETH"
]
Pairs File Usage

The pairs.json file is only used when no configuration is loaded (implicitly by naming, or via --config flag).

You can force the usage of this file via --pairs-file pairs.json - however we recommend to use the pairlist from within the configuration, either via exchange.pair_whitelist or pairs setting in the configuration.

Data Conversion​

Convert Between Data Formats​

Convert existing data between different formats:

# Convert from JSON to compressed JSON
freqtrade convert-data --format-from json --format-to jsongz --datadir ~/.freqtrade/data/binance -t 5m 15m --erase

# Convert from JSON to Feather
freqtrade convert-data --format-from json --format-to feather --datadir ~/.freqtrade/data/binance

# Convert all timeframes
freqtrade convert-data --format-from json --format-to feather --datadir ~/.freqtrade/data/binance -t 1m 5m 15m 1h 4h 1d

Convert Trade Data​

# Convert trade data from compressed JSON to JSON
freqtrade convert-trade-data --format-from jsongz --format-to json --datadir ~/.freqtrade/data/kraken --erase

Data Management​

List Available Data​

# List all available data
freqtrade list-data

# Show data with timeranges
freqtrade list-data --show-timerange

# List data for specific exchange
freqtrade list-data --exchange binance

# List data in specific format
freqtrade list-data --data-format-ohlcv feather

Data Directory Structure​

user_data/data/
├── binance/
│   ├── pairs.json
│   ├── BTC_USDT-1m.feather
│   ├── BTC_USDT-5m.feather
│   ├── ETH_USDT-1m.feather
│   └── ...
├── kraken/
│   ├── pairs.json
│   └── ...
└── ...

Troubleshooting​

Permission Errors​

If your configuration directory user_data was made by docker, you may get permission errors:

cp: cannot create regular file 'user_data/data/binance/pairs.json': Permission denied

Fix permissions:

sudo chown -R $UID:$GID user_data

Common Issues​

Download fails:

  • Check internet connection
  • Verify exchange name is correct
  • Ensure API limits aren't exceeded
  • Check if pairs exist on the exchange

Large memory usage:

  • Use compressed formats (jsongz)
  • Download smaller time ranges
  • Use fewer pairs
  • Consider using parquet format

Slow downloads:

  • Use multiple timeframes in single command
  • Check exchange rate limits
  • Consider using different data format
  • Verify network speed

Best Practices​

Efficient Data Management​

  1. Start Small: Download a few pairs and short timerange first
  2. Incremental Updates: Use relative timeranges for regular updates
  3. Format Selection: Use feather for best performance
  4. Regular Cleanup: Remove unused data periodically
  5. Backup Important Data: Keep copies of critical datasets

Download Strategy​

  1. Initial Download: Get 1-2 years of data for main pairs
  2. Regular Updates: Daily/weekly incremental downloads
  3. New Pairs: Use --new-pairs-days for consistency
  4. Multiple Timeframes: Download all needed timeframes together

Storage Optimization​

  1. Use Compression: jsongz for archival storage
  2. Remove Unused Data: Clean up old or unused pairs
  3. Monitor Disk Space: Large datasets can consume significant space
  4. Consider Cloud Storage: For backup and sharing

Next Steps​

After downloading data:

  1. Backtesting - Test strategies with historical data
  2. Hyperopt - Optimize strategy parameters
  3. Data Analysis - Analyze market data
  4. Strategy Development - Create trading strategies

Resources​

Last updated on by huojiabin