Reinforcement Learning for Forex Trading: A Deep Dive into PPO

1. Introduction: Why RL for Trading?

Reinforcement Learning (RL) offers a fundamentally different approach to algorithmic trading compared to traditional supervised learning. Instead of predicting price movements directly, RL agents learn optimal trading policies through trial-and-error interaction with the market environment.

In this deep dive, we analyze a complete forex trading implementation using Proximal Policy Optimization (PPO) — one of the most robust and widely-used RL algorithms. We'll explore:

• Custom trading environment design
• Action space with discrete SL/TP combinations
• Technical indicator feature engineering
• Training and testing pipelines

2. The PPO Algorithm

PPO, introduced by OpenAI in 2017, has become the go-to algorithm for many RL applications due to its stability and sample efficiency. It's an Actor-Critic method that:

from stable_baselines3 import PPO
from stable_baselines3.common.vec_env import DummyVecEnv

# Define RL model (PPO)
model = PPO(
    policy="MlpPolicy",      # Multi-layer perceptron
    env=vec_env,
    verbose=1,
    tensorboard_log="./tensorboard_log/"
)

# Train the model
model.learn(total_timesteps=10100)
model.save("model_eurusd")

We use Stable-Baselines3's implementation with an MlpPolicy (neural network with two hidden layers of 64 units each by default). TensorBoard logging enables visualization of training metrics like policy loss and value loss.

3. Trading Environment Design

The heart of any RL trading system is the environment. Our ForexTradingEnv extends OpenAI Gym and defines how the agent interacts with market data.

# Action 0 => No Trade
# Then for direction in [0=short, 1=long] and each sl, tp
self.action_map = [(None, None, None)]  # no trade

for direction in [0, 1]:  # 0=short, 1=long
    for sl in self.sl_options:      # [30, 60, 80]
        for tp in self.tp_options:  # [30, 60, 80]
            self.action_map.append((direction, sl, tp))

self.action_space = spaces.Discrete(len(self.action_map))

4. Reward Function Design

The reward function is critical for shaping agent behavior. This implementation uses a single-bar P&L approach — calculating profit/loss based on the next bar's price action relative to the chosen SL/TP levels.

# Convert pips to price distance
pip_value = 0.0001
sl_price_distance = sl * pip_value  # e.g., 60 pips = 0.006
tp_price_distance = tp * pip_value

# For LONG trades
if direction == 1:
    stop_loss = entry_price - sl_price_distance
    take_profit = entry_price + tp_price_distance
    
    if next_low <= stop_loss:
        pnl = -sl_price_distance  # Hit stop loss
    elif next_high >= take_profit:
        pnl = tp_price_distance   # Hit take profit
    else:
        pnl = next_close - entry_price  # Partial move

# Reward in pips (multiply by 10,000)
reward = pnl * 10000

5. Feature Engineering: Technical Indicators

Raw price data alone is often insufficient for RL models. We enrich the observation space with technical indicators using the pandas-ta library:

import pandas_ta as ta

def load_and_preprocess_data(csv_path: str) -> pd.DataFrame:
    df = pd.read_csv(csv_path, parse_dates=True, index_col='Gmt time')
    df.sort_index(inplace=True)
    
    # Technical indicators
    df['rsi_14'] = ta.rsi(df['Close'], length=14)
    df['ma_20'] = ta.sma(df['Close'], length=20)
    df['ma_50'] = ta.sma(df['Close'], length=50)
    df['atr'] = ta.atr(df['High'], df['Low'], df['Close'], length=14)
    df['ma_20_slope'] = df['ma_20'].diff()
    
    df.dropna(inplace=True)
    return df

6. Training Pipeline

The training script orchestrates the entire learning process:

7. Testing & Evaluation

The test script evaluates the trained model on unseen data — a critical step for validating generalization:

# Load DIFFERENT test data (out-of-sample)
test_df = load_and_preprocess_data(
    "data/test_EURUSD_Candlestick_1_Hour_BID_20.02.2023-22.02.2025.csv"
)

# Load trained model
model = PPO.load("model_eurusd", env=vec_test_env)

# Run deterministic evaluation
while not done:
    action, _states = model.predict(obs, deterministic=True)
    obs, rewards, dones, info = vec_test_env.step(action)
    
    # Log trades and equity
    trade_info = vec_test_env.get_attr("last_trade_info")[0]
    trade_history.append(trade_info)
    equity_curve.append(vec_test_env.get_attr("equity")[0])

# Save results
trades_df.to_csv("trade_history_output.csv")

Key outputs include:

• Equity Curve: Visual representation of cumulative P&L
• Trade History CSV: Entry/exit prices and per-trade P&L

8. Key Insights & Takeaways

9. Conclusion

This implementation demonstrates a complete RL trading system using PPO for forex markets. While there's room for improvement in terms of realism (slippage, costs, position sizing), the core architecture provides a solid foundation for experimentation.

The key takeaway: RL for trading works best when you carefully design the action space, reward function, and observation features to match real trading constraints. The discrete SL/TP approach used here is particularly elegant because it forces the agent to commit to a risk-reward ratio at trade entry.