Commodity future carry#

The category group contains daily indicators of commodity futures’ carry, based on the price ratio between the first and second front future contracts and specific rolling rules. For commodities with seasonal supply and demand fluctuations and significant storage costs, this carry has a seasonal pattern during the year and seasonally-adjusted carry indicators are estimated.

Nominal commodity future carry#

Ticker: COCRY_NSA / COCRY_VT10

Label: Nominal commodity future carry: % annualized / % annualized for 10% volatility target.

Definition: Estimated carry on continuously rolling futures, based on difference between front and back contracts: % annualized of notional of the contract / % annualized of risk capital on position scaled to 10% (annualized) volatility target.

Notes:

  • The nominal carry is equivalent to the return earned by going long in the front contract and going short in the back contract for an unchanged futures curve. The carry is annualised using the ratio of total business days in a year to the difference in the days to maturity between the two contracts. Thus, the carry metric accounts for differences in the number of trading days between front and second contracts.

  • For volatility-targeted carry, positions are scaled to a 10% vol target based on historic standard deviation of the commodity future returns for an exponential moving average with a half-life of 11 days. Positions are rebalanced at the end of each month.

  • We construct a continuously rolling future series directly from the individual contracts that are part of the regular trading cycle for that commodity. The return calculation assumes that the future position is rolled (from front to second) on the first day of the month when the old front contract becomes deliverable. Not all commodities have monthly contracts.

  • For information on the components comprising each commodity group, see Appendix 1.

Nominal seasonally-adjusted commodity future carry#

Ticker: COCRY_SA / COCRY_SAVT10

Label: Nominal seasonally-adjusted commodity future carry: % annualized / % annualized for 10% volatility target.

Definition: Seasonally-adjusted carry estimated from nominal unadjusted carry, based on point-in-time estimates of seasonal factors: annualized of notional of the contract / % annualized of risk capital on position scaled to 10% (annualized) volatility target.

Notes:

  • The purpose of seasonal adjustment is to remove regular seasonal price differences from carry calculations, because those do not plausibly reflecte premia, such as those arising from convenience yields and hedging pressure.

  • JPMaQS applies additive seasonal adjustment to nominal carry at each point in time based on the concurrently available vintage. The seasonal adjustment uses the U.S. Census Bureau’s X-13 seasonal adjustment tools.

  • In order to estimate seasonal factors, nominal carry is downsampled from trading daily to monthly frequency. For all the months in the vintage bar the last one, we use the average monthly value of the carry as basis of seasonal adjustment. Given that JPMaQS carry metrics always assume contract roll at the end of the month the settlement date of the underlying contract is equal across the month and clearly identified. The seasonal component that is estimated for each point in time is then subtracted from the daily carry values that are based on future prices. Although it is applied daily, the seasonal factor only changes monthly in accordance with changes ins settlement dates.

  • For commodities whose settlement dates do not change regularly at monthly frequency but remain unchanged for a few months we still estimate monthly carry adjustment factors. Although the season of the settlement dates does not always change on these occasions, the seasonal factor may still depend on the time distance to the settlement date, affecting marketing uncertianty about demand and supply conditions.

  • JPMaQS requires a minimum of 42 months for out-of sample seasonal adjustment. This implies that the initial 3 years and half of seasonally adjusted carry are estimated in sample and have a lower grading.

Real commodity future carry#

Ticker: COCRR_NSA / COCRR_VT10

Label: Real commodity future carry: % annualized / % annualized for 10% volatility target.

Definition: Estimated real carry on continuously rolling futures, based on difference between front and back contracts: % annualized of notional of the contract / % annualized of risk capital on position scaled to 10% (annualized) volatility target.

Notes:

  • Real carry means that the conventional (nominal) carry is increased by expected U.S. inflation over the period between first and second contract. If inflation is expected to be positive between the two dates, the real (inflation-adjusted) difference between the first and second contract is higher than the nominal one.

  • The inflation expectation proxy is the 1-year ahead estimated inflation expectation according to Macrosynergy methodology. See notes on category tickers INFE1Y_JA on the page “Inflation expectations (Macrosynergy methodology)”.

  • See the important notes on ‘nominal commodity future carry’ above.

Real seasonally-adjusted commodity future carry#

Ticker: COCRR_SA / COCRR_SAVT10

Label: Real seasonally-adjusted commodity future carry: % annualized / % annualized for 10% volatility target.

Definition: Estimated real carry on continuously rolling futures, based on the seasonally-adjusted difference between front and back contracts: % annualized of notional of the contract / % annualized of risk capital on position scaled to 10% (annualized) volatility target.

Notes:

  • The purpose of seasonal adjustment is to remove regular seasonal price differences from carry calculations, because those do not plausibly reflect premia, such as those arising from convenience yields and hedging pressure.

  • We apply the seasonal and inflation adjustment procedures described above.

Imports#

Only the standard Python data science packages and the specialized macrosynergy package are needed.

import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import math

import json
import yaml

import macrosynergy.management as msm
import macrosynergy.panel as msp
import macrosynergy.signal as mss
import macrosynergy.pnl as msn


from macrosynergy.download import JPMaQSDownload

from timeit import default_timer as timer
from datetime import timedelta, date, datetime

import warnings

warnings.simplefilter("ignore")

The JPMaQS indicators we consider are downloaded using the J.P. Morgan Dataquery API interface within the macrosynergy package. This is done by specifying ticker strings, formed by appending an indicator category code <category> to a currency area code <cross_section>. These constitute the main part of a full quantamental indicator ticker, taking the form DB(JPMAQS,<cross_section>_<category>,<info>), where <info> denotes the time series of information for the given cross-section and category. The following types of information are available:

  • value giving the latest available values for the indicator

  • eop_lag referring to days elapsed since the end of the observation period

  • mop_lag referring to the number of days elapsed since the mean observation period

  • grade denoting a grade of the observation, giving a metric of real time information quality.

After instantiating the JPMaQSDownload class within the macrosynergy.download module, one can use the download(tickers,start_date,metrics) method to easily download the necessary data, where tickers is an array of ticker strings, start_date is the first collection date to be considered and metrics is an array comprising the times series information to be downloaded.

cids_nfm = ["GLD", "SIV", "PAL", "PLT"]
cids_fme = ["ALM", "CPR", "LED", "NIC", "TIN", "ZNC"]
cids_ene = ["BRT", "WTI", "NGS", "GSO", "HOL"]
cids_sta = ["COR", "WHT", "SOY", "CTN"]
cids_liv = ["CAT", "HOG"]
cids_mis = ["CFE", "SGR", "NJO", "CLB"]

cids = cids_nfm + cids_fme + cids_ene + cids_sta + cids_liv + cids_mis
cids_ns = cids_nfm + cids_fme + ["BRT", "WTI", "CFE"]  # presumed non-seasonal commodities
cids_s = list(set(cids) - set(cids_ns + ["NGS"]))  # presumed seasonal commodities
main = ["COCRY_NSA", "COCRY_SA", "COCRY_VT10", "COCRY_SAVT10", 
        "COCRR_NSA", "COCRR_VT10", "COCRR_SA", "COCRR_SAVT10"]
econ = []
mark = ["COXR_NSA", "COXR_VT10"]

xcats = main + econ + mark
# Download series from J.P. Morgan DataQuery by tickers

start_date = "1990-01-01"
tickers = [cid + "_" + xcat for cid in cids for xcat in xcats]
print(f"Maximum number of tickers is {len(tickers)}")

# Retrieve credentials

client_id: str = os.getenv("DQ_CLIENT_ID")
client_secret: str = os.getenv("DQ_CLIENT_SECRET")

# Download from DataQuery

with JPMaQSDownload(client_id=client_id, client_secret=client_secret) as downloader:
    start = timer()
    df = downloader.download(
        tickers=tickers,
        start_date=start_date,
        metrics=["value", "eop_lag", "mop_lag", "grading"],
        suppress_warning=True,
        show_progress=True,
    )
    end = timer()

dfd = df

print("Download time from DQ: " + str(timedelta(seconds=end - start)))
Maximum number of tickers is 250
Downloading data from JPMaQS.
Timestamp UTC:  2023-11-08 12:13:30
Connection successful!
Number of expressions requested: 1000
Requesting data: 100%|██████████| 50/50 [00:16<00:00,  3.11it/s]
Downloading data: 100%|██████████| 50/50 [00:15<00:00,  3.20it/s]
Download time from DQ: 0:00:52.046036

Availability#

cids_exp = cids
msm.missing_in_df(dfd, xcats=main, cids=cids_exp)
Missing xcats across df:  []
Missing cids for COCRR_NSA:  []
Missing cids for COCRR_SA:  []
Missing cids for COCRR_SAVT10:  []
Missing cids for COCRR_VT10:  []
Missing cids for COCRY_NSA:  []
Missing cids for COCRY_SA:  []
Missing cids for COCRY_SAVT10:  []
Missing cids for COCRY_VT10:  []

Commodity carry data mostly date back to the mid-1990s. Real carry data is only available from 2000 onwards.

For the explanation of currency symbols, which are related to currency areas or countries for which categories are available, please view Appendix 1.

xcatx = main
cidx = cids_exp

dfx = msm.reduce_df(dfd, xcats=xcatx, cids=cidx)
dfs = msm.check_startyears(
    dfx,
)
msm.visual_paneldates(dfs, size=(20, 5))

print("Last updated:", date.today())
../_images/Commodity future carry_20_0.png
Last updated: 2023-11-08
plot = msm.check_availability(
    dfd, xcats=main, cids=cids_exp, start_size=(20, 5), start_years=False
)
../_images/Commodity future carry_21_0.png
plot = msp.heatmap_grades(
    dfd,
    xcats=main,
    cids=cids_exp,
    size=(19, 5),
    title=f"Average vintage grades from {start_date} onwards",
)
../_images/Commodity future carry_22_0.png
xcatx = main
cidx = cids_exp
msp.view_ranges(
    dfd,
    xcats=xcatx,
    cids=cidx,
    val="eop_lag",
    title="End of observation period lags (ranges of time elapsed since end of observation period in days)",
    start="2000-01-01",
    kind="box",
    size=(16, 4),
)
msp.view_ranges(
    dfd,
    xcats=xcatx,
    cids=cidx,
    val="mop_lag",
    title="Median of observation period lags (ranges of time elapsed since middle of observation period in days)",
    start="2000-01-01",
    kind="box",
    size=(16, 4),
)
../_images/Commodity future carry_23_0.png ../_images/Commodity future carry_23_1.png

History#

Nominal commodity carry#

Unlike financial assets, commodities have significant storage costs. This means that intertemporal substitution can be expensive and dependent on storage capacity. All other things equal, the costlier the storage, the greater is the segmentation and the variability of carry. This has consequences for carry dynamics.

  • Commodities with seasonal supply or demand fluctuations often display a seasonal pattern in recorded carry. This can be seen clearly in livestock and gasoline contracts.

  • There can be huge outliers in carry if storage limits are reached. These can be seen in many commodities.

  • Commodities with relatively low storage costs, such as gold and silver, tend to have the most stable carry since substitution between futures dates is inexpensive.

xcatx = ["COCRY_NSA", "COCRY_SA"]
cidx = cids_exp

msp.view_ranges(
    dfd,
    xcats=xcatx,
    cids=cidx,
    sort_cids_by="mean",
    start="2000-01-01",
    title="Means and standard deviations of nominal commodity carry, since 2000",
    kind="bar",
    size=(16, 8),
    xcat_labels=['Outright', 'Seasonally adjusted']
)
../_images/Commodity future carry_27_0.png
xcatx = ["COCRY_NSA", "COCRY_SA"]
cidx = cids_exp

msp.view_timelines(
    dfd,
    xcats=xcatx,
    cids=cidx,
    start="2000-01-01",
    title="Nominal commodity carry, outright and seasonally adjusted, since 2000",
    title_adj=1.03,
    title_xadj=0.52,
    title_fontsize=27,
    cumsum=False,
    ncol=4,
    same_y=False,
    size=(12, 7),
    aspect=1.5,
    all_xticks=True,
    xcat_labels=['Outright', 'Seasonally adjusted'],
)
../_images/Commodity future carry_28_0.png

Standard seasonal adjustment of carry does not remove all seasonal influences. It typically removes or at least mitigates the directional bias of carry provided seasonal influences of price differences across delivery dates do not have strong trends of their own. However, seasonal adjustment cannot prevent seasonal increases in volatility and absolute carry values, which may arise from weather and harvest conditions.

xcatx = ["COCRY_NSA", "COCRY_SA"]
cidx = ["CAT", "GSO", "SGR", "WHT"]

msp.view_timelines(
    dfd,
    xcats=xcatx,
    cids=cidx,
    start="2000-01-01",
    title="Carry of selected seasonal commodities, outright and seasonally adjusted",
    title_adj=1.03,
    title_xadj=0.52,
    title_fontsize=15,
    cumsum=False,
    ncol=2,
    same_y=False,
    size=(12, 7),
    aspect=1.5,
    all_xticks=True,
    xcat_labels=['Outright', 'Seasonally adjusted'],
)
../_images/Commodity future carry_30_0.png

Overall carry is not strongly correlated across commodities. There is groupwise positive correlation in the metals and energy groups.

msp.correl_matrix(
    dfd,
    xcats="COCRY_SA",
    cids=cids_exp,
    title="Cross-sectional correlations of nominal commodity carry, since 2000",
    cluster=True,
    size=(20, 14),
)
../_images/Commodity future carry_32_0.png

Real commodity carry#

Real carry is equal to nominal carry plus expected U.S. inflation. Thus, it is on average a little bit higher, but otherwise shows a very similar dynamic.

xcatx = ["COCRR_SA", "COCRR_NSA"]
cidx = cids_exp

msp.view_ranges(
    dfd,
    xcats=xcatx,
    cids=cidx,
    sort_cids_by="mean",
    start="2000-01-01",
    title="Means and standard deviations of commodity carry, since 2000",
    xcat_labels=["Nominal", "Real"],
    kind="bar",
    size=(16, 8),
)
../_images/Commodity future carry_35_0.png
xcatx = ["COCRR_NSA", "COCRR_SA"]
cidx = cids_s

msp.view_timelines(
    dfd,
    xcats=xcatx,
    cids=cidx,
    start="2000-01-01",
    title="Real carry of seasonal commodities, outright and with seasonal adjustment",
    title_adj=1.03,
    title_fontsize=27,
    title_xadj=0.47,
    legend_fontsize=17,
    cumsum=False,
    ncol=3,
    same_y=False,
    size=(12, 7),
    aspect=1.7,
    all_xticks=True,
    xcat_labels=['Outright', 'Seasonally adjusted']
)
../_images/Commodity future carry_36_0.png

Vol-targeted commodity carry#

Even when positions are adjusted for return volatility, differences in carry variations are vast. Lean hogs, cattle, gasoline and soy have posted the widest standard deviations. Seasonality significantly contributed to heteroscedasticity across contracts’ carry.

xcatx = ["COCRY_VT10", "COCRY_SAVT10"]
cidx = cids_exp

msp.view_ranges(
    dfd,
    xcats=xcatx,
    cids=cidx,
    sort_cids_by="mean",
    start="2000-01-01",
    kind="bar",
    title="Means and standard deviations of nominal commodity carry, at 10% vol-target, since 2000",
    size=(16, 8),
    xcat_labels=['Outright', 'Seasonally adjusted']
)
../_images/Commodity future carry_39_0.png
xcatx = ["COCRY_NSA", "COCRY_VT10"]
cidx = cids_exp

msp.view_timelines(
    dfd,
    xcats=xcatx,
    cids=cidx,
    start="2000-01-01",
    title="Nominal commodity carry, without and with volatility targeting",
    title_adj=1.03,
    title_fontsize=27,
    title_xadj=0.47,
    legend_fontsize=17,
    cumsum=False,
    ncol=4,
    same_y=False,
    size=(12, 7),
    aspect=1.7,
    all_xticks=True,
    xcat_labels=['Outright', '10% vol-target']
)
../_images/Commodity future carry_40_0.png

Importance#

Empirical clues#

Real commodity carry, when adjusted for seasonal effects and intertemporal segmentation, often reflects a convenience yield paid to efficient financial investors by producers and/or consumers of a commodity. For commodities with little or no seasonal fluctuations (precious and ferrous metals, coffee and international crude), there has been a significant positive relation between carry and subsequent monthly and quartery returns.

cidx = cids_ns

cr = msp.CategoryRelations(
    dfd,
    xcats=["COCRR_NSA", "COXR_NSA"],
    cids=cidx,
    freq="Q",
    lag=1,
    xcat_aggs=["mean", "sum"],
    fwin=1,
    start="2000-01-01",
)
cr.reg_scatter(
    title="Real commodity carry and subsequent future returns (non-seasonal products)",
    labels=False,
    coef_box="lower right",
    xlab="Real commodity carry, % annualized, quarterly means",
    ylab="Subsequent quarterly commodity future returns",
    prob_est="map",
)
../_images/Commodity future carry_48_0.png

For seasonal commodities, i.e. , agricultural commodities, livestock and regional energy contracts, the relation there has not been a signficant predictive relation between (unadjusted) future curry and subsequent returns.

cidx = cids_s

cr = msp.CategoryRelations(
    dfd,
    xcats=["COCRR_NSA", "COXR_NSA"],
    cids=cidx,
    freq="Q",
    lag=1,
    xcat_aggs=["mean", "sum"],
    fwin=1,
    start="2000-01-01",
)
cr.reg_scatter(
    title="Seasonal commodities: unadjusted commodity carry and subsequent future returns",
    labels=False,
    coef_box="lower right",
    xlab="Real commodity carry, % annualized, quarterly means",
    ylab="Subsequent quarterly commodity future returns",
    prob_est="map",
)
../_images/Commodity future carry_50_0.png

However, using seasonally-adjusted carry, there has a stronger predictive relation between carry and return, even if it is not statistically significant. Lack of significance in this simple setting probably reflects that across the panel carry-return relations are very different and that without volatility adjustments the most volatile contracts dominate the results.

cidx = cids_s

cr = msp.CategoryRelations(
    dfd,
    xcats=["COCRR_SA", "COXR_NSA"],
    cids=cidx,
    freq="Q",
    lag=1,
    xcat_aggs=["mean", "sum"],
    fwin=1,
    start="2000-01-01",
)
cr.reg_scatter(
    title="Seasonal commodities: seasonally-adjusted commodity carry and subsequent future returns",
    labels=False,
    coef_box="lower right",
    xlab="Commodity carry, % annualized, quarterly means",
    ylab="Subsequent quarterly commodity future returns",
    prob_est="map",
)
../_images/Commodity future carry_52_0.png

The combination of seasonal adjustment and regular volatility adjustment reveals a clear and positive relation between carry and returns even for the seasonal commodities at a monthly or quarterly frequency.

cidx = cids_s

cr = msp.CategoryRelations(
    dfd,
    xcats=["COCRR_SAVT10", "COXR_VT10"],
    cids=cidx,
    freq="Q",
    lag=1,
    xcat_aggs=["mean", "sum"],
    fwin=1,
    start="2000-01-01",
)
cr.reg_scatter(
    title="Seasonal commodities: seasonally-adjusted vol-targeted commodity carry and subsequent future returns",
    labels=False,
    coef_box="lower right",
    xlab="Real commodity carry, seasonally adjusted and vol-targeted, % annualized, quarterly means",
    ylab="Subsequent vol-targeted (10%) quarterly commodity returns",
    prob_est="map",
)
../_images/Commodity future carry_54_0.png

Based on seasonally adjusted and volatility-target carry and volatility targeted returns one can detect a highly significant predictive carry-return relation at a monthly or quarterly frequency.

cidx = cids_s + cids_ns

cr = msp.CategoryRelations(
    dfd,
    xcats=["COCRR_SAVT10", "COXR_VT10"],
    cids=cidx,
    freq="Q",
    lag=1,
    xcat_aggs=["mean", "sum"],
    fwin=1,
    start="2000-01-01",
)
cr.reg_scatter(
    title="All commodities: seasonally-adjusted vol-targeted commodity carry and subsequent future returns",
    labels=False,
    coef_box="lower right",
    xlab="Commodity carry, % annualized, quarterly means",
    ylab="Subsequent quarterly commodity future returns",
    prob_est="map",
)
../_images/Commodity future carry_56_0.png

Appendices#

Appendix 1: Commodity group definitions and symbols#

The commodity groups considered are: energy, base metals, precious metals, agricultural commodities and livestock.

  • The energy commodity group contains:

    • BRT : ICE Brent crude

    • WTI : NYMEX WTI light crude

    • NGS : NYMEX natural gas, Henry Hub

    • GSO : NYMEX RBOB Gasoline

    • HOL : NYMEX Heating oil, New York Harbor ULSD

  • The base metals group contains:

    • ALM : London Metal Exchange aluminium

    • CPR : Comex copper

    • LED : London Metal Exchange Lead

    • NIC : London Metal Exchange Nickel

    • TIN : London Metal Exchange Tin

    • ZNC : London Metal Exchange Zinc

  • The precious metals group contains:

    • GLD : COMEX gold 100 Ounce

    • SIV : COMEX silver 5000 Ounce

    • PAL : NYMEX palladium

    • PLT : NYMEX platinum

  • The agricultural commodity group contains:

    • COR : Chicago Board of Trade corn composite

    • WHT : Chicago Board of Trade wheat composite

    • SOY : Chicago Board of Trade soybeans composite

    • CTN : NYBOT / ICE cotton #2

    • CFE : NYBOT / ICE coffee ‘C’ Arabica

    • SGR : NYBOT / ICE raw cane sugar #11

    • NJO : NYBOT / NYCE FCOJ frozen orange juice concentrate

    • CLB : Chicago Mercantile Exchange random length lumber

  • The (U.S.) livestock commodity group contains:

    • CAT : Chicago Mercantile Exchange live cattle composite

    • HOG : Chicago Mercantile Exchange lean hogs composite