Commodity Trading Goes Back To The Future

The combination of unprecedented political uncertainty, trade wars, and rapidly evolving technologies is making commodity markets almost as unpredictable as they were during the financial crisis. But the chances of repeating the industry’s most profitable year to date are remote.

Black and grey swan events will continue to result in intermittent spikes in volatility. But these will only provide temporary relief from the relentless erosion in trading margins that started in 2014. In fact, we estimate margins could likely decline by at least another 15 percent over the next five years as commodity markets become more stable and more transparent and competition becomes more intense. (See Exhibit 1.)

This cutthroat environment will weed out the players that continue to follow the tactics of the past from those pioneering new trading strategies by investing in advanced predictive analytics to develop proprietary digital intelligence. As the amount of new digital data available continues to soar, the top players will return to their roots and rely more on developing information advantages from which they can increase their profits. The traders who develop the operating model to support these analytics will succeed and grow, while smaller players that can’t afford to invest in new digital capabilities will be forced to retrench. 

COMMODITY TRADING MARGIN MELTDOWN 

Signs of the coming dramatic shakeout that will result from the commodity trading margin squeeze are already at hand. Trading margins have fallen by more than 20 percent from their recent peak in 2015.

This trading margin meltdown will continue as commodity markets become more mature, stable, and liquid. Over the past decade, the volume of commodity contracts traded nearly tripled, and the value of contracts traded on standard electronic platforms doubled. Commodity market data is also increasingly available and widely distributed, as greater numbers of players sell information and provide services to commodity traders. These new sources of data allow commodity traders to estimate more precisely events that impact their trading strategies, such as  when commodities will arrive at a specific destination and when local stockpiles will be high or low. 

The combination of increased transparency and gluts in almost every commodity should keep volatility in the relatively tight band it has been confined to since 2012. (See Exhibit 2.)

This means margin will continue to be taken away from traders. Major commodity producers and consumers such as national oil companies and miners will charge higher premiums and claim more margin as they expand their global reach and become more sophisticated market participants. Simultaneously, physical infrastructureservice providers and new online platforms will impinge on traders’ traditional roles. These players are making traders less essential by removing bottlenecks to correct supply imbalances and connecting more commodity producers and consumers directly.

EXHIBIT 1: SHRINKING COMMODITY TRADING MARGINS

^{Source: Oliver Wyman proprietary data and analysis} 

Exhibit 2: MINIMAL COMMODITY PRICE VOLATILITY 

So traders are forced to do more and more to gain an edge by further increasing their scale. Top players are snapping up multibilliondollar commodity assets to protect their positions, bulking up their massive trading portfolios. Medium-size traders are shutting down noncore businesses. Many small traders are pulling out of the commodity trading business entirely.

PROPRIETARY DIGITAL INTELLIGENCE

But the most important driver of this shakeout is trading giants’ investments in predictive analytics, which are providing them with significant information advantages.

Large commodity traders once again consider proprietary intelligence critical. When commodity trading firms as we recognize them today were first established in the 1970s and 1980s, top traders relied primarily on privileged information to outperform their peers. More recently, players have tried to gain an edge by focusing on ways that advanced technologies can improve their efficiency primarily through automation.

Now, commodity traders are going back to the future. Instead of relying on extensive traditional information networks to gain advantages from proprietary data, traders who are large enough to invest in the sophisticated systems and dedicated teams required to compete are focusing on how to use predictive analytics to draw valuable proprietary insights from common data sources. (See Exhibit 3.)

Exhibit 3: THE RISING IMPORTANCE OF COMMODITY TRADING PROPRIETARY INTELLIGENCE 

Today’s proprietary intelligence is obtained from a combination of more widely available data sources, such as sensors, geospatial satellites, and even social media run through advanced algorithms developed in-house by data scientists. These algorithms are run on hardware with exponentially greater processing power, to rapidly translate and interpret the huge volumes of data, enabling traders to formulate unique market views. Then, the results are monetized across major trader firms’ expansive global trading portfolios. Large trading volumes are essential to ensure returns from advanced analytics. Without them, traders cannot generate enough value to cover their development costs.

As commodity markets become more liquid and accessible, commodity traders are relying more and more on algorithmic trading, coupling predictive analytics with robotic trade execution. Traders are improving their ability to hedge and speculate by developing code that more nimbly identifies trades and executes them across a broader set of tradable instruments.

But this new digital approach to developing proprietary intelligence requires completely revamping traders’ operating models. Commodity traders need to be willing and able to experiment in order to be players in the new and evolving world of predictive analytics. At a time when trading margins are razor thin, top traders often need to invest millions of dollars in multiple, large pilot programs to discover two or three viable new trading strategies. It is often unclear if anticipated relationships between data feeds and commodity prices actually exist; and even if they do, it is not certain the volume of data is sufficient to make meaningful predictions. For example, it is incredibly difficult to analyze global satellite imagery to identify precisely the daily flow of commodities, given the frequency with which images are being taken. Depending on the specific market, these signals are often also relatively limited compared to just market sentiment when forecasting in the horizon of interest.

More broadly, commodity traders need to embrace new ways of working. Developing topnotch predictive analytics often requires setting up data science teams in a way that they can operate differently from traditional commodity trading organizations, while remaining deeply linked. So commodity traders are establishing incubators, entering joint ventures, and striking partnerships with consultancies and universities. In some cases, they are even pioneering ways that they can monetize proprietary intelligence outside of their own trading and sales activities. 

Simultaneously, existing teams of traders, quantitative analysts, and developers must work more nimbly to make the most of shortlived information advantages. Until now, traders and quantitative analysts identified key metrics for trading strategies and then asked developers to provide them with forecasting tools over a period of weeks or months.

Moving forward, rather than spending time transforming spreadsheets into code, quants and data scientists must concurrently work in code and draw from shared data lakes to transform insights into pilots and proofs of concepts for trading strategies. To predict market behavior, they must use predictive analytics techniques, ranging from single factor analyses to more complex techniques such as convolutional neural networks. And quantitative analysts need to become more familiar with data science techniques, tools, and ways of working.

DIGITAL TALENT GAP 

Commodity traders must also find new ways to attract and keep talent in an increasingly digital workplace. Commercially driven quantitative traders have worked in commodity trading organizations for decades. But to gain a competitive edge, commodity traders increasingly need data scientists in addition to their usual quant teams to transform much more complex and varied digital data feeds into trading strategies.

The challenge is that dedicated data scientists are universally in high demand. Generally, these digital data problem solvers prefer to work for prestigious tech giants and high-paying hedge funds rather than commodity traders. So commodity traders need to recalibrate their recruiting strategies to attract data scientists primarily by interesting them in pioneering digital solutions to seemingly impossible complex problems. After that, commodity traders need to make sure there is a steady flow of such challenges to retain their interest.

Commodity traders must also revamp their organizations so that the expertise of their data scientists can be spread broadly. For example, commodity traders should set up core teams of purely data scientists in data labs to tackle their toughest challenges and empower them to bring in external supplementary talent when necessary. However, this team will not be as commercially driven as traders and analysts. As a result, in order to transform theoretical solutions into the practical profits, traders also need a deep “bench” of quants with commodity trading expertise that are fluent in the language and application of data science to partner with and be guided by a core team of data scientists.

These data science-infused quants should be deployed to work on developing or monetizing cutting-edge analytics with the data scientists. 

Much as banks, law firms, and consultancies recruit and train generalists, traders will hire quants for their raw capabilities and then assign them to the task where they display the greatest aptitude. The growing importance of these generalist quants will require a cultural shift for trading companies, which will need to hire them at an entry level and have their bench become part of the standard rotations of newly hired college graduates.

Members of the core team of data scientists will also have to be moved through different parts of the organization so that they can interact with the larger group of existing top quantitative analysts and traders. They can then mentor others who are just starting out on what will eventually become a broader shift for the industry, from developing analyses in spreadsheets to working them out in code. Once this happens, analysts with coding capabilities will be able to abstract and reuse solutions to improve traders’ strategies across the board or even move themselves into the front office, trading on the back of their models.

Exhibit 4: THE NEW COMMODITY TRADING LANDSCAPE 

The return of the importance of proprietary market intelligence will force all commodity traders to adopt new strategies

^{Source: Oliver Wyman analysis} 

RETURN TO THE CORE 

Only the largest traders can currently invest in predictive analytics because they alone have the scale to monetize results profitably and the capital to invest in sophisticated systems and dedicated teams. This gives them a significant edge over smaller traders and will result in even more margin becoming concentrated in major players. Consolidation will accelerate, at least until the cost of advanced analytics decreases. Top traders will continue to gain an even larger share of the industry’s profits by taking advantage of predictive analytics, while forcing other players to focus even more on only their core competitive advantages. (See Exhibit 4.)

To compete, commodity producers, national oil companies, large commodity consumers, and midsize traders will need to reassess their strengths and invest in them. They will have to be sure to optimize any advantages they have in terms of their access to assets, equity flows, proprietary data, niche trading capabilities, or differentiated business models. Advanced analytics may even enable players with sufficient scale in their niche to strengthen and defend their position. However, to do so, many may need to enter joint ventures or partnerships to achieve sufficient scale to monetize their advantage. At the same time, these traders will have to continue to lower costs and investigate operating and commercial models that give them an even bigger bang for the buck.

That means midsize players must shed noncore businesses, strike alliances to gain access to larger portfolios, and automate back-office functions. By relying more on robots to conduct basic tasks such as reconciling data and blockchain to manage intercompany paperwork exchanges, some commodity traders have lowered their costs by as much as 30 percent. But it’s clear from the past several years that, like other industries, commodity traders cannot simply cost cut their way to success.

THE PATH FORWARD 

Instead, traders need to make maximizing the potential of information advantages their top priority. Previously unthinkable digital capabilities will determine who will be the industry’s leaders in the long term.