Publication Types:

Wind energy forecasting with missing values within a fully conditional specification framework

2024ArticleJournal paper
H. Wen, P. Pinson, J. Gu, Z. Jin
International Journal of Forecasting 40(1), pp. 77-95
Publication year: 2024

Wind power forecasting is essential to power system operation and electricity markets. As abundant data became available thanks to the deployment of measurement infrastructures and the democratization of meteorological modeling, extensive data-driven approaches have been developed within both point and probabilistic forecasting frameworks. These models usually assume that the dataset at hand is complete and overlook missing value issues that often occur in practice. In contrast to that common approach, we rigorously consider here the wind power forecasting problem in the presence of missing values, by jointly accommodating imputation and forecasting tasks. Our approach allows inferring the joint distribution of input features and target variables at the model estimation stage based on incomplete observations only. We place emphasis on a fully conditional specification method owing to its desirable properties, e.g., being assumption-free when it comes to these joint distributions. Then, at the operational forecasting stage, with available features at hand, one can issue forecasts by implicitly imputing all missing entries. The approach is applicable to both point and probabilistic forecasting, while yielding competitive forecast quality within both simulation and real-world case studies. It confirms that by using a powerful universal imputation method based on fully conditional specification, the proposed universal imputation approach is superior to the common impute-then-predict approach, especially in the context of probabilistic forecasting.

Towards replication-robust analytics markets

2024ArticlePreprint
T Falconer, P Pinson, J Kazempour
preprint, under review
Publication year: 2024

Many industries rely on data-driven analytics, yet useful datasets are often distributed amongst market competitors that are reluctant to collaborate and share information. Recent literature proposes analytics markets to provide monetary incentives for data sharing, however many of these market designs are vulnerable to malicious forms of replication — whereby agents replicate their data and act under multiple identities to increase revenue. We develop a replication-robust analytics market, centering on supervised learning for regression. To allocate revenue, we use a Shapley value-based attribution policy, framing the features of agents as players and their interactions as a characteristic function game. We show that there are different ways to describe such a game, each with causal nuances that affect robustness to replication. Our proposal is validated using a real world wind power forecasting case study.

Strategic coalition for data pricing in IoT data markets

2024ArticleJournal paper
S. R. Pandey, P. Pinson, P. Popovski
IEEE Internet of Things Journal 11(4), pp. 6454-6468
Publication year: 2024

This paper establishes a market for trading Internet of Things (IoT) data that is used to train machine learning models. The data, either raw or processed, is supplied to the market platform through a network, and the price of such data is controlled based on the value it brings to the machine learning model under the adversity of the correlation property of data. Eventually, a simplified distributed solution for a data trading mechanism is derived that improves the mutual benefit of devices and the market. Our key proposal is an efficient algorithm for data markets that jointly addresses the challenges of availability and heterogeneity in participation, as well as the transfer of trust and the economic value of data exchange in IoT networks. The proposed approach establishes the data market by reinforcing collaboration opportunities between devices with correlated data to limit information leakage. Therein, we develop a networkwide optimization problem that maximizes the social value of coalition among the IoT devices of similar data types; at the same time, it minimizes the cost due to network externalities, i.e., the impact of information leakage due to data correlation, as well as the opportunity costs. Finally, we reveal the structure of the formulated problem as a distributed coalition game and solve it following the simplified split-and-merge algorithm. Simulation results show the efficacy of our proposed mechanism design toward a trusted IoT data market, with up to 32.72% gain in the average payoff for each seller.

Seamless and multi-resolution energy forecasting

2024ArticleJournal paperPreprint
C. Wang, P. Pinson, Y. Wang
preprint, under review
Publication year: 2024

Forecasting is pivotal in energy systems, by providing fundamentals for operation at different horizons and resolutions. Though energy forecasting has been widely studied for capturing temporal information, very few works concentrate on the frequency information provided by forecasts. They are consequently often limited to single-resolution applications (e.g., hourly). Here, we propose a unified energy forecasting framework based on Laplace transform in the multi-resolution context. The forecasts can be seamlessly produced at different desired resolutions without re-training or post-processing. Case studies on both energy demand and supply data show that the forecasts from our proposed method can provide accurate information in both time and frequency domains. Across the resolutions, the forecasts also demonstrate high consistency. More importantly, we explore the operational effects of our produced forecasts in the day-ahead and intra-day energy scheduling. The relationship between (i) errors in both time and frequency domains and (ii) operational value of the forecasts is analysed. Significant operational benefits are obtained.

Robust generation dispatch with purchase of renewable power and load predictions

2024ArticleIn press/Available onlineJournal paper
R. Xie, P. Pinson, Y. Xu, Y. Chen
IEEE Transactions on Sustainable Energy, in press/available online
Publication year: 2024

The increasing use of renewable energy sources (RESs) and responsive loads has made power systems more uncertain. Meanwhile, thanks to the development of advanced metering and forecasting technologies, predictions by RESs and load owners are now attainable. Many recent studies have revealed that pooling the predictions from RESs and loads can help the operators predict more accurately and make better dispatch decisions. However, how the prediction purchase decisions are made during the dispatch processes needs further investigation. This paper fills the research gap by proposing a novel robust generation dispatch model considering the purchase and use of predictions from RESs and loads. The prediction purchase decisions are made in the first stage, which influence the accuracy of predictions from RESs and loads, and further the uncertainty set and the worst-case second-stage dispatch performance. This two-stage procedure is essentially a robust optimization problem with decision dependent uncertainty (DDU). A mapping-based column-and-constraint generation (C&CG) algorithm is developed to overcome the potential failures of traditional solution methods in detecting feasibility, guaranteeing convergence, and reaching optimal strategies under DDU. Case studies demonstrate the effectiveness, necessity, and scalability of the proposed model and algorithm.

Passenger ferry operations in the digital era: Forecasting and revenue management at Molslinjen

2024ArticleJournal paperPreprint
P. Pinson, M. Bjørn, S. Kristiansen, C.B. Nielsen, L. Janerka, J. Skovgaard, K. Durhuus
INFORMS Journal of Applied Analytics, under review (invited paper - winner of the INFORMS Edelman Award 2024)
Publication year: 2024

Molslinjen, one of the world’s largest operators of fast-moving catamaran ferries, based in Denmark, adopted a strategical focus on digitalization to profoundly change their operations and business practice. They partnered with Halfspace, a data, analytics and AI company based in Copenhagen, Denmark, to support that transition. Halfspace and Molslinjen have jointly developed and deployed a successful forecasting and revenue management toolbox for the data-driven operation of ferries in Denmark, rolled out operationally since 2020. This has resulted in \$2.6-3.2 million yearly savings, significant reduction in number of delayed departures and average delays, and a 3% reduction in fuel costs and emissions. This toolbox relies on some of the latest advances in machine learning for forecasting and in analytics approaches to revenue management. The potential for generalizing to the global ferry industry is significant, with an impact on both revenues and ESG criteria.

On tracking varying bounds when forecasting bounded time series

2024ArticleIn press/Available onlineJournal paper
A. Pierrot, P. Pinson
Technometrics, in press/available online
Publication year: 2024

We consider a new framework where a continuous, though bounded, random variable has unobserved bounds that vary over time. In the context of univariate time series, we look at the bounds as parameters of the distribution of the bounded random variable. We introduce an extended log-likelihood estimation and design algorithms to track the bound through online maximum likelihood estimation. Since the resulting optimization problem is not convex, we make use of recent theoretical results on stochastic quasiconvex optimization, to eventually derive an Online Normalized Gradient Descent algorithm. We illustrate and discuss the workings of our approach based on both simulation studies and a real-world wind power forecasting problem.

On the efficiency of energy markets with non-merchant storage

2024ArticleIn press/Available onlineJournal paper
L. Frölke, E. Prat, P. Pinson, R. M. Lusby, J. Kazempour
Energy Systems, in press/available online
Publication year: 2024

Energy market designs with non-merchant storage have been proposed in recent years, with the aim of achieving optimal integration of storage. In order to handle the time linking constraints that are introduced in such markets, existing works commonly make simplifying assumptions about the end-of-horizon storage level. This work analyses market properties under such assumptions, as well as in their absence. We find that, although they ensure cost recovery for all market participants, these assumptions generally lead to market inefficiencies. Therefore we consider the design of markets with non-merchant storage without such simplifying assumptions. Using an illustrative example, as well as detailed proofs, we provide conditions under which market prices in subsequent market horizons fail to reflect the value of stored energy. We show that this problem is essential to address in order to preserve market efficiency and cost recovery. Finally, we propose a method for restoring these market properties in a perfect-foresight setting.

Fairness by design in shared-energy allocation problems

2024ArticlePreprint
Z Fornier, V Leclėre, P Pinson
preprint, under review
Publication year: 2024

This paper studies how to aggregate prosumers (or large consumers) and their collective decisions in electricity markets, with a focus on fairness. Fairness is essential for prosumers to participate in aggregation schemes. Some prosumers may not be able to access the energy market directly, even though it would be beneficial for them. Therefore, new companies offer to aggregate them and promise to treat them fairly. This leads to a fair resource allocation problem.

We propose to use acceptability constraints to guarantee that each prosumer gains from the aggregation.
Moreover, we aim to distribute the costs and benefits fairly, taking into account the multi-period and uncertain nature of the problem. Rather than using financial mechanisms to adjust for fairness issues, we focus on various objectives and constraints, within decision problems, that achieve fairness by design. We start from a simple single-period and deterministic model, and then generalize it to a dynamic and stochastic setting using, e.g., stochastic dominance constraints.

Bayesian regression markets

2024ArticleJournal paperPreprint
T. Falconer, J. Kazempour, P. Pinson
preprint, under review
Publication year: 2024

Machine learning tasks are vulnerable to the quality of data used as input. Yet, it is often challenging for firms to obtain adequate datasets, with them being naturally distributed amongst owners, that in practice, may be competitors in a downstream market and reluctant to share information. Focusing on supervised learning for regression tasks, we develop a regression market to provide a monetary incentive for data sharing. Our proposed mechanism adopts a Bayesian framework, allowing us to consider a more general class of regression tasks. We present a thorough exploration of the market properties, and show that similar proposals in current literature expose the market agents to sizeable financial risks, which can be mitigated in our probabilistic setting.

A market for trading forecasts: A wagering mechanism

2024ArticleJournal paper
A. Raja, J. Kazempour, P. Pinson, S. Grammatico
International Journal of Forecasting 40(1), pp. 142-159
Publication year: 2024

In many areas of industry and society, e.g., energy, healthcare, logistics, agents collect vast amounts of data that they deem proprietary. These data owners extract predictive information of varying quality and relevance from data depending on quantity, inherent information content and their own technical expertise. Aggregating these data and heterogeneous predictive skills, which are distributed in terms of ownership, can result in a higher collective value for a prediction task. In this paper, we envision a platform for improving predictions via implicit pooling of private information in return for possible remuneration. Specifically, we design a wagering-based forecast elicitation market platform, where a buyer intending to improve their forecasts posts a prediction task, and sellers respond to it with their forecast reports and wagers. This market delivers an aggregated forecast to the buyer (pre event) and allocates a payoff to the sellers (post-event) for their contribution. We propose a payoff mechanism and prove that it satisfies several desirable economic properties, including those specific to electronic platforms. Furthermore, we discuss the properties of the forecast aggregation operator and scoring rules to emphasise their effect on the sellers’ payoff. Finally, we provide numerical examples to illustrate the structure and properties of the proposed market platform.