Wind power forecasting is essential to power system operation and electricity markets. As abundant data became available thanks to the deployment of measurement infrastructures and the democratization of meteorological modeling, extensive data-driven approaches have been developed within both point and probabilistic forecasting frameworks. These models usually assume that the dataset at hand is complete and overlook missing value issues that often occur in practice. In contrast to that common approach, we rigorously consider here the wind power forecasting problem in the presence of missing values, by jointly accommodating imputation and forecasting tasks. Our approach allows inferring the joint distribution of input features and target variables at the model estimation stage based on incomplete observations only. We place emphasis on a fully conditional specification method owing to its desirable properties, e.g., being assumption-free when it comes to these joint distributions. Then, at the operational forecasting stage, with available features at hand, one can issue forecasts by implicitly imputing all missing entries. The approach is applicable to both point and probabilistic forecasting, while yielding competitive forecast quality within both simulation and real-world case studies. It confirms that by using a powerful universal imputation method based on fully conditional specification, the proposed universal imputation approach is superior to the common impute-then-predict approach, especially in the context of probabilistic forecasting.

This paper establishes a market for trading Internet of Things (IoT) data that is used to train machine learning models. The data, either raw or processed, is supplied to the market platform through a network, and the price of such data is controlled based on the value it brings to the machine learning model under the adversity of the correlation property of data. Eventually, a simplified distributed solution for a data trading mechanism is derived that improves the mutual benefit of devices and the market. Our key proposal is an efficient algorithm for data markets that jointly addresses the challenges of availability and heterogeneity in participation, as well as the transfer of trust and the economic value of data exchange in IoT networks. The proposed approach establishes the data market by reinforcing collaboration opportunities between devices with correlated data to limit information leakage. Therein, we develop a networkwide optimization problem that maximizes the social value of coalition among the IoT devices of similar data types; at the same time, it minimizes the cost due to network externalities, i.e., the impact of information leakage due to data correlation, as well as the opportunity costs. Finally, we reveal the structure of the formulated problem as a distributed coalition game and solve it following the simplified split-and-merge algorithm. Simulation results show the efficacy of our proposed mechanism design toward a trusted IoT data market, with up to 32.72% gain in the average payoff for each seller.

The increasing use of renewable energy sources (RESs) and responsive loads has made power systems more uncertain. Meanwhile, thanks to the development of advanced metering and forecasting technologies, predictions by RESs and load owners are now attainable. Many recent studies have revealed that pooling the predictions from RESs and loads can help the operators predict more accurately and make better dispatch decisions. However, how the prediction purchase decisions are made during the dispatch processes needs further investigation. This paper fills the research gap by proposing a novel robust generation dispatch model considering the purchase and use of predictions from RESs and loads. The prediction purchase decisions are made in the first stage, which influence the accuracy of predictions from RESs and loads, and further the uncertainty set and the worst-case second-stage dispatch performance. This two-stage procedure is essentially a robust optimization problem with decision dependent uncertainty (DDU). A mapping-based column-and-constraint generation (C&CG) algorithm is developed to overcome the potential failures of traditional solution methods in detecting feasibility, guaranteeing convergence, and reaching optimal strategies under DDU. Case studies demonstrate the effectiveness, necessity, and scalability of the proposed model and algorithm.

We consider a new framework where a continuous, though bounded, random variable has unobserved bounds that vary over time. In the context of univariate time series, we look at the bounds as parameters of the distribution of the bounded random variable. We introduce an extended log-likelihood estimation and design algorithms to track the bound through online maximum likelihood estimation. Since the resulting optimization problem is not convex, we make use of recent theoretical results on stochastic quasiconvex optimization, to eventually derive an Online Normalized Gradient Descent algorithm. We illustrate and discuss the workings of our approach based on both simulation studies and a real-world wind power forecasting problem.

Forecast combination improves upon the component forecasts. Most often, combination approaches are restricted to the linear setting only. However, theory shows that if the component forecasts are neutrally dispersed—a requirement for probabilistic calibration—linear forecast combination will only increase dispersion and thus lead to miscalibration. Furthermore, the accuracy of the component forecasts may vary over time and the combination weights should vary accordingly, necessitating updates as time progresses. In this paper, we develop an online version of the beta-transformed linear pool, which theoretically can transform the probabilistic forecasts such that they are neutrally dispersed. We show that, in case of stationary synthetic time series, the performance of the developed method converges to that of the optimal combination in hindsight. Moreover, in case of nonstationary real-world time series from a wind farm in mid-west France, the developed model outperforms the optimal combination in hindsight.

The safe and efficient execution of offshore operations requires short-term (1 to 6 hours ahead) high-quality probabilistic forecasts of metocean variables. The development areas for offshore wind projects, potentially in high depths, make it difficult to gather measurement data. This paper explores the use of deep learning for wind speed forecasting at an unobserved offshore location. The proposed convolutional architecture jointly exploits coastal measurements and numerical weather predictions to emulate multivariate probabilistic short-term forecasts. We explore both Gaussian and non-Gaussian neural representations using normalizing flows. We benchmark these approaches with respect to state-of-art data-driven schemes, including analog methods and quantile forecasting. The performance of the models, and resulting forecast quality, are analyzed in terms of probabilistic calibration, probabilistic and deterministic metrics, and as a function of weather situations. We report numerical experiments for a real case-study off the French Mediterranean coast. Our results highlight the role of regional numerical weather prediction and coastal in situ measurement in the performance of the post-processing. For single-valued forecasts, a 40\% decrease in RMSE is observed compared to the direct use of numerical weather predictions. Significant skill improvements are also obtained for the probabilistic forecasts, in terms of various scores, as well as an acceptable probabilistic calibration. The proposed architecture can process a large amount of heterogeneous input data, and offers a versatile probabilistic framework for multivariate forecasting.

Machine learning tasks are vulnerable to the quality of data used as input. Yet, it is often challenging for firms to obtain adequate datasets, with them being naturally distributed amongst owners, that in practice, may be competitors in a downstream market and reluctant to share information. Focusing on supervised learning for regression tasks, we develop a regression market to provide a monetary incentive for data sharing. Our proposed mechanism adopts a Bayesian framework, allowing us to consider a more general class of regression tasks. We present a thorough exploration of the market properties, and show that similar proposals in current literature expose the market agents to sizeable financial risks, which can be mitigated in our probabilistic setting.

In many areas of industry and society, e.g., energy, healthcare, logistics, agents collect vast amounts of data that they deem proprietary. These data owners extract predictive information of varying quality and relevance from data depending on quantity, inherent information content and their own technical expertise. Aggregating these data and heterogeneous predictive skills, which are distributed in terms of ownership, can result in a higher collective value for a prediction task. In this paper, we envision a platform for improving predictions via implicit pooling of private information in return for possible remuneration. Specifically, we design a wagering-based forecast elicitation market platform, where a buyer intending to improve their forecasts posts a prediction task, and sellers respond to it with their forecast reports and wagers. This market delivers an aggregated forecast to the buyer (pre event) and allocates a payoff to the sellers (post-event) for their contribution. We propose a payoff mechanism and prove that it satisfies several desirable economic properties, including those specific to electronic platforms. Furthermore, we discuss the properties of the forecast aggregation operator and scoring rules to emphasise their effect on the sellers’ payoff. Finally, we provide numerical examples to illustrate the structure and properties of the proposed market platform.

Should the organization, design and functioning of electricity markets be taken for granted? Definitely not. While decades of evolution of electricity markets in developed countries made us believe that we may have found the right and future-proof model, the substantially and rapidly evolving context of our power and energy systems is challenging this idea in many ways. Actually, that situation brings both challenges and opportunities. Challenges include accommodation of renewable energy generation, decentralization and support to investment, while opportunities are mainly that advances in technical and social sciences provide us with many more options in terms of future market design. We here take a holistic point of view, by trying to understand where we are coming from with electricity markets and where we may be going. Future electricity markets should be made fit for purpose by considering them as a way to organize and operate a socio-techno-economic system.

As prosumers and energy communities gain prominence in power systems, energy trading between prosumers in local P2P markets is paramount. Within this novel market design, peers can directly exchange energy with each other, leading to economic advantages while supporting the decarbonization of the sector. To ensure that voltage and congestion issues are properly addressed, a thorough coordination between the P2P market and the Distribution System Operator is required. This paper presents and compares three mutual-benefit coordination methods. The first method entails applying product differentiation on an iterative basis to avoid exceeding the lines thermal limits, which is performed through penalties on P2P exchanges that may be overloading the network. The second method uses the P2P market with an AC-OPF, ensuring network operation through a flexibility market via upward and downward flexibility. The last one proposes an integrated operation of the P2P market with AC-OPF. All methods are assessed in a typical distribution network with high prosumers integration. The results show that the second method is the one that, fulfilling the network constraints, presents greater social welfare.

We propose and develop a new algorithm for trading wind energy in electricity markets, within an online learning and optimization framework. In particular, we combine a component-wise adaptive variant of the gradient descent algorithm with recent advances in the feature-driven newsvendor model. This results in an online offering approach capable of leveraging data-rich environments, while adapting to the nonstationary characteristics of energy generation and electricity markets, also with a minimal computational burden. The performance of our approach is analyzed based on several numerical experiments, showing both better adaptability to nonstationary uncertain parameters and significant economic gains.

Permanently increasing penetration of converter-interfaced generation and renewable energy sources (RESs) makes modern electrical power systems more vulnerable to low probability and high impact events, such as extreme weather, which could lead to severe contingencies, even blackouts. These contingencies can be further propagated to neighbouring energy systems over coupling components/technologies and consequently negatively influence the entire multi-energy system (MES) (such as gas, heating and electricity) operation and its resilience. In recent years, machine learning-based techniques (MLBTs) have been intensively applied to solve various power system problems, including system planning, or security and reliability assessment. This paper aims to review MES resilience quantification methods and the application of MLBTs to assess the resilience level of future sustainable energy systems. The open research questions are identified and discussed, whereas the future research directions are identified.

We propose a new forward electricity market framework that admits heterogeneous market participants with second-order cone strategy sets, who accurately express the nonlinearities in their costs and constraints through conic bids, and a network operator facing conic operational constraints. In contrast to the prevalent linear-programming-based electricity markets, we highlight how the inclusion of second-order cone constraints improves uncertainty-, asset-, and network-awareness of the market, which is key to the successful transition towards an electricity system based on weather-dependent renewable energy sources. We analyze our general market-clearing proposal using conic duality theory to derive efficient spatially-differentiated prices for the multiple commodities, comprised of energy and flexibility services. Under the assumption of perfect competition, we prove the equivalence of the centrally-solved market-clearing optimization problem to a competitive spatial price equilibrium involving a set of rational and self-interested participants and a price setter. Finally, under common assumptions, we prove that moving towards conic markets does not incur the loss of desirable economic properties of markets, namely market efficiency, cost recovery, and revenue adequacy. Our numerical studies focus on the specific use case of uncertainty-aware market design and demonstrate that the proposed conic market brings advantages over existing alternatives within the linear programming market framework.

The increasing penetration of electric vehicles brings a consequent increase in charging facilities in the low-voltage electricity network. Serving all charging requests on-demand can endanger the safety of the electrical power distribution network. This creates the issue of fairly allocating the charging energy among electric vehicles while maintaining the system within safe operational margins. However, calculating efficient charging schedules for the charging stations bears a high computational burden due to the non-convexities of charging stations’ models. In this paper, we consider a tri-level system with electric vehicles, charging stations, and a power distribution system operator. The objective of each station is formulated as a max-min fairness, mixed-integer linear optimisation problem, while the network constraints are modelled using a second-order conic formulation. In order to tackle the computational complexity of the problem, we decompose it and use a novel approximation method tailored to this problem. We compare the performance of the proposed method with that of the popular alternating direction method of multipliers. Our simulation results indicate that the proposed method achieves a near-optimal solution along with promising scalability properties.

Renewable energy generation is offered through electricity markets, quite some time in advance. This then leads to a problem of decision-making under uncertainty, which may be seen as a newsvendor problem. Contrarily to the conventional case for which underage and overage penalties are known, such penalties in the case of electricity markets are unknown, and difficult to estimate. In addition, one is actually only penalized for either overage or underage, not both. Consequently, we look at a slightly different form of a newsvendor problem, for a price-taker participant offering in electricity markets, which we refer to as Bernoulli newsvendor problem. After showing that its solution is consistent with that for the classical newsvendor problem, we then introduce distributionally robust versions, with ambiguity possibly about both the probabilistic forecasts for power generation and the chance of success of the Bernoulli variable. Both versions of the distributionally robust Bernoulli newsvendor problem admit closed-form solutions. We finally use simulation studies, as well as a real-world case-study application, to illustrate the workings and benefits from the approach.