Neural net implementation of steam properties on FPGA

ABSTRACT


INTRODUCTION
Recent trends in power plant and industrial applications indicate developments like data reconciliation (DR), Guo et al. [1] model predictive control (MPC), Wang et al. [2] and optimization, Niegodajew et al. [3] to improve performance of plants and processes. They are based on the first principle thermodynamic formulations as well complex mathematical models. They are limited to offline simulations or implemented in standalone desktop systems. Direct implementation of these algorithms in microcontrollers and field-programmable gate arrays (FPGAs), often encountered in power and process plant distributed control systems (DCS), is difficult and computationally inefficient. One of the constraints is efficient implementation of thermodynamic properties of the working fluids like water, steam and gasses, in the FPGAs. The applications require real time prediction of thermodynamic properties like enthalpy, along with their first and second order partial derivatives. The properties are represented by nonlinear Gibbs free energy formulations [4]. These functions are implemented in higher level languages running into 5000-7000 lines of code. These are evaluated iteratively. They are repeatedly called in the applications and are not  [5], [6]. However, many of these efforts are not directly suited for implementation in FPGAs used in the commercial DSC platform. Cardaliaguet and Euvrard [7] reported artificial neural networks (ANNs) with explicit weights to approximate a function and its derivatives and highlighted usage for control. Hashem and Schmeiser [8] showed that a linear combination of a set of feed forward ANNs trained to approximate a function and its derivatives improve accuracy as against a single ANN. While these lay the foundation for using ANNs for approximating a function and its derivatives, Krishnadutt and Krishnaiah [9] demonstrated the power of a few simple ANNs to simultaneously approximate steam properties and the corresponding derivatives of entire steam properties; thus replacing thousands of lines of code in higher language. These ANNs approximate the complex steam properties to the required degree of accuracy for real time applications.
In spite of the availability of many high-level language tools, the progress in implementing ANNs in FPGA as seen in literature [10]- [15] is slow. Recent surveys [16]- [18] highlight that FPGA based neural network accelerators are preferred over the application specific integrated circuits (ASICs) due to their energy efficiency with a small footprint, but many problems and challenges need to be addressed. These bring out the challenges in training deep neural networks (DNNs) with FPGA and propose a performance metric and evaluation workflow to compare the FPGA-based systems for DNN training specific to computer vision tasks. However, for a pretrained network, only device utilization and energy efficiency may be considered. Reviews highlight major issues like data representation, implementation of inner product between weight matrix and previous layer outputs, activation functions, device utilization, energy efficiency, for different applications.
Non-linear activation function is the main concern of many researchers while implementing an ANN in FPGA. Piazza et al. [19] reported adaptable look up table (LUT)-based activation functions for neurons, in learning by backward difference with different learning rates and two look up tables, one for weights and the for coefficients respectively, whereas Bieu et al. [20] computed the sigmoid function and its derivative in digital hardware by a sum of steps resulting in area-efficiency. Reconfiguration capabilities of the Atmel FPGA are exploited by Lysight et al. [21] for implementing larger ANN with individual layers of the network with time multiplexing on the logic array at the cost of system performance. Tisan et al. [22] investigated different approximating functions from the point of view of hardware resource utilization and induced errors and concluded that piecewise linear approximation of the activation function is the best. Problems encountered in implementing an ANN in VHDL are reported by P. Dondon et al. [23] wherein sigmoid activation is approximated by sampling of Logsig function with argument between 0 to 1. Ngah et al.
[24] used combination of second order non-linear function (SONF) and differential LUT for implementing an ANN. The two-step approach is reported to have an improved accuracy that is 10 times better than that of using only SONF and twice better than just using LUT. Li et al. [25] implemented a neuron block with sigmoid function using the CORDIC algorithm.
Some of the applications of ANN implementation on FPGA include classification of the region of pixels i.e., hand regions by Krips et al. [26] using three inputs representing RGB values with a single hidden layer and one output with data represented by integers and weights scaled up and rounded off to the nearest integer. Recognition of digits using a network with 300 inputs and 10 outputs with a single neuron is reported by Latino et al. [27]. A method of configurable MLP with a single neuron block with floating point add and multiply units along with activation function as LUT for a smart position sensor of solar panels has been studied by Dąbrowski et al. [28]. A neural classifier with fixed point representation and 12-bits for detecting damaged toothed gears using vibroacoustic signals is highlighted by Polat and Yildirim [29].
These studies consider sigmoid function with argument in the range (0-1). FPGA implementation of an ANN for simultaneous prediction of thermodynamic properties and their derivatives required in advanced applications like MPS, DR, and optimization is not reported. Custom ANNs suitable for simultaneous estimation of both enthalpy and its derivatives suitable for power plant applications are taken up for implementation in FPGA. In the following sections, FPGA implementation of a single neural network called SteamNet is described. The simultaneous use of LUT and Taylor series for large arguments of sigmoid function is presented. Figure 1 represents water and steam properties used in industrial applications [4]. The different regions represent subcooled 1). Supercritical water/steam. 2). Super heat steam. 3). Saturation. 4). High  The enthalpy functions for all the regions are represented by different neural networks as shown in Figure 2. The networks SupNet, SubCNet, SatVnet and SatFnet represent enthalpy in superheated, sub cooled, saturated vapor and saturated fluid regions of Figure 1. P2T network is used to decide the zone in which steam property is required. Function (P, T) in Figure 2 represents thermodynamic property, for example enthalpy, as a function of pressure P and temperature T. A single neural network, SteamNet, with compact architecture 2x10x5x3, shown in Figure 3, is used to represent all these different networks. The architecture is chosen such that the accuracy of and both the derivatives  and  are obtained with good accuracy. SteamNet stores different weight and bias matrices suitable for different regions of Figure 2. SteamNet is designed with sigmoid activation function represented by (1) and (2). Sigmoid function takes the weighted sum of outputs from the previous layer. FPGA implementation is achieved by using single perceptron with sigmoid activation function.

CUSTOM NETS FOR THERMODYNAMIC PROPERTIES OF WATER/STEAM
Where and are weight matrix between layers and and bias vector at layer respectively. is the input vector in the th layer. In the SteamNet, typical values of and are in the range (-16.1735 to +23.174). For normalized values of , this results in larger values of |Z|, greater than 1. Accurate exponentiation of |Z|, results in higher order terms of Taylor series which results in an overflow in FPGA due to factorial terms in denominators in series representation. However, acceptable accuracy with few terms in series is obtained when |Z| < 1.
Hence, the exponential function appearing in (2) is recast as in (3) and (4) to facilitate its implementation in FPGA.

= + (4)
Argument Z is split into a and b, such that -1<= <=1 and b is an integer; is a set of predefined constants (Bj) stored in look up table (LUT). Term is obtained by Taylor series using (5).
= max(0, ) Saving in device utilization is seen in ReLU as compared to sigmoid vide Table.1. However, ReLu is less accurate as shown in Figure 4. Hence, SteamNet regression is implemented using sigmoid. Marginal gain in device utilization in SteamNet can be achieved by reducing the number of terms in (5), vide Figure 5.   SUBTRACT. The SteamNet requires four or five coefficients, depending on accuracy for and nine constants for . Figure 7 shows the percent error in sigmoid implemented in Verilog as per (5) as compared to standard numerical library functions.
Function P2T in the SteamNet, P2T with pressure (Pr -kg/cm 2 ) as input and saturation temperature (Tsat -o C) as output using a custom three-layer network is realized using Xilinx ISE Design suite and Verilog HDL with Virtex family FPGA. Figure 9 and Figure 10 show the structural model of steam Net using Xilinx ISE synthesis tool and the device utilization summary of steam net implemented in Virtex XC6VCX240T FPGA board, respectively.

RESULTS AND DISCUSSION
The above section details the implementation of SteamNet using a single neuron model wherein different weight matrices and bias vectors, at the three layers of the network in Figure 3, are stored in memory. Computation of the output is achieved by repeated usage of single neuron model implemented as per Figure 6 and Figure 8. The Figure 11, Figure 12 and Figure 13 show the layer-1, layer-2 and layer-3 input-output simulation results respectively, obtained from Xilinx ISE simulator for a typical normalized input of 0.1 kg/cm 2 . The simulation shows, at time t=0, for layer-1, an input value of x=oxh3dcccccd produces outputs at five neurons as fx1=ox3ee39af7, fx2=oxf4ea1cf, fx3=3eddf270, fx4=3ed6dece and fx5=3f0d5195. Similarly, the outputs are shown for the second and third layers. This results in a normalized output of Tsat of 0.14791287 corresponding to 63.699696 deg C. This has a maximum error of 0.61% as shown in Figure 14.

CONCLUSION
A pre-trained compact single neural net for simultaneous estimation of steam properties and their partial derivatives is taken up for implementation in FPGA which is useful for real time control and monitoring of power plants. This technique enables entire steam properties and their derivatives to be implemented in FPGA without the need for porting nearly 5000 lines of C/C++ code. FPGA implementation of one network showing the relationship between pressure and saturation temperature is demonstrated. Verilog HDL simulation results obtained for XC6VCX240T device of Virtex family with IEEE 754 floatingpoint 32-bit data representation with an efficient sigmoid activation having large arguments are presented. Error in the sigmoid function for the argument range (-30-30) is within 0.02% which gives a mean accuracy of final outputs of the SteamNet to be around 0.5%. Results of FPGA simulation on a target system show the cycle time is of the order of 700ns. Device utilization for the network are presented using the approach. This technique enables to implementation of manytranscendental functions in FPGA without the need for porting the iterative complex codes; hence it is expected many real time applications like enthalpy-based steam temperature control in boilers, often used in power and process, can be easily developed.