The eltwise primitive applies an operation to every element of the tensor:
\[ dst(\overline{x}) = Operation(src(\overline{x})), \]
where \(\overline{x} = (x_n, .., x_0)\).
The following operations are supported:
Operation | DNNL algorithm kind | Formula |
---|---|---|
abs | dnnl_eltwise_abs | \( f(x) = \begin{cases}</td> </tr> </table> x & \text{if}\ x > 0 \\ -x & \text{if}\ x \leq 0 \end{cases} \) | bounded_relu | dnnl_eltwise_bounded_relu | \( f(x) = \begin{cases} \alpha & \text{if}\ x > \alpha, \alpha \geq 0 \\ x & \text{if}\ 0 < x \leq \alpha \\ 0 & \text{if}\ x \leq 0 \end{cases} \) | clip | dnnl_eltwise_clip | \( f(x) = \begin{cases} \beta & \text{if}\ x > \beta, \beta \geq \alpha \\ x & \text{if}\ \alpha < x \leq \beta \\ \alpha & \text{if}\ x \leq \alpha \end{cases} \) | elu | dnnl_eltwise_elu | \( f(x) = \begin{cases} x & \text{if}\ x > 0 \\ \alpha (e^x - 1) & \text{if}\ x \leq 0 \end{cases} \) | exp | dnnl_eltwise_exp | \( f(x) = e^x \) | gelu | dnnl_eltwise_gelu | \( f(x) = 0.5 x (1 + tanh[\sqrt{\frac{2}{\pi}} (x + 0.044715 x^3)])\) | linear | dnnl_eltwise_linear | \( f(x) = \alpha x + \beta \) | log | dnnl_eltwise_log | \( f(x) = \log_{e}{x} \) | logistic | dnnl_eltwise_logistic | \( f(x) = \frac{1}{1+e^{-x}} \) | relu | dnnl_eltwise_relu | \( f(x) = \begin{cases} x & \text{if}\ x > 0 \\ \alpha x & \text{if}\ x \leq 0 \end{cases} \) | soft_relu | dnnl_eltwise_soft_relu | \( f(x) = \log_{e}(1+e^x) \) | sqrt | dnnl_eltwise_sqrt | \( f(x) = \sqrt{x} \) | square | dnnl_eltwise_square | \( f(x) = x^2 \) | swish | dnnl_eltwise_swish | \( f(x) = \frac{x}{1+e^{-\alpha x}} \) | tanh | dnnl_eltwise_tanh | \( f(x) = \tanh{x} \) #### Difference Between Forward Training and Forward Inference There is no difference between the dnnl_forward_training and dnnl_forward_inference propagation kinds. ### Backward The backward propagation computes \(diff\_src(\overline{x})\), based on \(diff\_dst(\overline{x})\) and \(src(\overline{x})\). ## Implementation Details ### General Notes
### Data Type Support The eltwise primitive supports the following combinations of data types: | Propagation | Source / Destination | Intermediate data type | :– | :– | :– | forward / backward | f32, bf16 | f32 | forward | f16 | f16 | forward | s32 / s8 / u8 | f32
Here the intermediate data type means that the values coming in are first converted to the intermediate data type, then the operation is applied, and finally the result is converted to the output data type. ### Data Representation The eltwise primitive works with arbitrary data tensors. There is no special meaning associated with any logical dimensions. ### Post-ops and Attributes The eltwise primitive doesn't support any post-ops or attributes.
## Performance Tips
|