4.1.0
Neural-ART - Supported operators and limitations


ST Edge AI Core

Neural-ART - Supported operators and limitations


for STM32 target, based on ST Edge AI Core Technology 4.1.0



r1.4

Overview

This document describes the model formats, quantization schemes, supported operators, and hardware limitations relevant to deploying neural networks on the Neural-ART™ accelerator.

Quantized Model Format

The Neural-ART compiler supports the following quantized model formats, subject to the supported data types:

  • TensorFlow Lite / LiteRT models quantized using post-training quantization (PTQ) or quantization-aware training (QAT). TFLite can quantize only operators with supported quantized kernels and compatible operator patterns.
  • ONNX quantized models in QDQ format. In this format, QuantizeLinear and DequantizeLinear operators represent quantization and dequantization around operators in the graph. These models can be generated using ONNX Runtime quantization tools.

Only full integer static quantization is supported. It quantizes both weights and activations into integers, using representative calibration data to determine quantization parameters. Weight-only and dynamic quantization are not supported.

If a graph contains unsupported operators, the model architecture may need to be modified or those operators replaced. Where available, the compiler can use a software fallback. Mixed-precision schemes across layers or operators are supported. Unsupported hardware mappings fall back to floating-point kernels; mixed N-bit-to-M-bit activation mapping is not enabled on the hardware activation path.

Supported precision and data types

type weight precision activation precision quantization scheme notes
w8a8 int8 int8 full integer, scale/zero-point, ss/sa per-channel/per-tensor for weights. Only per-tensor for activations.
w4a8 int4 int8 full integer, scale/zero-point, ss/sa per-channel/per-tensor for weights. Only per-tensor for activations. Convolution and dense layer type only.
w8a16 int8 int16 full integer, fixed point number (or qmn) format only, ss/ss per-tensor, scale factor is always a power of 2 and offset/zero point is equal to 0. Only for weights: scales can be both per-channel and per-tensor.

To preserve correctness for some quantization parameters, the compiler may map individual layers to floating-point software fallback.

Inference Model Constraints

Neural-ART is a re-configurable inference-only engine capable of accelerating in hardware the quantized inference models, no training mode is supported.

  • Input and output tensors must be static
  • Variable-length batch dimension (i.e. (None,)) is considered as equal to 1
  • Operator with un-connected output is not supported
  • Mixed data operations (i.e hybrid operator) are not supported, activations and weights should be quantized
  • Data type for the weights/activations tensors must be:
    • int8 (scale/offset format) ss/sa scheme (see Quantized models – per-channel)
    • if float32 operation is requested, it will be mapped on a SW operation

Custom Layer Support - Post-Processing

The custom operator like TFLite_Detection_PostProcess is not supported. For object-detection models, consider removing this operator before quantization and retaining only the backbone and detection head. Post-processing, including non-maximum suppression (NMS), can then be performed by an external library on the host MCU.

TFlite Operation Mapping

The following table describes how TensorFlow Lite (TFLite) operations map to the Neural-ART stack. Only supported quantized operations in w8a8 format can be mapped directly to Neural-ART hardware. If an operation is not mapped to hardware, an int8 or float32 software fallback may be used on the host subsystem. Software quantize and dequantize operations are inserted as needed to convert between int8 and float32; these operations are deployed as software epochs.

TFLite Operation mapped on precision comment
ABS HW a8
ADD HW a8 SW mapping if broadcasting out of HW supported broadcasting
ADD_N SW_FLOAT
ARG_MAX SW_INT a8
ARG_MIN SW_INT a8
AVERAGE_POOL_2D HW a8 See Pool Limitations
BATCH_MATMUL HW w8a8 Second input should be constant else SW fallback is considered
CAST HW a8
CEIL HW a8
CONCATENATION HW a8
CONV_2D HW w8a8 see Optimal Kernel Size / Stride Size and Tips. Weights tensor should be constant else SW fallback is considered.
COS HW a8
COSH HW a8
DEPTH_TO_SPACE HW a8 with same input/output quantization
DEPTHWISE_CONV_2D HW w8a8
DEQUANTIZE SW_FLOAT uint8/int8/int16 -> float
DIV SW_INT
ELU HW a8
EXP HW a8
EXPAND_DIMS HW a8
FLOOR SW_FLOAT
FULLY_CONNECTED HW w8a8
GATHER SW_INT a8
GELU HW a8
HARD_SWISH HW a8
L2_NORMALIZATION SW_FLOAT
LEAKY_RELU HW a8
LOCAL_RESPONSE_NORMALIZATION SW_FLOAT
LOG SW_FLOAT
LOGISTIC HW a8
MAX_POOL_2D HW w8a8 Decomposition in multiple operations if kernel Height or Width are greater than 3
Horizonal and vertical strides ranging from 1 to 15
See Pool Limitations
MAXIMUM HW a8
MEAN SW_FLOAT
MINIMUM HW a8
MUL HW a8 SW fallback with broadcast or more than 4 dimensions. See broadcasting limitations
NEG HW a8
PACK HW a8
PAD HW a8 Partial – according to the parameters, it can be HW-assisted with the extra epochs.
POW HW a8 Supported in HW for 8bit Scale/Offset input and constant exponent. Otherwise, fallback to float.
PRELU HW a8 slope attribut should be quantized.
QUANTIZE SW_FLOAT float -> uint8/int8/int16
(RE) QUANTIZE HW uint8 <-> int8
REDUCE_MAX HW a8 HW if it can be convered to GlobalMaxPool. No overhead if the reduced axes are the right-most ones, and the input shape.size() = number of reduced_axes+2. Additional Transposition and Reshapes might be required otherwise. SW fallback available
REDUCE_MIN HW a8 HW if it can be convered to GlobalMinPool. No overhead if the reduced axes are the right-most ones, and the input shape.size() = number of reduced_axes+2. Additional Transposition and Reshapes might be required otherwise. SW fallback available
REDUCE_PROD SW_FLOAT
RELU HW a8
RELU6 HW a8
RESHAPE HW a8
RESIZE_BILINEAR SW_INT
RESIZE_NEAREST_NEIGHBOR HW/SW_INT a8 HW if (coordinate_transformation_mode=‘asymmetric’ AND nearest_mode=‘floor’). SW_INT otherwise
ROUND SW_FLOAT
RSQRT HW a8
SHAPE HW a8
SIN HW a8
SLICE HW a8
SOFTMAX HW a8 Maximum supported dimensions: 4. There must be a leading dimension equal to 1 and up to 4 dimensions: (1, D1, D2, D3). Softmax computation dimension must be smaller or equal to 2048. The SW fallback can be enabled using the option –disable-hw-softmax.
SPACE_TO_DEPTH HW a8 with same input/output quantization
SPLIT HW/SW_INT a8 HW or SW_INT depends on axis
SPLIT_V HW/SW_INT HW or SW_INT depends on axis
STRIDED_SLICE HW a8
SQUARE HW a8
SQUEEZE HW a8
SQRT HW a8
SUB HW a8
SUM SW_FLOAT
TANH HW a8
TRANSPOSE HW a8 Partial – according to the parameters, it can be HW-assisted with the extra epochs.
TRANSPOSE_CONV HW a8 Partial – according to the parameters, it can be HW-assisted with the extra epochs.
TOPK_V2 SW_FLOAT a8
UNPACK HW a8

ONNX Operation Mapping

The following table describes how ONNX operations map to the Neural-ART stack. Supported quantized operations in w8a8, w4a8, or w8a16 format can be mapped directly to Neural-ART processing units. If an operation is not mapped to hardware, an int8 or float32 software fallback may be used on the host subsystem. Software quantize and dequantize operations are inserted as needed to convert between int8 and float32; these operations are deployed as software epochs.

ONNX Operator Pytorch Name mapped on precision comment
Abs abs HW/SW_FLOAT a8 HW if Offset is 0, SW_FLOAT otherwise
Acos acos HW a8
Acosh HW a8
Add/sum add HW a8, a16 SW mapping if broadcasting out of HW supported broadcasting
And HW a8
ArgMax argmax SW_INT a8
ArgMin argmin SW_INT a8
Asin asin HW a8
Asinh HW a8
Atan atan HW a8
Atanh argmin HW a8
AveragePool avg_pool1d/avg_pool2d HW a8 See Pool limitations
BatchNormalization BatchNorm1D/BatchNorm2D HW w8a8, w8a16
Cast HW a8 Software library mapped on DMAs or pure software
Ceil ceil SW_FLOAT
Clip clip/ReLU6 HW a8, a16
Concat concat/concatenate HW a8, a16 Mapped on HW or SW depending on the batch and concatenation axis
Conv Conv1D/Conv2D HW w8a8, w4a8, w8a16 see Optimal Kernel Size / Stride Size and Tips. Weights tensor should be constant else SW fallback is considered.
ConvTranspose HW w8a8, w8a16
Cos cos HW a8
Cosh cosh HW a8
DepthToSpace SW a8, a16 DMA is used with mode “DCR”
DequantizeLinear dequantize SW_FLOAT w8a8 int8/uint8/int16 -> float32
Div div SW_INT a8 HW if second operand is a constant
Elu elu HW a8
Erf erf HW a8
Exp exp HW a8
Flatten flatten HW a8, a16
Floor floor SW_FLOAT
Gather gather SW_INT
Gemm HW w8a8, w4a8, w8a16 Params input should be constant else SW fallback is used
Gelu GELU HW a8 Supported as single ONNX GELU operator and expanded version (see --GELU-recognition option)
GlobalAveragePool HW a8, a16 Limited window. Software fallback available
GlobalMaxPool HW a8, a16 Limited window. Software fallback available
Hardmax SW_FLOAT
HardSigmoid HW a8
HardSwitch HW a8
Identity identity HW a8, a16
InstanceNormalization InstanceNorm1/InstanceNorm2 SW_FLOAT
LeakyRelu LeakyReLU HW a8, a16
LRN LocalResponseNormalization SW_FLOAT
Log log SW_FLOAT
LpNormalization SW_FLOAT
MatMul HW w8a8, w8a16 Non-constant inputs leading dimension must be equal to 1 and with up to 4 dimensions: (1, D1, D2, D3).The SW fallback can be enabled using the option –disable-hw-matmul.
MaxPool HW a8, a16 Decomposition in multiple operations if kernel Height or Width are greater than 3. Horizonal and vertical strides ranging from 1 to 15. See Pool limitations
Max HW a8
Min HW a8
Mod SW_FLOAT
Mul mul HW a8, a16 see broadcasting limitations
Neg neg HW a8
Pad ConstantPad1D/ConstantPad2D/ZeroPad2D HW a8, a16 Mapped on ConvAcc, DMA or SW
Pow HW a8 Supported in HW for 8bit Scale/Offset input and constant exponent. Otherwise, fallback to float.
PRelu HW a8 slope attribut should be quantized.
QLinearAdd HW a8, a16 Same as Add
QLinearAveragePool HW a8 Same as AveragePool
QLinearConcat HW a8, a16 Same as Concat
QLinearConv HW w8a8, w8a16 Same as Conv
QLinearGlobalAveragePool HW a8, a16 same as GlobalAveragePool
QLinearMatMul HW w8a8, w8a16 same as MatMul
QLinearMul HW w8a8, w8a16 same as Mul
QuantizeLinear quantize_per_tensor SW_FLOAT int8/uint8 -> float/int8 requantization supported in HW
Reciprocal reciprocal HW a8
ReduceLogSumExp SW_FLOAT a8
ReduceMax HW a8 HW if it can be convered to GlobalMaxPool. No overhead if the reduced axes are the right-most ones, and the input shape.size() = number of reduced_axes+2. Additional Transposition and Reshapes might be required otherwise. SW fallback available
ReduceMean HW/SW_INT a8, a16 HW if it can be convered to GlobalAveragePool. No overhead if the reduced axes are the right-most ones, and the input shape.size() = number of reduced_axes+2. Additional Transposition and Reshapes might be required otherwise. SW fallback available
ReduceMin HW a8 HW if it can be convered to GlobalMinPool. No overhead if the reduced axes are the right-most ones, and the input shape.size() = number of reduced_axes+2. Additional Transposition and Reshapes might be required otherwise. SW fallback available
ReduceProd SW_FLOAT a8
ReduceSum HW a8, a16 HW if it can be convered to GlobalAveragePool. No overhead if the reduced axes are the right-most ones, and the input shape.size() = number of reduced_axes+2. Additional Transposition and Reshapes might be required otherwise. SW fallback available
Relu relu HW a8, a16
Reshape shape HW a8, a16
Resize upsample SW_INT a8, a16 Resize Nearest Neighbor on HW if coordinate_transformation_mode=‘asymmetric’ AND nearest_mode=‘floor’
Round round SW_FLOAT
Selu selu HW a8
Shape HW a8, a16
Sin sin HW a8
Sinh sinh HW a8
Sigmoid sigmoid HW a8 HW Acceleration available also for X*Sigmoid(X)
Slice HW a8, a16 DMA acceleration
Softmax softmax HW a8 Maximum supported dimensions: 4. There must be a leading dimension equal to 1 and up to 4 dimensions: (1, D1, D2, D3). Softmax computation dimension must be smaller or equal to 2048. The SW fallback can be enabled using the option –disable-hw-softmax.
Softplus softplus SW_FLOAT a8
SoftSign softsign SW_FLOAT a8
Split split HW a8, a16
Squeeze squeeze HW a8, a16
Sqrt sqrt HW a8
Sub sub HW a8
Swish SiLU SW_FLOAT / HW a8 Not supported as single ONNX SWISH operator, however the expanded version is supported (see --SWISH-recognition option)
Tan tan HW a8
Tanh tanh HW a8
ThresholdedRelu HW a8
Tile SW_INT a8 Optimized by constant folding passing through front-end, SW_INT fallback otherwise
Transpose transpose HW a8, a16
Topk SW_FLOAT
Unsqueeze unsqeeze HW a8, a16
Upsample upsample SW_FLOAT

Neural-ART™ Processor Unit Tips and Limitations

Optimal Kernel Size / Stride Size

Maximum kernel width supported in hardware is 6 for stride 1 and 12 for stride of minimum 2. Vertical kernel size is limited to max 3. All larger kernel dimensions lead to a decomposition done by the compiler which has to be processed iteratively.

  • Best use of a kernel height is 3 (using just 2 will not help)
  • Best use of a kernel width is 3, 6, 12 (anything between and larger than 1 will not be better)
  • Avoid horizontal strides that are not 1, 2, or 4 if possible (2 and 4 are valid but lead to less data reuse)
  • Avoid vertical strides that are not 1, 2, or 4 if possible (vertical strides are in general inefficient)
  • Kernel dimensions of 1 (horizontal or vertical) are special cases (e.g., 1x1, 2x1, 1x2, 2x1, 1x3, 1x6, 1x12, etc.) and can be handled efficiently. They will run a lot faster than, for example, 3x2 or 2x3
  • Kernels with height 1 have no restrictions on the feature width (theoretical max is 2^16-1).

Don’t use strides larger than the kernel dimension (useless data). The compiler will have to play tricks as this is not natively supported in hardware. Note that layers with more input channels (ICH) than output channels (OCH) can often be mapped with better efficiency.

Special Case: 1x1 Kernel

1x1 kernel can be handled very efficiently if the number of input channels (ICH) is equal to N* 72…128 and the number of output channels (OCH) is equal to M(16..24) (best M24), with N, M as integers. If these numbers are larger than the amount of available CONV processing units (4 CA for STM32N6 NeuralART™), the compiler will translate this in multiple iterations.

To illustrate, three typical examples:

  • ICH=128 and OCH=96 would run at the same speed as ICH=32 and OCH=96 even when there are 4 times fewer MACs as in the first one (4 CA in parallel)
  • ICH=512 and OCH=24 would run at the same speed as ICH=512 and OCH=4 even when there are 6 times fewer MACs as in the first one (4 CA in series)
  • ICH=256 and OCH=48 would run at the same speed as ICH=129 and OCH=25 even when there are almost 4 times fewer MACs as in the first one. (2 CA in series, 2 chains in parallel)

Zeros in the Kernel or Feature Data

They will save power (gated MAC units) but not affect the performance.

Optimal Feature Width

For 8-bit feature data, feature width multiplied by the used batch depth (input channels) must be equal to or lower than 2048. Thus, for 512 wide features, batch depth will be limited to 4 and wider feature sizes further reduce the number of input channels and may cause the compiler to even split the feature into multiple columns.

Optimal Pool Operations with up to 3x3 Pooling Windows

  • Don’t use pooling windows larger than 3, else they will be decomposed by the compiler.
  • The number of input channels is limited. Check that a previous operation is not generating more input channels, else chaining will not be possible.
  • The line buffer of the pooling unit is limited. Avoid using data with [width x input_channels] larger than 2048 else the compiler has to split the operator into multiple columns.

Hardware supported broadcasting

  • Only Unidirectional broadcasting is supported in hardware
  • The best performance is achieved with a number of elements less or equal to 512.
  • Broadcasting is mapped on Arithmetic Unit with best performance in case of
    • Scalar broadcasting
    • Channel broadcasting (C <= 512, other dimensions =1)
    • Height broadcasting (H <= 512, other dimensions =1)
    • Width broadcasting (W <= 512, other dimensions =1)
    • Height-Width broadcasting ((H*W) <= 512, other dimensions =1)
  • DMAs supported broadcasting:
    • Channel broadcasting
    • Height broadcasting
    • Width broadcasting
    • Height-Width broadcasting
    • Channel-Width broadcasting
    • Channel-Height broadcasting
  • When the broadcasted input is the second input of the node
    • Supported only for commutative operations

Convolutions tips

  • Avoid big dilation factor, big dilation factors leads to non-optimal performance.
  • Avoid using prime number as number of kernels/channels, it will limit the heuristic for kernel/channel splitting.
  • When using a very large kernel, use a multiple of 3 for width and height
  • Horizontal strides lead to less data reuseVertical strides are in general inefficient.
  • Pad <= 2, are supported without impacting performance.
  • Zeros in the kernel or feature data will save power (gated MAC) but not affect the performance. Values that are almost zero cannot be used to gate the MAC units, thus no power saving.

Pool limitations

  • Only 1D and 2D supported
  • Kernel shape: Height and Width between 1 and 3 included
  • Horizonal and vertical strides ranging from 1 to 15
  • Pad
    • Left, right and top padding ranging from 0 to 7
    • Max bottom padding supported is up to Window Height minus 1
  • input features Width <=2048 when 8 bits are used, and <=1024 when 16 bits are used.

Note that for N6 HW mapping: padded values are always included in the computation. Fallback on SW available otherwise.