Glossary

(d) Softmax

involves exponentials that can overflow in fp16 (max fp16 value is 65,504; $e^{11} \approx 60,000$ already approaches the limit).

Learn More

Related Terms