Civil Rights
Movements, leaders, victories and the continuing fight for equality.
q2K BHP
Black History Portal
THE BHP WIRE —
HIDDEN TRUTHS
What's New!
Explore the people, places, events, achievements, struggles and stories that shaped our journey.
Movements, leaders, victories and the continuing fight for equality.
Innovation, patents, science, technology and world-changing contributions.
Pioneers, champions, Negro Leagues, records, activism and excellence.
Meet the people whose lives, choices and achievements shaped the journey.
Black towns, communities, institutions and places where history happened.
Moments that changed communities, movements, institutions and the nation.
Mansa Musa was the ruler of the Mali Empire in West Africa. Details recorded here should be sourced; unknown information is left blank.
MORE →Reflects the personal views, recollections, and perspective of the author, Mike Davis.
This is a personal recollection on the Move fire on May 13, 1985
| Floating-point formats |
|---|
| IEEE 754 |
|
| Other |
| Alternatives |
The bfloat16 (brain floating point)[1][2] floating-point format is a computer number format occupying 16 bits in computer memory; it represents a wide dynamic range of numeric values by using a floating radix point. This format is a shortened (16-bit) version of the 32-bit IEEE 754 single-precision floating-point format (binary32) with the intent of accelerating machine learning and near-sensor computing.[3] It preserves the approximate dynamic range of 32-bit floating-point numbers by retaining 8 exponent bits, but supports only an 8-bit precision rather than the 24-bit significand of the binary32 format. More so than single-precision 32-bit floating-point numbers, bfloat16 numbers are unsuitable for integer calculations, but this is not their intended use. Bfloat16 is used to reduce the storage requirements and increase the calculation speed of machine learning algorithms.[4]
The bfloat16 format was developed by Google Brain, an artificial intelligence research group at Google, for use in its TPU v2, released in 2017.[1] It is utilized in many CPUs, GPUs, and AI processors, such as Intel Xeon processors (AVX-512 BF16 extensions), Intel Data Center GPU, Intel Nervana NNP-L1000, Intel FPGAs,[5][6][7] AMD Zen, AMD Instinct, NVIDIA GPUs, Google Cloud TPUs,[8][9][10] AWS Inferentia, AWS Trainium, ARMv8.6-A,[11] and Apple's M2[12] and therefore A15 chips and later. Many libraries support bfloat16, such as CUDA,[13] Intel oneAPI Math Kernel Library, AMD ROCm,[14] AMD Optimizing CPU Libraries, PyTorch, and TensorFlow.[10][15] On these platforms, bfloat16 may also be used in mixed-precision arithmetic, where bfloat16 numbers may be operated on and expanded to wider data types.
bfloat16 has the following format:
The bfloat16 format, being a shortened IEEE 754 single-precision 32-bit float, allows for fast conversion to and from an IEEE 754 single-precision 32-bit float; in conversion to the bfloat16 format, the exponent bits are preserved while the significand field can be reduced by truncation (thus corresponding to round toward 0) or other rounding mechanisms, ignoring the NaN special case. Preserving the exponent bits maintains the 32-bit float's range of ≈ 10−38 to ≈ 3 × 1038.[16]
The bits are laid out as follows:
| IEEE half-precision 16-bit float | ||||||||||||||||||||||||||||||||||
| sign | exponent (5 bit) | fraction (10 bit) | ||||||||||||||||||||||||||||||||
| ┃ |
|
| ||||||||||||||||||||||||||||||||
| 0 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |||||||||||||||||||
| 15 | 14 | 10 | 9 | 0 | ||||||||||||||||||||||||||||||
| bfloat16 | ||||||||||||||||||||||||||||||||||
| sign | exponent (8 bit) | fraction (7 bit) | ||||||||||||||||||||||||||||||||
| ┃ |
|
| ||||||||||||||||||||||||||||||||
| 0 | 0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | |||||||||||||||||||
| 15 | 14 | 7 | 6 | 0 | ||||||||||||||||||||||||||||||
| IBM's DLFloat (16 bits)[17] | ||||||||||||||||||||||||||||||||||
| sign | exponent (6 bit) | fraction (9 bit) | ||||||||||||||||||||||||||||||||
| ┃ |
|
| ||||||||||||||||||||||||||||||||
| 0 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |||||||||||||||||||
| 15 | 14 | 9 | 8 | 0 | ||||||||||||||||||||||||||||||
| Nvidia's TensorFloat-32 (19 bits) | ||||||||||||||||||||||||||||||||||
| sign | exponent (8 bit) | fraction (10 bit) | ||||||||||||||||||||||||||||||||
| ┃ |
|
| ||||||||||||||||||||||||||||||||
| 0 | 0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | ||||||||||||||||
| 18 | 17 | 10 | 9 | 0 | ||||||||||||||||||||||||||||||
| ATI's fp24 format[18] | ||||||||||||||||||||||||||||||||||
| sign | exponent (7 bit) | fraction (16 bit) | ||||||||||||||||||||||||||||||||
| ┃ |
|
| ||||||||||||||||||||||||||||||||
| 0 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |||||||||||
| 23 | 22 | 16 | 15 | 0 | ||||||||||||||||||||||||||||||
| Pixar's PXR24 format | ||||||||||||||||||||||||||||||||||
| sign | exponent (8 bit) | fraction (15 bit) | ||||||||||||||||||||||||||||||||
| ┃ |
|
| ||||||||||||||||||||||||||||||||
| 0 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |||||||||||
| 23 | 22 | 15 | 14 | 0 | ||||||||||||||||||||||||||||||
| IEEE 754 single-precision 32-bit float | ||||||||||||||||||||||||||||||||||
| sign | exponent (8 bit) | fraction (23 bit) | ||||||||||||||||||||||||||||||||
| ┃ |
|
| ||||||||||||||||||||||||||||||||
| 0 | 0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |||
| 31 | 30 | 23 | 22 | 0 | ||||||||||||||||||||||||||||||
The bfloat16 binary floating-point exponent is encoded using an offset-binary representation, with the zero offset being 127; also known as exponent bias in the IEEE 754 standard.
Thus, in order to get the true exponent as defined by the offset-binary representation, the offset of 127 has to be subtracted from the value of the exponent field.
The minimum and maximum values of the exponent field (00H and FFH) are interpreted specially, like in the IEEE 754 standard formats.
| Exponent | Significand zero | Significand non-zero | Equation |
|---|---|---|---|
| 00H | zero, −0 | subnormal numbers | (−1)signbit×2−126× 0.significandbits |
| 01H, ..., FEH | normalized value | (−1)signbit×2exponentbits−127× 1.significandbits | |
| FFH | ±infinity | NaN (quiet, signaling) | |
The minimum positive normal value is 2−126 ≈ 1.18 × 10−38 and the minimum positive (subnormal) value is 2−126−7 = 2−133 ≈ 9.2 × 10−41.
The most common use case is the conversion between IEEE 754 binary32 and bfloat16. The following section describes the conversion process and its rounding scheme in the conversion. Note that there are other possible scenarios of format conversions to or from bfloat16. For example, int16 and bfloat16.
Just as in IEEE 754, positive and negative infinity are represented with their corresponding sign bits, all 8 exponent bits set (FFhex) and all significand bits zero. Explicitly,
val s_exponent_signcnd
+inf = 0_11111111_0000000
-inf = 1_11111111_0000000
Just as in IEEE 754, NaN values are represented with either sign bit, all 8 exponent bits set (FFhex) and not all significand bits zero. Explicitly,
val s_exponent_signcnd
+NaN = 0_11111111_klmnopq
-NaN = 1_11111111_klmnopq
where at least one of k, l, m, n, o, p, or q is 1. As with IEEE 754, NaN values can be quiet or signaling, although there are no known uses of signaling bfloat16 NaNs.
Bfloat16 is designed to maintain the number range from the 32-bit IEEE 754 single-precision floating-point format (binary32), while reducing the precision from 24 bits to 8 bits. This means that the precision is between two and three decimal digits, and bfloat16 can represent finite values up to about 3.4 × 1038.
These examples are given in bit representation, in hexadecimal and binary, of the floating-point value. This includes the sign, (biased) exponent, and significand.
3f80 = 0 01111111 0000000 = 1 c000 = 1 10000000 0000000 = −2
7f7f = 0 11111110 1111111 = (28 − 1) × 2−7 × 2127 ≈ 3.38953139 × 1038 (max finite positive value in bfloat16 precision) 0080 = 0 00000001 0000000 = 2−126 ≈ 1.175494351 × 10−38 (min normalized positive value in bfloat16 precision and single-precision floating point)
The maximum positive finite value of a normal bfloat16 number is 3.38953139 × 1038, slightly below (224 − 1) × 2−23 × 2127 = 3.402823466 × 1038, the max finite positive value representable in single precision.
0000 = 0 00000000 0000000 = 0 8000 = 1 00000000 0000000 = −0
7f80 = 0 11111111 0000000 = infinity ff80 = 1 11111111 0000000 = −infinity
4049 = 0 10000000 1001001 = 3.140625 ≈ π ( pi ) 3eab = 0 01111101 0101011 = 0.333984375 ≈ 1/3
ffc1 = x 11111111 1000001 => qNaN ff81 = x 11111111 0000001 => sNaN
Google invented its own internal floating point format called "bfloat" for "brain floating point" (after Google Brain).
This custom floating point format is called "Brain Floating Point Format," or "bfloat16" for short. The name flows from "Google Brain", which is an artificial intelligence research group at Google where the idea for this format was conceived.
The bfloat16 standard is a targeted way of representing numbers that give the range of a full 32-bit number, but in the data size of a 16-bit number, keeping the accuracy close to zero but being a bit more loose with the accuracy near the limits of the standard. The bfloat16 standard has a lot of uses inside machine learning algorithms, by offering better accuracy of values inside the algorithm while affording double the data in any given dataset (or doubling the speed in those calculation sections).
...Intel will be extending bfloat16 support across our AI product lines, including Intel Xeon processors and Intel FPGAs.
Intel plans to support this format across all their AI products, including the Xeon and FPGA lines
Intel said that the NNP-L1000 would also support bfloat16, a numerical format that's being adopted by all the ML industry players for neural networks. The company will also support bfloat16 in its FPGAs, Xeons, and other ML products. The Nervana NNP-L1000 is scheduled for release in 2019.
This page lists the TensorFlow Python APIs and graph operators available on Cloud TPU.
For the Cloud TPU, Google recommended we use the bfloat16 implementation from the official TPU repository with TensorFlow 1.7.0. Both the TPU and GPU implementations make use of mixed-precision computation on the respective architecture and store most tensors with half-precision.
All operations in TensorFlow Distributions are numerically stable across half, single, and double floating-point precisions (as TensorFlow dtypes: tf.bfloat16 (truncated floating point), tf.float16, tf.float32, tf.float64). Class constructors have a validate_args flag for numerical asserts
In many models this is a drop-in replacement for float-32
On TPU, the rounding scheme in the conversion is round to nearest even and overflow to inf.
Uses the non-IEEE Round-to-Odd rounding mode.
Converts float number to nv_bfloat16 precision in round-to-nearest-even mode and returns nv_bfloat16 with converted value.
Source: Wikipedia. Article content is retrieved live through the MediaWiki API.
The bfloat16 (brain floating point) floating-point format is a computer number format occupying 16 bits in computer memory; it represents a wide dynamic range of numeric values by using a floating radix point. This format is a shortened (16-bit) version of the 32-bit IEEE 754 single-precision floating-point format (binary32) with the intent of accelerating machine learning and near-sensor computing. It preserves the approximate dynamic range of 32-bit floating-point numbers by retaining 8 exponent bits, but supports only an 8-bit precision rather than the 24-bit significand of the binary32 format. More so than single-precision 32-bit floating-point numbers, bfloat16 numbers are unsuitable for integer calculations, but this is not their intended use. Bfloat16 is used to reduce the storage requirements and increase the calculation speed of machine learning algorithms. The bfloat16 format was developed by Google Brain, an artificial intelligence research group at Google, for use in its TPU v2, released in 2017. It is utilized in many CPUs, GPUs, and AI processors, such as Intel Xeon processors (AVX-512 BF16 extensions), Intel Data Center GPU, Intel Nervana NNP-L1000, Intel FPGAs, AMD Zen, AMD Instinct, NVIDIA GPUs, Google Cloud TPUs, AWS Inferentia, AWS Trainium, ARMv8.6-A, and Apple's M2 and therefore A15 chips and later. Many libraries support bfloat16, such as CUDA, Intel oneAPI Math Kernel Library, AMD ROCm, AMD Optimizing CPU Libraries, PyTorch, and TensorFlow. On these platforms, bfloat16 may also be used in mixed-precision arithmetic, where bfloat16 numbers may be operated on and expanded to wider data types.
TensorFloat-32 (TF32) is a numeric floating point format designed for Tensor Core running on certain Nvidia GPUs. It was first implemented in the Ampere architecture. TensorFloat-32 combines the 8-bit exponent size of IEEE single precision with the 10-bit mantissa size of half precision for a total of 19 bits per number. It is comparable to the bfloat16 format, which uses a 7-bit mantissa.
Ampere is the codename for a graphics processing unit (GPU) microarchitecture developed by Nvidia as the successor to both the Volta and Turing architectures. It was officially announced on May 14, 2020, and is named after French mathematician and physicist André-Marie Ampère. Nvidia announced the Ampere architecture GeForce 30 series consumer GPUs at a GeForce Special Event on September 1, 2020. Nvidia announced the A100 80 GB GPU at SC20 on November 16, 2020. Mobile RTX graphics cards and the RTX 3060 based on the Ampere architecture were revealed on January 12, 2021. Nvidia announced Ampere's successor, Hopper, at GTC 2022, and "Ampere Next Next" (Blackwell) for a 2024 release at GPU Technology Conference 2021.
In computing, floating-point arithmetic (FP) is arithmetic on subsets of real numbers formed by a significand (a signed sequence of a fixed number of digits in some base) multiplied by an integer power of that base. Numbers of this form are called floating-point numbers. For example, the number 2469/200 is a floating-point number in base ten with five digits: 2469 / 200 = 12.345 = 12345 ⏟ significand × 10 ⏟ base − 3 ⏞ exponent {\displaystyle 2469/200=12.345=\!\underbrace {12345} _{\text{significand}}\!\times \!\underbrace {10} _{\text{base}}\!\!\!\!\!\!\!\overbrace {{}^{-3}} ^{\text{exponent}}} However, 7716/625 = 12.3456 is not a floating-point number in base ten with five digits—it needs six digits. The nearest floating-point number with only five digits is 12.346. And 1/3 = 0.3333… is not a floating-point number in base ten with any finite number of digits. In practice, most floating-point systems use base two, though base ten (decimal floating point) is also common. Floating-point arithmetic operations, such as addition and division, approximate the corresponding real number arithmetic operations by rounding any result that is not a floating-point number itself to a nearby floating-point number. For example, in a floating-point arithmetic with five base-ten digits, the sum 12.345 + 1.0001 = 13.3451 might be rounded to 13.345. The term floating point refers to the fact that the number's radix point can "float" anywhere to the left, right, or between the significant digits of the number. This position is indicated by the exponent, so floating point can be considered a form of scientific notation. A floating-point system can be used to represent, with a fixed number of digits, numbers of very different orders of magnitude — such as the number of meters between galaxies or between protons in an atom. For this reason, floating-point arithmetic is often used to allow very small and very large real numbers that require fast processing times. The result of this dynamic range is that the numbers that can be represented are not uniformly spaced; the difference between two consecutive representable numbers varies with their exponent. Over the years, a variety of floating-point representations have been used in computers. In 1985, the IEEE 754 Standard for Floating-Point Arithmetic was established, and since the 1990s, the most commonly encountered representations are those defined by the IEEE. The speed of floating-point operations, commonly measured in terms of FLOPS, is an important characteristic of a computer system, especially for applications that involve intensive mathematical calculations. Floating-point numbers can be computed using software implementations (softfloat) or hardware implementations (hardfloat). Floating-point units (FPUs, colloquially math coprocessors) are specially designed to carry out operations on floating-point numbers and are part of most computer systems. When FPUs are not available, software implementations can be used instead.
Before the 1921 destruction of Tulsa’s Greenwood District, Black residents had created a remarkable center of business and community life. The district included stores, professional offices, entertainment venues and homes owned by Black citizens. Understanding Greenwood means learning what was built—not only what was burned.
MORE →Madam C.J. Walker