Low-Bit Computing Opens New Paths to Greater Large-Model Efficiency
ICMEChallenge

IEEE ICME 2026 Low-Bit Large-Model Challenge Workshop Held in Bangkok
On the afternoon of July 5, the IEEE ICME 2026 Workshop on the Low-Bit Large-Model Efficient Computing Challenge took place in Bangkok, Thailand. Focusing on frontier topics including low-bit quantization of large models and video generation model inference and training, the workshop brought together outstanding teams from universities, research institutes, and industry to share their challenge results.

As video generation models, large multimodal models, and large language models continue to grow, deployment faces high accelerator memory usage, slow inference, and expensive training. Low-bit computing offers an important way to address these challenges. Yun Zhao, representing the GCC-HiFloat community, opened with an overview of the IEEE ICME Low-Bit Large-Model Quantization Challenge. Seven award-winning teams then presented their work on HiFloat and related approaches.

HUST_YAO: Making Quantization Aware of Diffusion Timesteps

HUST_YAO addressed W4A4 quantization of Wan2.2-I2V. Their core approach, Timestep-Aware SVDQuant-GPTQ, uses the low-rank branch in SVDQuant to absorb activation outliers while dividing diffusion denoising into early, middle, and late stages to search for suitable activation clipping ratios for different layers and stages. For weight quantization, the team further used GPTQ to reduce reconstruction error in 4-bit weights.
The innovation introduces timestep awareness into video diffusion model quantization, allowing quantization parameters to adapt dynamically throughout generation. The team argued that quantizing video generation models requires attention to differences between denoising stages, module sensitivity, and changes in visual quality, beyond aggregate statistics.
BITS PLEASE: Stabilizing 4-Bit Video Generation with Smoothing, Rotation, and Calibration

BITS PLEASE summarized its approach in four steps: apply SmoothQuant for channel-wise balancing to reduce the impact of outliers on quantization ranges; redistribute activation energy through Hadamard rotations to make data easier to represent at low bit widths; build conditional calibration tables for timesteps, classifier-free guidance (CFG) branches, and high- and low-noise experts; and finally perform W4A4 inference with HiFloat4.
Experiments showed that HiSQRot4 largely preserved full-precision quality across multiple VBench metrics, with improvements in image quality and subject consistency. The team concluded that low-bit quantization of video diffusion models must explicitly account for timestep variation, conditional branches, and differences between noise experts. A single global quantization range is insufficient for high-quality video generation.
LOW TEAM: Protecting Critical Boundary Layers with Modular HiF8 Quantization

LOW TEAM focused on W8A8 HiFloat8 quantization of Wan2.1-T2V-14B and proposed a Boundary-Protection strategy. Analysis of activation distributions showed that several modules at the beginning and end of the model behave differently from intermediate modules: errors at the input propagate through residual connections, while errors at the output directly affect visual quality after VAE decoding.
Based on these findings, the team retained five critical boundary blocks in BF16 and quantized the remaining blocks to W8A8 HiF8. Rather than applying uniform quantization, the approach protects the most sensitive input and output modules to balance efficiency and quality. This post-training quantization (PTQ) approach matched or slightly exceeded the BF16 baseline on four VBench metrics. The team also explored quantization-aware training (QAT), which did not necessarily outperform PTQ under single-accelerator constraints. Quantization optimization therefore depends on hardware, training strategies, and engineering constraints as well as algorithms.
USTC-ZHASLAB: Moving from Maximum-Based Calibration to Tail Awareness

USTC-ZHASLAB presented Tail-Aware HiFloat4: W4A4 Post-Training Quantization for Wan2.2. Activations often exhibit long-tailed distributions. A small number of extreme outliers can expand the quantization range, leaving most ordinary values represented by coarse quantization intervals and degrading generation quality.
Drawing on ViDiT-Q, the team adapted the approach to HiFloat4 W4A4 and replaced conventional maximum-based calibration with tail-aware percentile calibration. This prevents a handful of outliers from determining the entire quantization range, giving more common values finer representation. The approach preserved image quality and overall consistency, with only a slight decline in motion smoothness. The team emphasized that low-bit quantization must consider where outliers originate and which quality dimensions are most vulnerable, alongside compression ratios.
PACMAN: Combining SmoothQuant and MixQ to Protect Outlier Channels

PACMAN proposed a W4A4 inference quantization approach for Wan2.2-I2V-A14B, adapting ideas from large language model quantization. SmoothQuant balances channels globally by redistributing dynamic range between activations and weights, while MixQ retains a small number of outlier channels in a higher-bit branch and processes most channels with W4A4 computation.
The approach improved most VBench metrics by approximately 2 percentage points over the native HiFloat4 reference, keeping the gap to FP16 at around 2%–3.5%. The team noted that SmoothQuant addresses global range issues while MixQ protects local outliers. Together, they better address the different failure modes of W4A4 quantization.
“梯度消失了个寂寞”: Beware of “Loss Lies”—Windowed Maxima Improve Stability

The “梯度消失了个寂寞” team discussed HiF8 W8A8 quantization-aware training, focusing on how to estimate quantization scales during training, particularly how delayed scaling can avoid saturation and clipping when current activation peaks suddenly exceed historical scales.
The team compared several amax estimation strategies, including most_recent, exp_smooth, current scaling, and max-window. Its final approach combined a 64-step windowed maximum, 500 steps of BF16 warmup, and a lower learning rate. The innovation jointly targets scale stability and training dynamics rather than relying solely on training loss, motivating the observation that “Loss lies.” The final approach achieved nearly lossless HiF8 QAT on MMLU, HellaSwag, and ARC-Challenge. The team concluded that low-bit training must control numerical saturation, learning rates, and knowledge forgetting together; loss alone can misrepresent actual model capabilities.
IFLYBITBRAIN: HiFloat8 Enables Low-Bit Training of a 14B Video Generation Model

IFLYBITBRAIN presented research on HiFloat8-based low-bit training of the Wan2.1 14B video generation model. The team highlighted HiF8’s tapered-precision design, which allocates more significant bits to frequently occurring values near zero while providing a wider dynamic range for less frequent large values.
The team simulated HiF8 quantization at key computation points and compared several strategies. Using Huawei Ascend 910B hardware and the MindSpeed framework, it fine-tuned Wan2.1-T2V-14B for 1,000 steps. Both HiF8 strategies produced training loss curves closely aligned with the BF16 baseline, offering comparable accuracy with lower computational overhead and greater suitability for practical deployment. The team recommended coordinated advances in numerical formats, scale-update mechanisms, hardware platforms, and software stacks to bring low-bit training into production.
Lively Discussions and Exchanges
The presentations were followed by lively discussions. Representatives from participating teams, universities, research institutes, and industry explored low-bit quantization strategies, adaptation to HiFloat formats, deployment challenges for video generation models, and training and inference efficiency. Drawing on research and engineering experience, attendees raised questions and offered suggestions on suitable applications, technical limitations, and future optimization directions.




Technical Exchange Advances Practical Adoption of the Low-Bit Ecosystem
From 4-bit inference to 8-bit training, and from video generation models to large language models, the workshop showcased multiple technical paths toward efficient low-bit large-model computing. Despite different methods, the teams shared a common goal: using low-bit data formats to make large models faster, less memory-intensive, and easier to deploy while preserving model quality as much as possible.
The event showcased the challenge’s results and provided valuable case studies for further research and industrial adoption of low-bit formats such as HiFloat4 and HiFloat8. As native hardware support, software stacks, and algorithmic strategies improve, low-bit computing is poised to become key infrastructure for bringing large models from laboratories into real production environments.