Commit ea5703b9 authored by David Flynn's avatar David Flynn
Browse files

attr/m51481: fix insufficient precision in lifting quantization

This commit extends the operating range of the forward and inverse
quantization tables in order to reduce distortion at low bit rates.
parent e5ebf33b
......@@ -78,8 +78,8 @@ inline int64_t
Quantizer::quantize(int64_t x) const
{
// Forward quantisation avoids division by using the multiplicative inverse
// with 14 fractional bits.
int64_t fracBits = 14 + kFixedPointAttributeShift;
// with 18 fractional bits.
int64_t fracBits = 18 + kFixedPointAttributeShift;
// NB, the folowing offsets quantizes with a different deadzone to the
// reconstruction function.
......
......@@ -444,7 +444,7 @@ const uint32_t pcc::kMortonCode256X[256] = {
const int16_t pcc::kQpStep[6] = {161, 181, 203, 228, 256, 287};
const int32_t pcc::kQpStepRecip[6] = {26052, 23173, 20662,
18396, 16384, 14614};
const int32_t pcc::kQpStepRecip[6] = {416825, 370767, 330586,
294337, 262144, 233829};
//============================================================================
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment