Lecture06-Arithmetic Code-2-Algorithm Implementation-P2.pdf

1
Lih-Jen Kau
Signal Proc. & Intell. Electron. Group
National Taipei Univ. of Technology
1/46
Lecture 06
Nov. 03, 2022
Instructor：高立人
電子工程研究所
國立臺北科技大學
November 3, 2022
Lih-Jen Kau
2/46
Arithmetic Code
Algorithm Implementation
November 3, 2022

2
Lih-Jen Kau
3/46
 Algorithm Implementation: An Overview
 In previous section, i.e., Section 4.3.1, we developed a
recursive algorithm for the boundaries of the interval
containing the tag for the sequence being encoded as
       
   
       
    (4.23)
(4.22)
1
1
1
1
1
1
1
n
X
n
n
n
n
n
X
n
n
n
n
x
F
l
u
l
u
x
F
l
u
l
l













where, xn is the value of the random variable
corresponding to the nth observed symbol, l(n) is the
lower limit of the tag interval at the nth iteration, and u(n)
is the upper limit of the tag interval at the nth iteration.
November 3, 2022
Lih-Jen Kau
4/46
 Before we can implement this algorithm, there is one
major problem we have to resolve.
 Recall that the reason for using numbers in the interval [0, 1) as
a tag was that there are an infinite number of numbers in this
interval.
 However, in practice the number of numbers that can be
uniquely represented on a machine is limited by the maximum
number of digits (or bits) we can use for representing the
number.
November 3, 2022

3
Lih-Jen Kau
5/46
 Consider the values of l(n) and u(n) in Example 4.3.5.
 As n gets larger, these values come closer and closer together.
 This means that in order to represent all the subintervals
uniquely we need increasing precision as the length of the
sequence increases.
 In a system with finite precision, the two values are bound to
converge, and we will lose all information about the sequence
from the point at which the two values converged.
November 3, 2022
Lih-Jen Kau
6/46
 To avoid this situation, we need to rescale the interval.
However, we have to do it in a way that will preserve the
information that is being transmitted.
 We would also like to perform the encoding
incrementally — that is, to transmit portions of the code
as the sequence is being observed, rather than wait
until the entire sequence has been observed before
transmitting the first bit.
 The algorithm we describe in this section takes care of
the problems of synchronized rescaling and incremental
encoding.
November 3, 2022

4
Lih-Jen Kau
7/46
 Interval Containing the Tag: An Analysis
 As the interval becomes narrower, we have three
possibilities:
 The interval is entirely confined to the lower half of the unit
interval [0, 0.5).（全部落在下半部）
 The interval is entirely confined to the upper half of the unit
interval [0.5, 1.0).（全部落在上半部）
 The interval straddles（跨坐）the midpoint of the unit interval.
（亦即跨坐於0.5的兩端；下限在0.5之下，上限在0.5之上）
 We will look at the third case a little later in this section.
November 3, 2022
Lih-Jen Kau
8/46
 First, let us examine the first two cases.
 Once the interval is confined to either the upper or lower half of
the unit interval, it is forever confined to that half of the unit
interval.
 The most significant bit of the binary representation of all
numbers in the interval [0, 0.5) is 0, and the most significant bit
of the binary representation of all numbers in the interval [0.5, 1]
is 1.
 Therefore, once the interval gets restricted to either the upper or
lower half of the unit interval, the most significant bit of the tag is
fully determined.
November 3, 2022

5
Lih-Jen Kau
9/46
 Therefore, without waiting to see what the rest of the sequence
looks like, we can indicate to the decoder whether the tag is
confined to the upper or lower half of the unit interval by sending
a 1 for the upper half and a 0 for the lower half.
 The bit that we send is also the first bit of the tag.
November 3, 2022
Lih-Jen Kau
10/46
 Once the encoder and decoder know which half
contains the tag, we can ignore the half of the unit
interval not containing the tag and concentrate on the
half containing the tag.
 As our arithmetic is of finite precision, we can do this
best by mapping the half interval containing the tag to
the full [0, 1) interval. The mappings required are
     
        (4.25)
.
5
.
0
2
;
1
,
0
1
,
5
.
0
:
(4.24)
2
;
1
,
0
5
.
0
,
0
:
2
2
1
1





x
x
E
E
x
x
E
E
November 3, 2022

6
Lih-Jen Kau
11/46
 As soon as we perform either of these mappings, we
lose all information about the most significant bit.
 However, this should not matter because we have
already sent that bit to the decoder.
 We can now continue with this process, generating
another bit of the tag every time the tag interval is
restricted to either half of the unit interval.
 This process of generating the bits of the tag without
waiting to see the entire sequence is called incremental
encoding.
November 3, 2022
Lih-Jen Kau
12/46
 Tag Generation with Scaling: Example 4.4.2
 Let's revisit Example 4.3.5.
 Recall that we wish to encode the sequence 1 3 2 1.
The probability model for the source is P(a1)=0.8, P(a2)=
0.02, P(a3)=0.18, i.e., Fx(0)=0, Fx(1)=0.8, Fx(2)=0.82, Fx(3)=1.
 Initializing u(0) to 1, and l(0) to 0, the first element of the
sequence, 1, results in the following update:
 
 
 
   .
8
.
0
8
.
0
0
1
0
0
0
0
1
0
1
1








u
l
 The interval [0, 0.8) is not confined to either the upper or
lower half of the unit interval, so we proceed.
November 3, 2022

7
Lih-Jen Kau
13/46
 Tag Generation with Scaling: Example 4.4.2 Cont.
 The second element of the sequence is 3. This results
in the update
 
   
 
    .
8
.
0
0
.
1
8
.
0
3
0
8
.
0
0
656
.
0
82
.
0
8
.
0
2
0
8
.
0
0
2
2












X
X
F
u
F
l
 The interval [0.656, 0.8) is contained entirely in the
upper half of the unit interval, so we send the binary
code 1 and rescale:
 
 
 
  .
6
.
0
5
.
0
8
.
0
2
312
.
0
5
.
0
656
.
0
2
2
2








u
l
November 3, 2022
Lih-Jen Kau
14/46
 The third element, 2, results in the following update
equations:
 The interval for the tag is [0.5424, 0.54816), which is
contained entirely in the upper half of the unit interval.
We transmit a 1 and go through another rescaling:
 
   
 
    .
54816
.
0
82
.
0
288
.
0
312
.
0
2
312
.
0
8
.
0
312
.
0
5424
.
0
8
.
0
288
.
0
312
.
0
1
312
.
0
6
.
0
312
.
0
3
3














X
X
F
u
F
l
 
 
 
  .
09632
.
0
5
.
0
54816
.
0
2
0848
.
0
5
.
0
5424
.
0
2
3
3








u
l
November 3, 2022

8
Lih-Jen Kau
15/46
 This interval is contained entirely in the lower half of the
unit interval, so we send a 0 and use the E1 mapping to
rescale:
 
 
 
  .
19264
.
0
09632
.
0
2
1696
.
0
0848
.
0
2
3
3






u
l
 The interval is still contained entirely in the lower half of
the unit interval, so we send another 0 and go through
another rescaling:
 
 
 
  .
38528
.
0
1696
.
0
2
3392
.
0
1696
.
0
2
3
3






u
l
November 3, 2022
Lih-Jen Kau
16/46
 Because the interval containing the tag remains in the
lower half of the unit interval, we send another 0 and
rescale one more time:
 Now the interval containing the tag is contained entirely
in the upper half of the unit interval. Therefore, we
transmit a 1 and rescale using the E2 mapping:
 
 
 
  .
77056
.
0
38528
.
0
2
6784
.
0
3392
.
0
2
3
3






u
l
 
 
 
  .
54112
.
0
5
.
0
77056
.
0
2
3568
.
0
5
.
0
6784
.
0
2
3
3








u
l
November 3, 2022

9
Lih-Jen Kau
17/46
 At each stage we are transmitting the most significant
bit that is the same in both the upper and lower limit of
the tag interval.
 If the most significant bits in the upper and lower limit
are the same, then the value of this bit will be identical
to the most significant bit of the tag.
 Therefore, by sending the most significant bits of the
upper and lower endpoint of the tag whenever they are
identical, we are actually sending the binary
representation of the tag.
 The rescaling operations can be viewed as left shifts, which
make the second most significant bit the most significant bit.
November 3, 2022
Lih-Jen Kau
18/46
 Continuing with the last element, the upper and lower
limits of the interval containing the tag are
 
   
 
    .
504256
.
0
8
.
0
18422
.
0
3568
.
0
1
3568
.
0
54112
.
0
3568
.
0
3568
.
0
0
.
0
18422
.
0
3568
.
0
0
3568
.
0
54112
.
0
3568
.
0
4
4














X
X
F
u
F
l
November 3, 2022

10
Lih-Jen Kau
19/46
 Encoding Termination: How to
 At this point, if we wished to stop encoding, all we need
to do is inform the receiver of the final status of the tag
value.
 We can do so by sending the binary representation of
any value in the final tag interval.
 Generally, this value is taken to be l(n).
 In this particular example, it is convenient to use the value of 0.5.
 The binary representation of 0.5 is .10....
 Thus, we would transmit a 1 followed by as many 0s as
required by the word length of the implementation being used.
（補0不影響數值的大小）
November 3, 2022
Lih-Jen Kau
20/46
 Notice that the tag interval size at this stage is
approximately 64 times the size it was when we were
using the unmodified algorithm.（和Example 4.3.5比較）
 Therefore, this technique solves the finite precision
problem.
 As we shall soon see, the bits that we have been
sending with each mapping constitute the tag itself,
which satisfies our desire for incremental encoding.
 
 
64
7712
.
0
773504
.
0
3568
.
0
504256
.
0


 相較於未做Scaling之演算法，
Scale方法讓區間放大了64倍！
November 3, 2022

11
Lih-Jen Kau
21/46
 The binary sequence generated during the encoding
process in the previous example is 1100011.
 We could simply treat this as the binary expansion of the tag.
 A binary number .1100011 corresponds to the decimal number
0.7734375.
 Looking back to Example 4.3.5, notice that this number lies
within the final tag interval. Therefore, we could use this to
decode the sequence.
November 3, 2022
Lih-Jen Kau
22/46
 Incremental Decoding:
 However, we would like to do incremental decoding as
well as incremental encoding. This raises three
questions:
 How do we start decoding?
 How do we continue decoding?
 How do we stop decoding?
 The second question is the easiest to answer.
 Once we have started decoding, all we have to do is
mimic the encoder algorithm. That is, once we have
started decoding, we know how to continue decoding.
November 3, 2022

12
Lih-Jen Kau
23/46
 Incremental Decoding: Cont.
 To begin the decoding process, we need to have
enough information to decode the first symbol
unambiguously.
 In order to guarantee unambiguous decoding, the
number of bits received should point to an interval
smaller than the smallest tag interval（這是指解碼開始
時至少必須有多少位元方可識別第一個符號的區間）.
 Based on the smallest tag interval, we can determine how many
bits we need before we start the decoding procedure.
 We will demonstrate this procedure in Example 4.4.4.
 First let's look at other aspects of decoding using the message
from Example 4.4.2.
November 3, 2022
Lih-Jen Kau
24/46
 Example 4.4.3: Deciphering with Scaling
 We will use a word length of 6 for this example（在本例
中，先取出從接收端所收到的前6個位元來解第1個符號；在
Integer Implementation時會再提及該選取多少位元之方法）.
 Note that because we are dealing with real numbers this
word length may not be sufficient for a different
sequence.
 As in the encoder, we start with initializing u(0) to 1 and
l(0) to 0.
 The sequence of received bits is 110001100. . . 0.
November 3, 2022

13
Lih-Jen Kau
25/46
 Example 4.4.3: Deciphering with Scaling Cont.
 The first 6 bits, i.e., 110001, correspond to a tag value of
0.765625, which means that the first element of the
sequence is 1（因為0.76562落在0 ~ 0.8之間，所以解出第
一個符號為1）, resulting in the following update:
 
 
 
   .
8
.
0
8
.
0
0
1
0
0
0
0
1
0
1
1








u
l
November 3, 2022
Lih-Jen Kau
26/46
 The interval [0,0.8) is not confined to either the upper or
lower half of the unit interval, so we proceed.
 The tag 0.765625 lies in the top 18% of the interval [0,
0.8); therefore, the second element of the sequence is 3.
 Updating the tag interval we get
 
   
 
    .
8
.
0
0
.
1
8
.
0
3
0
8
.
0
0
656
.
0
82
.
0
8
.
0
2
0
8
.
0
0
2
2












X
X
F
u
F
l
November 3, 2022

14
Lih-Jen Kau
27/46
 The interval [0.656, 0.8) is contained entirely in the
upper half of the unit interval.
 At the encoder, we sent the bit 1 and rescaled.
 At the decoder, we will shift 1 out of the receive buffer
and move the next bit in to make up the 6 bits in the tag
（這6個bit構成了新的Tag）.
 We also update the tag interval, resulting in
 
 
 
  .
6
.
0
5
.
0
8
.
0
2
312
.
0
5
.
0
656
.
0
2
2
2








u
l
while shifting a bit to give us a tag of 0.546875.
November 3, 2022
Lih-Jen Kau
28/46
 Now, the tag value is 0.546875.
 When we compare this value with the tag interval, we
can see that this value lies in the 80-82% range of the
tag interval, so we decode the next element of the
sequence as 2.（See the following for explanation!）
 
   
 
    .
54816
.
0
82
.
0
288
.
0
312
.
0
2
312
.
0
8
.
0
312
.
0
5424
.
0
8
.
0
288
.
0
312
.
0
1
312
.
0
6
.
0
312
.
0
3
3














X
X
F
u
F
l
 We can then update the equations for the tag interval as
815538
.
0
0.312
-
0.6
0.312
-
0.546875

November 3, 2022

15
Lih-Jen Kau
29/46
 As the tag interval is now contained entirely in the upper
half of the unit interval, we rescale using E2 to obtain
 
 
 
  .
09632
.
0
5
.
0
54816
.
0
2
0848
.
0
5
.
0
5424
.
0
2
3
3








u
l
 We also shift out a bit from the tag and shift in the next
bit.
 The tag is now 000110.
November 3, 2022
Lih-Jen Kau
30/46
 The interval is contained entirely in the lower half of the
unit interval. Therefore, we apply E1 and shift another
bit.
 The lower and upper limits of the tag interval become
 
 
 
  .
19264
.
0
09632
.
0
2
1696
.
0
0848
.
0
2
3
3






u
l
and the tag becomes 001100.
November 3, 2022

16
Lih-Jen Kau
31/46
 The interval is still contained entirely in the lower half of
the unit interval, so we shift out another 0 to get a tag of
011000 and go through another rescaling:
 
 
 
  .
38528
.
0
1696
.
0
2
3392
.
0
1696
.
0
2
3
3






u
l
 Because the interval containing the tag remains in the
lower half of the unit interval, we shift out another 0 from
the tag to get 110000 and rescale one more time:
 
 
 
  .
77056
.
0
38528
.
0
2
6784
.
0
3392
.
0
2
3
3






u
l
November 3, 2022
Lih-Jen Kau
32/46
 Now the interval containing the tag is contained entirely
in the upper half of the unit interval. Therefore, we shift
out a 1 from the tag and rescale using the E2 mapping:
 
 
 
  .
54112
.
0
5
.
0
77056
.
0
2
3568
.
0
5
.
0
6784
.
0
2
3
3








u
l
 Now we compare the tag value to the tag interval to
decode our final element.
 The tag is 100000, which corresponds to 0.5.
 This value lies in the first 80% of the interval, so we
decode this element as 1.
November 3, 2022

17
Lih-Jen Kau
33/46
Short Conclusions About Scaling
 Deciphering with Scaling: A Short Conclusion
 If the tag interval is entirely contained in the upper or
lower half of the unit interval, the scaling procedure
described will prevent the interval from continually
shrinking.
 Now we consider the case where the diminishing tag
interval straddles the midpoint of the unit interval.
November 3, 2022
Lih-Jen Kau
34/46
 As our trigger for rescaling, we check to see if the tag
interval is contained in the interval [0.25, 0.75). This will
happen when l(n) is greater than or equal to 0.25 and u(n)
is less than 0.75, i.e., we check to see if the following
condition holds.
 l(n) : 0.01XXX
 u(n): 0.10XXX
 When this happens, we double the tag interval using the
following mapping:
        (4.26)
.
25
.
0
2
;
1
,
0
75
.
0
,
25
.
0
: 3
3 

 x
x
E
E
November 3, 2022

18
Lih-Jen Kau
35/46
 We have used a 1 to transmit information about an E2
mapping, and a 0 to transmit information about an E1,
mapping.
 How do we transfer information about an E3 mapping to
the decoder?
 We use a somewhat different strategy in this case.
 At the time of the E3 mapping, we do not send any information
to the decoder; instead, we simply record the fact that we have
used the E3 mapping at the encoder.
November 3, 2022
Lih-Jen Kau
36/46
 Suppose that after this, the tag interval gets confined to
the upper half of the unit interval.（這是指假使沒有做E3
mapping，且其後出現的符號將區間帶至0.5以上的情況）
 At this point we would use an E2 mapping and send a 1 to the
receiver.
 Note that the tag interval at this stage is at least twice what it
would have been if we had not used the E3 mapping.
Furthermore, the upper limit of the tag interval would have been
less than 0.75.（這是指假使沒有做E3 mapping的情況）
 Therefore, if the E3 mapping had not taken place right before
the E2 mapping, the tag interval would have been contained
entirely in the lower half of the unit interval.（這是指做完E2
mapping後的情況）
November 3, 2022

19
Lih-Jen Kau
37/46
 At this point we would have used an E1 mapping and
transmitted a 0 to the receiver.
 In fact, the effect of the earlier E3 mapping can be mimicked at
the decoder by following the E2 mapping with an E1 mapping.
 At the encoder, right after we send a 1 to announce the E2
mapping, we send a 0 to help the decoder track the changes in
the tag interval at the decoder.（這是指假使做了E3 mapping後
的情況）
 If the first rescaling after the E3 mapping happens to be
an E1 mapping, we do exactly the opposite. That is, we
follow the 0 announcing an E1 mapping with a 1 to
mimic the effect of the E3 mapping at the encoder.
November 3, 2022
Lih-Jen Kau
38/46
 Deciphering with Scaling: A Short Conclusion Cont.
 What happens if we have to go through a series of E3
mappings at the encoder?
 We simply keep track of the number of E3 mappings and then
send that many bits of the opposite variety after the first E1 or
E2 mapping.
 If we went through three E3 mappings at the encoder, followed
by an E2 mapping, we would transmit a 1 followed by three 0s.
 On the other hand, if we went through an E1 mapping after
three E3 mappings, we would transmit a 0 followed by three 1s.
 Since the decoder mimics the encoder, the E3 mappings are
also applied at the decoder when the tag interval is contained in
the interval [0.25, 0.75).
November 3, 2022

20
Lih-Jen Kau
39/46
 Comments on E3 mapping:
 第36頁，37頁這一段話的意思在於：當E3 Mapping過後，若
下一個Symbol造成整個Interval落在上半部，則我們會做E2
Mapping！
 此外，若未做E3 Mapping，則Upper Bound不會超過0.75。
（原來的Upper Bound就已經小於0.75了，在未做放大的情況
下，後來的區間只會愈來愈小，不可能超過前一次的上限）
 因此做完E2 Mapping之後一定落在下半部！（因為E2
Mapping會減掉0.5再乘以2）
Comments on Scaling
November 3, 2022
Lih-Jen Kau
40/46
Comments on Scaling
 Questions:
 Q1: E3 Mapping過後若有E1 Mapping出現為何要送01?
 Q2: E3 Mapping過後若有E2 Mapping出現為何要送10?
 Discussions:
 若E3 Mapping之後，new symbol added造成Interval跑
到上半部（i.e., 0.5以上），接著便可做E2 Mapping，送
出1。
 問：為何還要再送一個0 ?
 Pls see next page for explanations.
November 3, 2022

21
Lih-Jen Kau
41/46
Comments on Scaling
l(n)
u(n)
0
0.5
1
0.25
0.75
0.11
0.10
0.01
0
0.5
0.25
0.75
l(n+1)
u(n+1)
l’(n): 假設做過E3 Mapping之後下限來到這裡
u’(n):假設做過E3 Mapping之後上限來到這裡
假設未做E3 Mapping，New added symbol
將Interval之上限帶到這裡（上限一定小於
0.75，but why? ）
因為u(n)就已經小於0.75了，後來的Interval
只有愈來愈窄的份！
 Comments on E3 mapping: Cont.
November 3, 2022
Lih-Jen Kau
42/46
Comments on Scaling
 首先考慮未做E3 Mapping的情況（咖啡色邊界符號）
 這個情況已經很明顯了，u(n+1)和l(n+1)都在Upper Interval，故
送出1！
 且u(n+1) <0.75，故E2 Mapping後，Interval必定小於0.5；因此
又可再送出0！
 Now the problem comes to be what happens if we had
applied E3 Mapping?
 The induction process will be shown in next page.
November 3, 2022

22
Lih-Jen Kau
43/46
Comments on Scaling
 其次考慮使用E3 Mapping的情況（紅色邊界符號）
   
 
   
 
25
.
0
2
25
.
0
2






n
l
n
l
n
u
n
u
This is what we get if E3 Mapping had been applied!
 先考慮l’(n+1)：亦即新的符號進入後
       
   
     
   
     
   
 
  5
.
0
1
2
5
.
0
1
2
1
2
5
.
0
2
1
1
1
1
1

























n
l
x
F
n
l
n
u
n
l
x
F
n
l
n
u
n
l
x
F
n
l
n
u
n
l
n
l
n
n
n
November 3, 2022
Lih-Jen Kau
44/46
Comments on Scaling
 In summary, we can write down
   
 
   
 
5
.
0
1
2
1
1
5
.
0
1
2
1
1










n
u
n
u
n
l
n
l
 Consider the following two cases:
 Case 1: 若新的符號將做過E3 Mapping的Interval帶至上半部，
則即便未做E3 Mapping，同樣的符號也會將新的Interval帶至上
半部。
 Case 2: 若新的符號將做過E3 Mapping的Interval帶至下半部，
則即便未做E3 Mapping，同樣的符號也會將新的Interval帶至下
半部。
 See next page for explanation.
November 3, 2022

23
Lih-Jen Kau
45/46
Comments on Scaling
 Case 1: 若新的符號將做過E3 Mapping的Interval帶至上半部
 
  











1
.
0
1
1
.
0
1
n
u
n
l  
  













0
.
1
5
.
0
1
0
.
1
5
.
0
1
n
u
n
l
 
 
 
  













10
.
0
5
.
0
1
2
1
10
.
0
5
.
0
1
2
1
n
u
n
l
   
 
   
 
5
.
0
1
2
1
1
5
.
0
1
2
1
1










n
u
n
u
n
l
n
l
 即便未做E3 Mapping，新的符號也會將新的Interval帶至上半部。
 因此E3 Mapping過後若發生E2 Mapping，要先送1再送0！
November 3, 2022
Lih-Jen Kau
46/46
Comments on Scaling
 Case 2: 若新的符號將做過E3 Mapping的Interval帶至下半部
 
  











0
.
0
1
0
.
0
1
n
u
n
l  
  













1
.
0
5
.
0
1
1
.
0
5
.
0
1
n
u
n
l
 
 
 
  













01
.
0
5
.
0
1
2
1
01
.
0
5
.
0
1
2
1
n
u
n
l
   
 
   
 
5
.
0
1
2
1
1
5
.
0
1
2
1
1










n
u
n
u
n
l
n
l
 即便未做E3 Mapping，新的符號也會將新的Interval帶至下半部。
 因此E3 Mapping過後若發生E1 Mapping，要先送0再送1！
November 3, 2022

Lecture06-Arithmetic Code-2-Algorithm Implementation-P2.pdf

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a Lecture06-Arithmetic Code-2-Algorithm Implementation-P2.pdf

Semelhante a Lecture06-Arithmetic Code-2-Algorithm Implementation-P2.pdf (20)

Mais de ssuserb4d806

Mais de ssuserb4d806 (20)

Último

Último (20)

Lecture06-Arithmetic Code-2-Algorithm Implementation-P2.pdf