This document discusses instruction-level parallelism (ILP) limitations. It covers ILP background using a MIPS example, hardware models that were studied including register renaming and branch/jump prediction assumptions. A study of ILP limitations found diminishing returns with larger window sizes and realizable processors are limited by complexity and power constraints. Simultaneous multithreading was explored as a technique to improve ILP but has its own design challenges. Today, x86 and ARM processors employ various ILP optimizations within pipeline constraints.
7. I. ILP: Structural Hazards
Conflict over the use of resources
Time (clock cycles)
I$
Load
Instr 1
Instr 2
Instr 3
Instr 4
I$ Reg D$ Reg
ALU
I$ Reg D$ Reg
ALU
I$ Reg D$ Reg
ALU
Reg D$ Reg
ALU
I$ Reg D$ Reg
ALU
8. I. ILP: Structural Hazards
Conflict over the use of resources
Time (clock cycles)
I$
Load
Instr 1
Instr 2
Instr 3
Instr 4
I$ Reg D$ Reg
ALU
I$ Reg D$ Reg
ALU
I$ Reg D$ Reg
ALU
Reg D$ Reg
ALU
I$ Reg D$ Reg
ALU
9. I. ILP: Structural Hazards
Time (clock cycles)
I$
Load
Instr 1
Instr 2
Instr 3
Instr 4
I$ Reg D$ Reg
ALU
I$ Reg D$ Reg
ALU
I$ Reg D$ Reg
ALU
Reg D$ Reg
ALU
I$ Reg D$ Reg
ALU
Solutions R/W:
*On same clock cycle
On different R/W ports
10. I. ILP: Data Hazards
Time (clock cycles)
I$
Instr 1
Instr 2
Instr 3
Instr 4
Instr 5
I$ Reg D$ Reg
ALU
I$ Reg D$ Reg
ALU
I$ Reg D$ Reg
ALU
Reg D$ Reg
ALU
I$ Reg D$ Reg
ALU
11. I. ILP: Data Hazards
add $t0, $t1, $t2
sub $t4, $t0 ,$t3
and $t5, $t0 ,$t6
or $t7, $t0 ,$t8
xor $t9, $t0 ,$t10
20. I. ILP: Control Hazards
Solutions:
Add HW to be able to compute branch on stage 2 (DECODE)
Predict Branch: To simplify hardware, predict branch as NOT TAKEN most of the times. End
of the loop will always be wrong, but then is just once
Insert instruction after branch, always gets executed. Compiler. MIPS
22. I. ILP: Optimizations
Instruction window: Trace of incoming instructions to analyze for execution.
Register Renaming: On false data dependences, hardware can rename the register.
Compilers should optimize this false dependences: R2R memory model.
Branch Prediction:
Static: Always not taken, always taken. Forward/Backward taken. Branch delay slot.
Dynamic: One-level (1bit, 2bit...), Two-level and Multiple Component
Jump Prediction: Static profiling. Dynamic: Last taken, 2bit tables, return stack.
Alias Analysis: Indirect memory references. Instruction Inspection.
23. I. ILP: Branch Prediction
Saturated counter: Increment on branch
taken, decrement on not taken. No
Over or Under flow.
Branch correlation: Inter/Intra
Two-level: Remembers the history of the
last n occurrences of the branch and
uses one saturating counter for each of
the possible 2n history patterns.
Many more...
24. CONTENT
I. ILP Background
II. Hardware Model
III. Study of Limitations
IV. Simultaneous Multithreading
V. ILP today
26. II. HW MODEL: Profiling Framework
A set of assumptions and a methodology to, experimentally, extract a parallelism
profile out of a set of benchmarks.
Program is executed completely, resulting in a trace of instructions.
Trace includes data addresses referenced, and the results of branches and jumps.
(D. Wall's 1993 study) Divides the trace in cycles of 64 instructions in flight.
The only limits on ILP in such a processor are those imposed by the actual data
flows through either registers or memory.
27. II. HW MODEL: Assumptions
• No limits on replicated functional units or ports to registers or memory.
• Register Renaming: Perfect, Infinite, Finite, None
• Branch Prediction: Perfect, Infinite, Finite, None
• Jump Prediction: Perfect, Infinite, Finite, None
• Memory Address Alias Analysis:
• Perfect Caches
• Unit cycle
• 2k Window size
28. II. HW MODEL: Register Renaming
• Perfect: Infinite number of registers to avoid false register dependencies.
• Finite: Normally 256 integer registers and 256 floating point registers used in LRU
(Least Recently Used) fashion.
• No renaming: Number of registers used in the code.
29. II. HW MODEL: Branch Prediction
• Perfect: All branches are correctly predicted.
• 2bit predictor with infinite tables: Dynamic. A 2bit counter per branch option (2).
Indexed by low-order bits of branch's address. Incremented on branch taken. Does not
overflow. Branch is taken if table entry is 2 or 3. Up to 512 2bit entries.
• 2bit predictor with infinite tables: Infinite number of counters.
• Tournament-based branch predictor: 2 2bit counters competing. A 2bit selector that is
decremented/incremented according to the correct prediction of the table entries.
• Profile based: Static predictions.
• No prediction: Every branch is predicted wrong.
Not in order of performance
30. II. HW MODEL: Jump Prediction
• Direct Jumps are known.
• Indirect jumps
– Perfect: Always performed correctly.
– Finite prediction: A table with destination addresses. The address of a jump
provides the index of the table. Whenever a jump is executed, we put its address in
the table. Next jump should be to address in the table.
– Infinite prediction: Infinite table entries.
• No prediction: Every jump is predicted wrong.
31. II. HW MODEL: Alias Analysis
• If two memory references do not refer to the same address, then they may
be safely interchanged.
• Indirect memory references are previous to the instruction execution.
• No need to predict the actual values, only whether those values conflict.
• Perfect: All global and stack reference predictions are perfect, heap
• Inspection: Examine base and offset
• None: All indirect memory references conflict.
32. II. HW MODEL: Window Size
• The set of instructions which is examined for simultaneous execution.
• The cycle width limits the number of instructions which can be scheduled.
• A window size of 2k will look at 2048 instructions.
• Cycle width: Assume we have found 111 instructions which can be parallelized. A
cycle width of 64 would limit actual parallelism to 64 in flight instructions.
33. II. HW MODEL
ctr: counter
gsh: gshared (global history)
34. CONTENT
I. ILP Background
II. Hardware Model
III. Study of Limitations
IV. Simultaneous Multithreading
V. ILP today
35. III. STUDY OF LIMITATIONS
• Effects of...
– Register Renaming
– Branch/Jump Prediction
– Alias Analysis
– Realizable processor
• Window Size (Discrete/Continuous)
• Results
42. III. LIMITATIONS: Realizable Processor
• Up to 64 instruction issues per clock with no issue restrictions, or roughly 10 times the
total issue width of the widest processor in 2011
• A tournament predictor with 1K entries and a 16-entry return predictor. This predictor
is comparable to the best predictors in 2011; the predictor is not a primary bottleneck
• Perfect disambiguation of memory references done dynamically—this is ambitious but
perhaps attainable for small window sizes (and hence small issue rates and load-store
buffers) or through address aliasing prediction
• Register renaming with 64 additional integer and 64 additional FP registers, which is
slightly less than the most aggressive processor in 2011
• No issue restrictions, no cache misses, unit latencies
• Variable Window Size (Power5 200, Intel Core i7 ~128)
44. III. LIMITATIONS: Conclusions
• Plateau behavior
• Window size effect on integer programs (3 top) is
not as severe. Due to loop-level parallelism.
• Designers are faced with the challenge:
– Simpler processors with larger caches and
higher clock rates
Vs
– ILP with slower clock and smaller caches
• Persistent limitations:
– WAW and WAR hazards through memory
– Unnecessary dependences
– Data flow limit
45. CONTENT
I. ILP Background
II. Hardware Model
III. Study of Limitations
IV. Simultaneous Multithreading
V. ILP today
46. IV. SIMULTANEOUS MULTITHREADING
• TLP Background
– TLP approaches
– Design Challenges
• Limits of Multiple-Issue Processors
– Power
– Complexity
47. IV. SMT: TLP Background
• Largely independent
– Separate copy of regFile, PC and page table
• Thread could represent
– A process that is part of a parallel program consisting of multiple processes
– An independent program on its own
• Thread level parallelism occurs naturally
• It can be used to employ the functional units idle when ILP is insufficient
49. IV. SMT: Changes
• Increasing the associativity of the L1 instruction cache and the instruction
address translation buffers
• Adding per-thread load and store queues
• Increasing the size of the L2 and L3 caches
• Adding separate instruction prefetch and buffering
• Increasing the number of virtual registers from 152 to 240
• Increasing the size of several issue queues
51. IV. SMT: Results
• SMT reduces energy by 7%
• “Because of the costs and diminishing returns in performance, however, rather than
implement wider superscalars and more aggressive versions of SMT, many designers are
opting to implement multiple CPU cores on a single die with slightly less aggressive support
for multiple issue and multithreading; we return to this topic in the next chapter.” - Hennessy
et al.
52. CONTENT
I. ILP Background
II. Hardware Model
III. Study of Limitations
IV. Simultaneous Multithreading
V. ILP today
53. V. ILP TODAY: x86
• Instruction fetch—The processor
uses a multilevel branch target buffer
to achieve a balance between speed
and prediction accuracy. There is also a
return address stack to speed up
function return. Mispredictions cause a
penalty of about 17 cycles. Using the
predicted address, the instruction fetch
unit fetches 16 bytes from the
instruction cache.
• Micro-code and Macro-code
• Total pipeline depth is 14 stages
• 128 reorder (renaming) buffer size
54. V. ILP TODAY: x86
• Hyper-Threading:
– SMT
– The processor may stall due to a
cache miss, branch misprediction,
or data dependency.
– Branch misprediction costs 17
cycles
56. V. ILP TODAY: ARM
- The average CPI for the ARM7 family is about 1.9 cycles per instruction.
- The average CPI for the ARM9 family is about 1.5 cycles per instruction.
- The average CPI for the ARM11 family is about 1.39 cycles per instruction.
57. SOURCES
Computer Architecture: A Quantitative Approach. Hennessy, J.L., Patterson, D.A., Asanović, K..
5th Ed. 2012. Morgan Kaufmann/Elsevier.
Limits of instruction-level parallelism. D. W. Wall. IV international conference on Architectural
Support for Programming Languages and Operating Systems (ASPLOS), pages 176–188, 1991.
Computer Science 61C - Lecture 31: Instruction Level Parallelism. Mike Franklin, Dan Garcia.
UC Berkeley. Fall. 2011
ILP and TLP in Shared Memory Applications: A Limit Study. E. Fatehi, P. V. Gratz, Proceedings of
the 23rd international conference on Parallel architectures and compilation, pages 113-126, 2014.
MIPS Multicycle Model: Pipelining. Michael Langer. Introduction to Computer Systems. McGill
University. 2012.
IBM Power5 Chip: A Dual-Core Multithreaded Processor. R. Kalla, B. Sinharoy, J. M. Tendler. IBM.
IEEE CS. 2004.