1. /VM
Tsuyoshi Ozawa
Twitter oza_x86
2009 11 9 1

2. 2009 11 9 2

3. 2009 11 9 3

4. 2009 11 9 4

5. 2009 11 9 5

6. KVM
2009 11 9 6

7. for x86
2009 11 9 7

8. Linux on Real Machine Linux on Virtual Machine
(=OS)
2009 11 9 8

9. 2009 11 9 9

10. x86
(Shadow Page Table)
2009 11 9 10

11. CPU
0x1000 0x11000
Virtual Physical
Memory Memory
0x2000 0x12000
... ...
2009 11 9 11

12. •
•
2009 11 9 12

13. •
•
2009 11 9 13

14. • OS
• 0x1000 0x11000
• 0x2000 0x12000
... ...
2009 11 9 14

15. •
0x1000 0x11000
• OS 0x2000 0x12000
... ...
2009 11 9 15

16. •
?
•
swap in
Physical swap out
Memory
2009 11 9 16

17. 1.
CR3
2. MMU
3. TLB CPU
CR3
x86
CPU MMU
TLB
2009 11 9 17

18. CR3
•
• Control Register 3
2009 11 9 18

19. • MMU
• Memory Management Unit
2009 11 9 19

20. • Translation Look-aside Buffer TLB
•
• CPU SRAM
2009 11 9 20

21. Virtual Address 0xc0003120 3
20bit 12bit 4.
CR3
2.
MMU Physical
Memory
TLB 3.
1.TLB
2009 11 9 21

22. 2009 11 9 22

23. VM
2009 11 9 23

24. 2009 11 9 24

25. VM
2009 11 9 25

26. VM
• OS
2009 11 9 26

27. VM
Virtual
Machine
Monitor
VMM
Hardware
2009 11 9 27

28. VMM
• 3 VMM
• Robert P. Goldberg. ,Architectural
Principles for Virtual Computer Systems.
2009 11 9 28

29. Type I VMM
•
• Xen,VMware ESX
VMM
Hardware
2009 11 9 29

30. Type II VMM
• OS
• Qemu Proc VMM
LilyVM
Bochs OS
Hardware
2009 11 9 30

31. Hybrid VMM
• Type I + Type II
•
• VMware Proc VMM
Workstation
Linux KVM
OS VMM
Hardware
2009 11 9 31

32. Hybrid VMM
•
VMM
OS VMM
Hardware
2009 11 9 32

33. VM
2009 11 9 33

34. 2
• Shadow Page Table ( SPT )
• Nested Page Table (Extended Page Table)
2009 11 9 34

35. Shadow Page Table
• OS
Shadow
OS
• OS Shadow
2009 11 9 35

36. SPT
Hybrid VMM
OS VMM
Hardware
2009 11 9 36

37. SPT
OS
VM
Guest side vTLB vCR3
Host side VMM
Hardware
TLB CR3
2009 11 9 37

38. SPT
create page table
OS Guest
page table
VM
Guest side vTLB vCR3
Host side VMM
Hardware
TLB CR3
2009 11 9 38

39. SPT
OS Guest
page table
VM
Guest side vTLB vCR3
Host side VMM
...
Hardware
TLB CR3
2009 11 9 39

40. SPT
OS Guest
page table
VM
Guest side vTLB vCR3
VMM
Host side VMM
Hardware
TLB CR3
2009 11 9 40

41. SPT
OS Guest
page table
VM
Guest side vTLB vCR3
VMM
Host side VMM PT
Hardware
TLB CR3
2009 11 9 41

42. SPT
OS Guest
page table
VM
Guest side vTLB vCR3
create SPT
Host side VMM Shadow
page table
Hardware
TLB CR3
2009 11 9 42

43. SPT
OS Guest
page table
Guest
Physical
VM Memory
vTLB vCR3
VMM Shadow
page table
OS vCR3
Host
Physical
Hardware Memory
TLB CR3
2009 11 9 43

44. SPT
OS Guest
page table
Guest
Physical
VM Memory
vTLB vCR3
Hook
VMM Shadow
VMM
page table Host
Physical
Hardware Memory
TLB CR3
2009 11 9 44

45. SPT
OS Guest
page table
Guest
SPT
Physical
VM Memory
vTLB vCR3 CR3
VMM Shadow
page table Host
Physical
Hardware Memory
TLB CR3
2009 11 9 45

46. 2009 11 9 46

47. SPT
OS Guest
page table
Guest
Physical
VM Memory
vTLB vCR3
VMM Shadow
page table Host
Physical
Hardware Memory
TLB CR3
2009 11 9 47

48. SPT
OS Guest
page table
Guest
Physical
VM Memory
vTLB vCR3
VMM Shadow
page table Host
Physical
Hardware Memory
TLB CR3
2009 11 9 48

49. SPT
OS Guest
page table
Guest
Physical
VM Memory
vTLB vCR3
VMM Shadow
page table Host
Physical
Hardware Memory
TLB CR3
2009 11 9 49

50. SPT
• OS
• VMM SPT
2009 11 9 50

51. SPT
• Pre-Validation
• Virtual TLB
• Virtual TLB + Memory Trace
2009 11 9 51

52. Pre-validation
• CR3
• PT
• SPT
2009 11 9 52

53. Pre-validation
create page table
OS Guest
page table
VM
Guest side vTLB vCR3
Host side VMM
Hardware
TLB CR3
2009 11 9 53

54. Pre-validation
OS Guest
page table
VM
Guest side vTLB vCR3
OS
Host side VMM
PT
Hardware
TLB CR3
2009 11 9 54

55. Pre-validation
OS Guest
page table
VM
Guest side vTLB vCR3
CR3
Host side VMM
Hardware
TLB CR3
2009 11 9 55

56. Pre-validation
OS Guest
page table
VM
Guest side vTLB vCR3
VMM
Host side VMM
Hardware
TLB CR3
2009 11 9 56

57. Pre-validation
OS Guest
page table
VM
Guest side vTLB vCR3
VMM
Host side VMM PT
Hardware
TLB CR3
2009 11 9 57

58. Pre-validation
OS Guest
page table
VM
Guest side vTLB vCR3
create SPT
Host side VMM Shadow
page table
Hardware
TLB CR3
2009 11 9 58

59. Pre-validation
OS Guest
page table
Guest
SPT
Physical
VM Memory
vTLB vCR3 CR3
VMM Shadow
page table Host
Physical
Hardware Memory
TLB CR3
2009 11 9 59

60. SPT
OS Guest
page table
Guest
Physical
VM Memory
vTLB vCR3
VMM Shadow
page table Host
SPT Physical
Hardware Memory
TLB CR3
2009 11 9 60

61. Pre-validation
•
( CR3 ) SPT
•
2009 11 9 61

62. Virtual TLB
• CR3
SPT 1
• TLB
SPT
• TLB
VMM
2009 11 9 62

63. Virtual TLB
create page table
OS Guest
page table
VM
Guest side vTLB vCR3
Host side VMM
Hardware
TLB CR3
2009 11 9 63

64. Virtual TLB
OS Guest
page table
VM
Guest side vTLB vCR3
OS
Host side VMM
PT
Hardware
TLB CR3
2009 11 9 64

65. Virtual TLB
OS Guest
page table
VM
Guest side vTLB vCR3
CR3
Host side VMM
Hardware
TLB CR3
2009 11 9 65

66. Virtual TLB
OS Guest
page table
VM
Guest side vTLB vCR3
VMM
Host side VMM
Hardware
TLB CR3
2009 11 9 66

67. Virtual TLB
OS Guest
page table
VM
Guest side vTLB vCR3
VMM
Host side VMM PT
Hardware
TLB CR3
2009 11 9 67

68. Virtual TLB
OS Guest
page table
VM
Guest side vTLB vCR3
create SPT
Host side VMM Shadow
page table
Hardware 1st stage
TLB CR3
2009 11 9 68

69. Virtual TLB
VMM
OS Guest
page table
VM
Guest side 1 vTLB vCR3
create SPT
Host side VMM Shadow
page table
Hardware 1st stage
TLB CR3
2009 11 9 69

70. Virtual TLB
OS Guest
page table
Guest
SPT
Physical
VM Memory
vTLB vCR3 CR3
VMM Shadow
page table Host
1st stage Physical
Hardware Memory
TLB CR3
2009 11 9 70

71. Virtual TLB
OS Guest
page table
Guest
Physical
VM Memory
vTLB vCR3
VMM Shadow
page table Host
1st stage Physical
Hardware Memory
TLB CR3
2009 11 9 71

72. Virtual TLB
OS Guest
page table
Guest
Physical
VM Memory
vTLB vCR3
VMM Shadow
page table Host
1st stage Physical
Hardware Memory
TLB CR3
2009 11 9 72

73. Virtual TLB
OS Guest
page table
Guest
Physical
VM Memory
vTLB vCR3
VMM Shadow
page table Host
1st stage Physical
Hardware Memory
TLB CR3
2009 11 9 73

74. Virtual TLB
OS Guest
page table
Guest
Physical
VM Memory
vTLB vCR3
VMM
VMM Shadow
page table PT Host
1st stage Physical
Hardware Memory
TLB CR3
2009 11 9 74

75. Virtual TLB
OS Guest
page table 2
Guest
Physical
VM Memory
vTLB vCR3
Shadow
VMM page table
Host
2nd stage Physical
Hardware Memory
TLB CR3
2009 11 9 75

76. Virtual TLB
OS Guest
page table
Guest
Physical
VM Memory
vTLB vCR3
Shadow
VMM page table
Host
2nd stage Physical
Hardware Memory
TLB CR3
( ) CPU
2009 11 9 76

77. Virtual TLB
OS
OS Guest
page table
Guest
Physical
VM Memory
vTLB vCR3
Shadow
VMM page table
Host
2nd stage Physical
Hardware Memory
TLB CR3
( ) CPU
2009 11 9 77

78. Virtual TLB
OS Guest
page table
Guest
Physical
VM Memory
vTLB vCR3
Shadow
VMM page table
Host
2nd stage Physical
Hardware Memory
TLB CR3
( ) CPU
2009 11 9 78

79. Virtual TLB
•
•
•
• SPT
2009 11 9 79

80. Virtual TLB
• TLB
• CR3
• invlpg
• OS
•
PTE
2009 11 9 80

81. SPT
• TLB
SPT
• OS PT SPT
2009 11 9 81

82. Memory Trace
• OS PT
• OS
SPT
(Trace)
2009 11 9 82

83. • x86
•
CPU
• Linux KVM
2009 11 9 83

84. kvm-88/kernel/x86/paging_tmpl.h
2009 11 9 84

85. Nested Page Table
(Extended Page Table)
•
0x1000 0x11000 0x21000
0x2000 0x12000 0x42000
... ...
2009 11 9 85

86. NPT/EPT
• AMD CPU
CPU
• Intel CPU
CPU(i7 )
• VMM
• SPT
2009 11 9 86

87. • VM
• SPT
• Pre-validation
• Virtual TLB
• NPT/EPT
2009 11 9 87