9. 9
CVE-2016-3086: Apache Hadoop YARN NodeMangaer
vulnerability
今回紹介する脆弱性の内容
The YARN NodeManager in Apache Hadoop 2.6.x before 2.6.5
and 2.7.x before 2.7.3 can leak the password for credential store
provider used by the NodeManager to YARN Applications.
If you use the CredentialProvider feature to encrypt passwords
used in NodeManager configs, it may be possible for any
Container launched by that NodeManager to gain access to the
encryption password. The other passwords themselves are not
directly exposed.
http://mail-archives.apache.org/mod_mbox/hadoop-general/201701.mbox/%3C0ed32746-5a53-9051-5877-2b1abd88beb6%40apache.org%3E
21. 21
ASF Project Security for Committers
https://www.apache.org/security/committers.html
脆弱性の修正と悟られないようにしている
9. The project team agrees the fix on their private list.
12. The project team commits the fix. No reference
should be made to the commit being related to a
security vulnerability.
22. 22
ASF Project Security for Committers
https://www.apache.org/security/committers.html
脆弱性の公表は、修正されたリリースが出た後
15.The project team announces the vulnerability. The
vulnerability announcement should be sent after, or at
the same time as, the release announcement to the
following destinations:
a. the same destinations as the release announcement
b. the vulnerability reporter
c. the project's security list
*snip*
24. 24
修正後、3系でMapReduceジョブが失敗するようになった
2016-12-02 04:54:52,413 INFO mapreduce.Job: Job
job_1480654443168_0001 failed with state FAILED due to: Application
application_1480654443168_0001 failed 2 times due to AM Container
for appattempt_1480654443168_0001_000002 exited with exitCode: 1
Failing this attempt.Diagnostics: Exception from container-launch.
Container id: container_1480654443168_0001_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:974)
at org.apache.hadoop.util.Shell.run(Shell.java:878)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.jav
a:1172)
(snip)
26. 26
MAPREDUCE-6704での議論を抜粋
*snip* since its a mapreduce property its not correct to
add to the whitelist to yarn.
Who cares? It's all Apache Hadoop. Users have an
expectation that this stuff will work out of the box and
be consistent. *snip*
There was pushback to remove it because of the
desire to keep Yarn and MR separate.
補足: HADOOP_MAPRED_HOMEの追加がデフォルトにならない理由
31. 31
Erasure Codingとは (簡単に)
データをk個のdata unitに分割し、r個のparity unitを生成
k+r個のunitのうち、任意のk個からデータを復旧可能
parity生成にはReed-Solomon(RS)がよく利用される
RS(k=10,r=4)の場合、実データの1.4倍のディスク消費
HDFSの通常の3-replicationなら3倍
unit 1 unit 2 unit 3 unit 4 unit 5 unit 6 unit 7
unit 8 unit 9 unit 10 unit 11 unit 12 unit 13 unit 14
33. 33
Erasure Codingでの障害復旧における課題
例: unit 6故障時の復旧パターン
任意の10 unitのデータを取得
unit 1 unit 2 unit 3 unit 4 unit 5 unit 7
unit 8 unit 9 unit 10 unit 11 unit 12 unit 13 unit 14
unit 1 unit 2 unit 3
unit 4 unit 5 unit 7
unit 8 unit 10 unit 12 unit 14
34. 34
例: unit 6故障時の復旧パターン
任意の10 unitのデータを取得
unit 6を生成
Erasure Codingでの障害復旧における課題
unit 1 unit 2 unit 3 unit 4 unit 5 unit 7
unit 8 unit 9 unit 10 unit 11 unit 12 unit 13 unit 14
unit 1 unit 2 unit 3
unit 4 unit 5 unit 7
unit 8 unit 10 unit 12 unit 14
unit 6
35. 35
例: unit 6故障時の復旧パターン
任意の10 unitのデータを取得
unit 6を生成
Erasure Codingでの障害復旧における課題
unit 1 unit 2 unit 3 unit 4 unit 5 unit 7
unit 8 unit 9 unit 10 unit 11 unit 12 unit 13 unit 14
unit 1 unit 2 unit 3
unit 4 unit 5 unit 7
unit 8 unit 10 unit 12 unit 14
unit 6
失ったデータの10倍の
データを読み込む必要がある!
(ディスク、NWに負荷)