Oracle Cloud Data Interface

1. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted
Oracle Cloud
Data Interface
김태완
@taewanme
https://taewan.kim

2. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
데이터 라이프 사이클
2
Ingestion
streaming
batch
replication
DataStore
Object store data
lake
Hadoop/HDFS
NOSQL/Cassandra
Transform
Data pipelines
ML pipelines
Data quality
Data Augmentation
Analyze
Python/R
Spark
Hive
Interactive BI
Learn
Machine learning
AI
Data science

3. Copyright © 2018,Oracle and/orits affiliates. All rights reserved.
Oracle Cloud
Data Lake
Oracle Object Storage
BigData Cloud
Event Hub Cloud
DIPC
Database Data Hub IaaS VM
NoSQL
OAC
ADW
*ADW: Autonomous Data warehouse
*DIPC: Data Integration Platform Cloud
* OAC: Oracle Analytics Cloud
Object Storage

4. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 4
OCI Object Storage
Case 01: Desktop & OCI Object Storage

5. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Oracle OCI Object Storage 특징
5

6. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
OCI Object Storage
6
참조: http://bit.ly/2Z0lyA3

7. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
OCI Object Storage: bucket
7
참조: http://bit.ly/2Z0lyA3

8. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
OCI Object Storage: Connection Info
8
참조: http://bit.ly/2Z0lyA3

9. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
OCI Object Storage: Connection Info
9
참조: http://bit.ly/2Z0lyA3

10. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
OCI Object Storage: CloudBerry S3 Explorer
10
테넌트이름.compat.objectstorage.리전이름.oraclecloud.com
참조: http://bit.ly/2Z0lyA3

11. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
OCI Object Storage: CloudBerry S3 Explorer
11
참조: http://bit.ly/2Z0lyA3

12. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 12
OCI Object Storage
현재 위치

13. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 13
OCI Object Storage
Case 02: OCI CLI & OCI Object Storage
oci cli

14. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
oci cli 설치
14
참조: http://bit.ly/2Z0lyA3
bash -c "$(curl -L https://raw.githubusercontent.com/oracle/oci-cli/master/scripts/install/install.sh)"
Linux / Mac OS
powershell -NoProfile -ExecutionPolicy Bypass -Command "iex ((New-Object
System.Net.WebClient).DownloadString('https://raw.githubusercontent.com/oracle/oci-
cli/master/scripts/install/install.ps1'))"
Window

15. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
oci cli 설치 확인
15
참조: http://bit.ly/2Z0lyA3

16. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
oci cli command
16

17. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
oci cli 설정: info
17
참조: http://bit.ly/2Z0lyA3
user OCID

18. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
oci cli 설정: info ➔ Tenancy OCID, Region ID
18

19. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
oci cli 설정: oci setup config
19

20. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
oci cli 설정: key files
20

21. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
oci cli 설정: public key 등록
21

22. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
oci cli 설정: public key 등록
22

23. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
oci cli 설정: public key 등록

24. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
oci cli 설정: Connection Test

25. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
oci cli: Bucket 생성
25
oci cli os bucket
--name
--compartment-id

26. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
oci cli: Bucket 생성
26

27. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
oci cli: 파일 업로드
27

28. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
oci cli: 파일 업로드
28

29. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 29
OCI Object Storage
현재 위치
oci cli

30. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 30
OCI Object Storage
Case 03: 대용량 파일 업로드
oci cli

31. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
oci cli: 데모 파일 생성
31

32. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
oci cli: 10GB 파일 업로드
32

33. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
oci cli: 10GB 파일 업로드
33

34. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 34
OCI Object Storage
현재 위치
oci cli

35. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 35
OCI Object Storage
Case 04: Java를 이용한 파일 업로드 다운로드
OCI Java SDK

36. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Oracle OCI Java SDK
36
https://search.maven.org/search?q=g:com.oracle.oci.sdkhttps://github.com/oracle/oci-java-sdk

37. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Oracle OCI Java SDK: Maven
37

38. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Oracle OCI Java SDK: Code
38

39. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Oracle OCI Java SDK: Code
39

40. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Oracle OCI Java SDK: Connection
40

41. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Oracle OCI Java SDK: Bucket 생성
41

42. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Oracle OCI Java SDK: 파일 추가
42

43. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 43
OCI Object Storage
현재 위치
oci cli
VM
OCI
Java
SDK

44. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 44
OCI Object Storage
Case 05: OCI SDK & Object storage

45. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Oracle OCI SDK & Rest API
45
https://github.com/oracle?utf8=%E2%9C%93&q=oci+sdk&type
=&language=

46. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Oracle OCI SDK & Rest API
46
https://github.com/oracle?utf8=%E2%9C%93&q=oci+sdk&type
=&language=

47. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Oracle OCI SDK Python: 설치
47

48. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Oracle OCI SDK Python: Code
48

49. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Oracle OCI SDK Python: Code
49

50. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Oracle OCI SDK Python: Code
50

51. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Oracle OCI SDK Python: Object 생성 결과
51

52. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Oracle OCI SDK Go-lang: Install
52
> go get -u github.com/oracle/oci-go-sdk
import (
"github.com/oracle/oci-go-sdk/common"
"github.com/oracle/oci-go-sdk/identity”
)
https://github.com/oracle/oci-go-sdk

53. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 53
OCI Object Storage
현재 위치
oci cli
VM
OCI
Java
SDK
VM VM
OCI
GO
SDK
VM
OCI
Ruby
SDK
OCI
Python
SDK

54. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 54
OCI Object Storage
Case 06: Hadoop & Object Storage
HDFS Connector
for Object Storage

55. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 55
Hadoop & Object Storage
https://github.com/oracle/oci-hdfs-connector/releases

56. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 56
HDFS Connector for OCI: 설치
https://github.com/oracle/oci-hdfs-connector/releases
속성 타입 설명
fs.oci.client.auth.tenantId String tenancy의 OCID
fs.oci.client.auth.userId String 사용자 OCID
fs.oci.client.auth.fingerprint String 사용자 Fingerprint
fs.oci.client.auth.pemfilepath String PEM 파일 위치
fs.oci.client.auth.passphrase String 암호화 문자열
core-site.xml

57. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 57
HDFS Connector for OCI: 설치
scala> sc.wholeTextFiles("oci://PipedUploadTest@internalbriangustafson/")

58. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 58
OCI Object Storage
현재 위치
oci cli
VM
OCI
Java
SDK
VM VM
OCI
GO
SDK
VM
OCI
Ruby
SDK
OCI
Python
SDK
HDFS Connector
for Object Storage

59. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 59
OCI Object Storage
Case 07: Kafka & Object Storage

60. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 60
OCI Object Storage
Case 07: Kafka & Object Storage
Apps
NoSQL
Search
RDBMS
Monitoring
RT Analysis
DWH
Hadoop

61. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
+ =
Managed Kafka Service
Oracle Event Hub CS

62. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |

63. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |

64. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |

65. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |

66. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |

67. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |

68. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
https://www.confluent.io/product/connectors/
- Scalable Framework
- Open Source
- Only Configuration

69. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
OEHCS의 Kafka Connector 지원

70. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Streaming Data + Database
Oracle
GoldenGate
추출 트랜잭션
추출 실시간 이관
JMS
Capture Trail Pump Route Deliver

71. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 71
OCI Object Storage
현재 위치
oci cli
VM
OCI
Java
SDK
VM VM
OCI
GO
SDK
VM
OCI
Ruby
SDK
OCI
Python
SDK
HDFS Connector
for Object Storage

72. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Case 10: ADW와 Object Storage
72

73. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Object Storage에 데이터 저장
73
parquet
df.write.parquet("oci://devmeetup@apackrsct01/sales_extended.parquet")

74. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Object Storage에 데이터 뷰
74
parquet
df.write.parquet("oci://devmeetup@apackrsct01/sales_extended.parquet")
$ parquet-tools schema sales_extended.parquet
message hive_schema {
optional int32 prod_id;
optional int32 cust_id;
optional binary time_id (UTF8);
optional int32 channel_id;
optional int32 promo_id;
optional int32 quantity_sold;
optional fixed_len_byte_array(5)
amount_sold (DECIMAL(10,2));
optional binary gender (UTF8);
optional binary city (UTF8);
optional binary state_province (UTF8);
optional binary income_level (UTF8);
}

75. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Object Storage에 데이터 뷰
75
parquet
df.write.parquet("oci://devmeetup@apackrsct01/sales_extended.parquet")
$ parquet-tools schema sales_extended.parquet
message hive_schema {
optional int32 prod_id;
optional int32 cust_id;
optional binary time_id (UTF8);
optional int32 channel_id;
optional int32 promo_id;
optional int32 quantity_sold;
optional fixed_len_byte_array(5)
amount_sold (DECIMAL(10,2));
optional binary gender (UTF8);
optional binary city (UTF8);
optional binary state_province (UTF8);
optional binary income_level (UTF8);
}
https://github.com/Guavus/parquet-mr-
1/tree/master/parquet-tools

76. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Object Storage에 데이터 뷰
76
parquet
df.write.parquet("oci://devmeetup@apackrsct01/sales_extended.parquet")
https://github.com/Guavus/parquet-mr-
1/tree/master/parquet-tools

77. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
ADW의 External Table
77

78. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
ADW와 Parquet External Table
78
External Table

79. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 79
OCI Object Storage
현재 위치
oci cli
VM
OCI
Java
SDK
VM VM
OCI
GO
SDK
VM
OCI
Ruby
SDK
OCI
Python
SDK
HDFS Connector
for Object Storage

80. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Case 11: 다양한 인터페이스? ➔ 데이터 가상화
80
Hadoop Security
ACL’s | Sentry | HDFS Encryption | Encryption in Motion
Big Data SQL Cloud Service

81. Copyright © 2018,Oracle and/orits affiliates. All rights reserved.
Oracle Cloud
Data Lake
Oracle Object Storage
BigData Cloud
Event Hub Cloud
DIPC
Database Data Hub IaaS VM
NoSQL
OAC
ADW
*ADW: Autonomous Data warehouse
*DIPC: Data Integration Platform Cloud
* OAC: Oracle Analytics Cloud
Object Storage

82. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
Oracle Data Platform
82

83. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 83