APAN39@Fukuoka Cloud Working Group

1. Building High-Performance Inter-Cloud
Infrastructure in Japan
Masaharu Munetomo
Professor & Vice Director,
Information Initiative Center,
Hokkaido University, Sapporo, JAPAN.
munetomo@iic.hokudai.ac.jp
46Campus Maps
... we are cosmopolitan, and accessible...
Picturesque Hakodate is home to Hokkaido University’s Faculty of Fisheries Science and is located on the south-west of the island.
With a population of approximately 280,000 people, the coastal city is at the base of Mount Hakodate, which boasts amazing natural beauty. The
view from the summit is renowned for having one of the most beautiful views in Japan, particulary at night. Since it opened in 1935, the Hakodate
Sapporo Campus
Hokkaido
Hakodate Campus
1

2. Masaharu Munetomo
• Professor & Vice director, Information Initiative Center,
Hokkaido university, Sapporo, JAPAN.
• Chief examiner, Cloud computing research group of national supercomputing
centers in Japan.
• Chief examiner, SIG Cloud, Academic eXchange for Information Environment
and Strategy (AXIES) in Japan.
• Chief examiner, SIG Mathematical Problem-Solving, Information Processing
Society of Japan (IPSJ)
• General advisor, Cloud Utilization Promotion Agency (CUPA) & Managed
Service Providers associations in Japan (MSPJ)
• Founding member and of steering committee, Open Compute Project in
Japan (OCPJ)
2

3. Information Initiative Center, Hokkaido University
• Founded in 1962 as a national supercomputing center.
• A member of High Performance Computing Infrastructure (HPCI) and Joint
Usage/Research Center for Interdisciplinary Large-scale Information
Infrastructure (JHPCN) in Japan.
• University R&D center for Supercomputing, Cloud computing, Networking, IT
systems for education
• Supercomputer (172TFlops) & Academic Cloud System (43TFlops)
3

4. HPCI (High Performance Computing Infrastructure)
• Collaboration of national supercomputing centers in Japan.
• RIKEN AICS (K computer) & Supercomputing Centers (University, Research
Institutes) connected via academic high-speed network (SINET4)
• Federations of users & systems management (GSI-SSH, Gfarm supported)
http://hpci-office.jp/
4

5. Hokkaido University Academic Cloud System
• Largest Academic Cloud System in Japan started services from
Nov. 2011: 43TFlops (5,000 cores), and more than 2,000 VMs
can be deployed.
• Employing CloudStack to provide cloud management portal.
• High-performance cloud system: each physical node has 40-
cores, 128GB memory. Network: 10GbE x 2, Shared Storage:
260TB (SAN) + 500TB (NAS) + 2PB (WebDAV, S3, Gfarm)
Hitach BladeSymphony BS2000
Xeon E7 8870 2.4GHz (10-core) x 4
128GB memory / 10GbE x 2
Hitachi NAS Storage
AMS2300: 260TB
AMS2500: 500TB 5

6. Use case: “Big Data” processing systems
• We provide “Big Data” service VM package consisting of Hadoop, Hive,
Mahout, and R.
• Automated deployment of VM clusters, customizing scheduling policies in
CloudStack to balance I/O overheads for cluster packages (Hadoop / MPI /
Torque).
Storage #3
Virtual(
Disk
Storage #4
Virtual(
Disk
Storage #2
Virtual(
Disk
Zone!
POD!
Shared Storage #1
Resource Pool #1
HyperVisor #2
HyperVisor #1
Virtual(
DiskVM(
Balancing!overheads!of!disk!I/O!with!
round8robin!assignment!of!Virtual!disks.!
Storage #1
VM(
VM(
VM(
VM(
Virtual(
DiskHadoop Cluster
Shared Storage #2
Resource Pool #2
HyperVisor #4
HyperVisor #3
Virtual(
Disk
VM(
Shared Storage #3
Resouce Pool #3
HyperVisor #6
HyperVisor #5
Virtual((
Disk
VM(
Shared Storage #4
Resouce Pool #4
HyperVisor #8
HyperVisor #7
Virtual(
Disk
VM(
6

7. Use case: simulation environment to replace in-
house computing servers or clusters
• Replacement of in-house clusters of laboratories employing L (10-core) or XL
(40-core) project servers.
• Filling in the gap between PCs and super-computers.
7

8. Use case: development of in-silico screening
system for drug design
• Center for Research and Education on Drug Discovery builds a Structure
Based Drug Design (SBDD) system for in-silico screening with the academic
cloud system
• A virtual private cloud system using XL servers (40-core): modeFRONTIER®
and AutoDock are installed as docking applications.
AutoDock[1]
AutoDock[2]
AutoDock
AutoDock
AutoDock
AutoDockContinuous
execution
of analysis
servers
8

9. Use-case: Fishing ground prediction system
• Researchers in department of fishery build a fishing ground prediction
system on Hokkaido university academic cloud system
• The system provides information on promising sea area for fishing boats to
catch squids, employing satellite images and data assimilation results.
Portal System
Satellite image processing
Data assimilation
Fishing ground prediction
INMARSAT
Satellite
Earth station
Satellite
Communications
Squid Fishing
Boats Fishing ground prediction system portal
9

10. Use-case: Employing PaaS for scalable interactive
evolutionary computation
• Building a scalable interactive evolutionary computation framework to evolve
solutions according to the preferences of millions of users.
CloudStack
VM
Ubuntu
instance
VM
Ubuntu
Redis
VM
Ubuntu
Redis
VM
Ubuntu
Redis
Database
・・・
VM
Ubuntu
instance
VM
Ubuntu
instance
・・・
Applycation resource
iGA iGA iGA
Load Balancer
CloudFoundry
Sever
・・・
Interactive Evolutionary
Computation using PaaS
Users select
solutions according
to their preferences
Present
cadndates of
solutions from
the system
10

11. Japanese academic inter-cloud infrastructure
• Development of the inter-cloud system over Japanese universities to
collaborate private clouds from Kitami (Northernmost) to Ryukyu
(Southernmost) universities through Japanese academic high-speed network
(SINET4).
Hokkaido
University
Kitami Institute
of Technology
University of Ryukyus
(Okinawa)
National Institute of
Informatics (NII)
11

12. Related projects
• Remote collaborations of distributed cloud systems (JHPCN)
• Federations technologies development toward academic inter-cloud
(Collaborative research project, National Institute of Informatics)
• Large-scale Distributed Design Exploration Framework (JHPCN)
• Development of distributed database infrastructure across Japan
• Inter-cloud resource optimization with multi-objective evolutionary algorithms
• Designing the next-generation Hokkaido university high-performance inter-
cloud system
12

13. Remote collaborations of distributed cloud systems
• Prototyping an inter-cloud manager and authentication infrastructure for
federation of academic cloud systems managed by different cloud
middleware (CloudStack, OpenStack, etc.)
• Designing a VPC (Virtual Private Cloud) management framework in the
distributed inter-cloud systems.
Cloud A IaaS
Cloud B IaaS
Cloud C IaaS
User
VPC 1
Internet
VM
VM
VM
VPC 2
220km
13

14. Large-scale Distributed Design Exploration
Framework (LDDEF)
• To establish a framework to support “parameter surveys” by
supercomputing simulations collaborating design engineers
sharing information on promising solutions with distributed DBs
• “Multi-objective design
exploration” explores
Pareto-fronts stored in
distributed DBs
• Optional info. Is
stored in object
storages for
visualization and
analysis
Solutions DB
(distributed)
Automated
replication
for DR and
load balancing
Visualization
Simulation
(Supercomputer)
Optimization &
DB management
(Cloud system)
Distributed
Database
Product
14

15. Grid Unified Framework for Optimization (Grid-UFO) &
MHGRID (Asim, Wahib, Munetomo, 2008-2010)
• A unified framework collaborating optimization algorithms libraries and
simulation programs to evaluate “fitness” values registered by different
developers in GRID computing distributed exec. environment.
GridUFO(Checks(compa3bility(of(sovler:obj(func(pair(
(
Solvers(Database(
(
Obj(Func(Database(
User(Develops(&(Registers(an((
Op3miza3on(Problem(
User(Develops(&(
Registers(a(Solver(
Solver(Developer(
User(Selects(a(Solver(&(an(Objec3ve(Func3on(
GridUFO(Deploys(the(Job(over(Grid(
Solver(
Obj(Func(
MHAPI(
Ninf:IDL(
Distributed(Implementa3on(over(Grid(
MHML(
Obj(Func(Developer(
Ordinary(User(
Submits(Op3miza3on(Job(
MHML(
15

16. LDDEF: System architecture overview
• Fully distributed and scalable architecture consisting of simulators in
supercomputers, optimization engines, analyzers object storages and
distributed database nodes in the inter-cloud environment.
DB
Object)
Storage(s)
DB
DB
Simulator
Optimizer
Simulator
Optimizer
<s,:f>
<s,:?>
<s,:f>
<s,:?>
replication
<p><p>
{:<s,:f>:}
<s’>
{:<s,:f>:}
<s’>
Analyzer:/
Visualizer)
Controller:&)
User:Interface
Distributed:DBs
{:<p>:}
{:<s,:f>:}
(feedback)
replication
16

17. Cassandra distributed database nodes deployed
across Japan
• We have built a testbed of Cassandra distributed database nodes across
Japan from Kitami (Hokkaido) to Okinawa connected via SDN (Vyatta).
• We have tested performance with/without replications and availability and
resiliency in cases of node and network faults.
0"
1000"
2000"
3000"
4000"
5000"
6000"
1" 11" 21" 31" 41" 51" 61" 71" 81"
0"
1000"
2000"
3000"
4000"
5000"
1" 11" 21" 31" 41" 51" 61" 71" 81"
Number"of"requestsNumber"of"requests
write8latency"(ms)
read8latency"(ms)
with"replicaCons without"replicaCons
Hokkaido'University'
Informa3on'Ini3a3ve'Center
Kitami'Ins3tute'
'of'Technology
University'of'the'Ryukyus'
70ms
60ms
10ms
17

18. Cloud Resource Deployment Optimization
(CReDO) in the Inter-Cloud Environment
• Optimizing deployment of virtualized systems requested from
users according to their system specifications using multi-
objective evolutionary algorithms such as NSGA-II/III.
• Semi-automated scheduling policy to “recommend” a variety of
system deployment patterns at Pareto-front to users.
CReDO
Solver /
Optimizer
DB
Request with
Spec. info
Response with
Deploy. info
Public Cloud A Public Cloud B Private Cloud
System info.,
Accounting, etc
18

19. Multi-objective inter-cloud resource optimization
using multi-objective evolutionary algorithms.
• We employ multi-objective evolutionary algorithms such as NSGA-II and
NSGA-III to solve resource optimization problems in the inter-cloud
environment.
• Solving multi-objective optimization considering cost, performance(response
time), and greenness (CO2 emission) simultaneously.
19

20. Toward the next generation of Hokkaido university
academic cloud as high-performance inter-cloud
• We are planning to develop a high-performance inter-cloud system as the
next generation Hokkaido university academic cloud
• Inter-cloud (service layer): multi-cloud controller & broker with cloud
exchange
• Inter-cloud (infrastructure layer): Inter-cloud connector with SDN controller
Private Cloud
with Supercompter
& BigData Storage
Inter-Cloud Portal
(multi-cloud controller)
Public Cloud A
VPN (SDN)
Public Cloud B
Inter-Cloud
Connector
Community Cloud CPublic/Comunity Clouds
Cloud
Exchange
HPC
20

21. High-performance inter-cloud design: an example
SW
Super'
computer
40GbE'(x'142)
FC'or'IB
Tape'
Archive
IaaS/'
HaaS'
IaaS/'
HaaS'
IaaS/'
HaaS'
Cloud'Shared'
Storage
App
WebDAV'
S3/Gfarm
HPC'
Storage
Campus'
LAN
FW'
Router
SW
Manageme
nt'servers'
Baremetal,'VMWare)
Baremetal,'CloudStack'or'OpenStack)
Baremetal)
100G'x'1
Tape'
ArchiveIaaS/'
HaaS'
IaaS/'
HaaS'
Public'clouds'
Community'clouds'
Remote'site'#1 Remote'site'#2
SW SW SW SW
IPS'
40GbE'(x'142)
SINET5'
Campus'DC
SDN' ontroller
22

22. Roadmap & Future direction
• 2016Q2: Upgrade network infrastructure (SINET5: 100Gbps)
• 2017Q2-Q3: Replacing inter-cloud infrastructure (including remote sites)
& supercomputer at Hokkaido university
• Regional inter-cloud collaborations in Hokkaido
• National inter-cloud collaborations with other universities, NII and other
research institutes to establish academic community cloud federations
• International inter-cloud collaborations (Asia-Pacific?)
• Investigations on future trends in inter-cloud applications such as IoT/
IoE, extreme-scale parallel and distributed computing including big data
processing and machine learning.
23

23. CloudWeek2015@Hokkaido University
• A collection of symposium, conference, and workshop related cloud
computing technologies, sponsored by information initiative center, Hokkaido
University.
• Sep.7th - 9th or 10th, 2015, at Hokkaido University, Sapporo, Japan.
• Academic Inter-Cloud Symposium 2015 for Universities, Research institutes
• Open Cloud Conference 2015 for Cloud service providers, vendors, etc.
• ITRC RICC (Regional Inter-Cloud Committee)
Workshop
• Call for international speakers!
Cloud Week@Hokkaido University
(cloud symposium)
Cloud Week 2013@Hokkaido University
K
R
fo
Te
RIKE
for C
「京」
● Kyushu University Research Institute
for Information Technology
O
inno
con
ers
(SIN
Th
cha
othe
O
prov
Cen
Pur
● To
high
com
any
● T
rese
com
nee
Th
serv
phy
As
com
Inte
mai
As the Hokkaido University Academic Cloud, which is one of Japan’s largest
academic clouds, was established in November 2011, symposiums that
contribute to exchange of opinions on the current status and future develop-
ment of cloud research have been held yearly since FY 2012, by gathering
cloud-related researchers from Japan and abroad.
The FY 2013 symposium, which was held for three days and involved more
than 300 participants contributed to the development of research technology by
inviting leaders of various fields in cloud-related technology, holding lectures
and exchanging detailed information.
■ Effective period of acknowledgment:
April 1, 2010 – March 31, 2016
■ Purpose of the base
The purpose of this network-type base is to contribute to further
advancement and constant development of Japan’s academic and research
bases through interdisciplinary joint usage/research concerning so-called grand
challenges, which have been considered to be extremely difficult to solve or
clarify, using super-large-sized computers, super-high-capacity storages/net-
works and other information infrastructures. It covers information processing
fields in general, including the global environment, energy, substances/materi-
als, genome information, web data, academic information, time-series data
from sensor networks, image data and program analysis.
■ Operation of the base
JHPCN is operated by the Steering Committee and Joint Research Project
Screening Committee established at the University of Tokyo’s Information
Technology Center, which is its core base.
■ Promotion of open-type joint research
For interdisciplinary research using large-scale information infrastructure,
JHPCN is conducting 34 joint research projects in FY 2014 (of which 7 is related
to our Center), by seeking research projects concerning application of
ultra-large-scale numerical calculation and data processing systems and
ultra-high-capacity network technology, as well as the field of ultra-large-scale
information systems integrating these technologies from the public.
24