SlideShare uma empresa Scribd logo

RISC-V 30908 patra

•
•
RISC-V International
RISC-V International

RISC-V Summit 2020 presentation

Tecnologia
1 de 30
Baixar para ler offline
© 2020 Western Digital Corporation or its affiliates. All rights reserved. 11/18/20 Building Cache-Coherent Scaleout Systems With OmniXtend Atish Patra, Tu Dang, Anup Patel, Damien Le Moal, Dejan Vucinic
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 2 Agenda Compute Node Architecture System Models Unified boot process Protocol Simulation System emulation Conclusion
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 3 OmniXtend Architectures Source: An Open and Coherent Memory Centric Architecture Enabled by RISC-V, Dejan Vucinic OmniXtend is a fully open cache-coherence protocol that works over ethernet layer (L2)
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 4 OmniXtend Reference Design Memory Fabric Innovation Platform Standardize RISC-V coherency bus leveraging OmniXtend
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 5 An OmniXtend Compute Node High-level view of each compute node Local DRAM Interconnect (TileLink) HARTs OmniXtend Bus Network Local Devices PLIC CLINT PCIe Ethernet SD/eMMC SPI Other Devices Remote Memory
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 6 Compute Node Address Space High-level view of physical address space Local MMIO (Non-cacheable) Global RAM (Cacheable) Global MMIO (Non-cacheable) • At reset time, node’s own RAM will be mapped to different part of Global RAM space • RAM from other nodes can be mapped in Global RAM space • Local MMIO space always maps to a node’s own MMIO devices • MMIO devices of other nodes, will be mapped in Global MMIO • Node’s own MMIO devices can also be mapped in Global MMIO space Local RAM (Cacheable but not shared) • Local RAM is only accessible to local node
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 7 OmniXtend Hardware Design High level view M Dynamic Xbar (Cacheable channels) • Reads/Writes to RAM of other nodes use both Tx and Rx channels • Outgoing/Incoming cache coherency messages use Tx/Rx channels • Dedicated TLoE endpoint similar to C dynamic Xbar C Dynamic Xbar (Non-cacheable channels) • Reads/Writes to MMIO of other nodes use both Tx and Rx channels • Each Tx/Rx channel has a dedicated TLoE endpoint Local MMIO (Non-cacheable) Global RAM (Cacheable) Global MMIO (Non-cacheable) Local RAM (Cacheable but not shared) Cores Cacheable (M)emory Space >= 0x20_0000_000 (C)ontrol/MMIO Space < 0x20_0000_0000 AddressAduster Local/Remote AddressAduster Local/Remote M Dynamic Xbar Local Devices AddressControl HartId Prefix M Prefix C Prefix Local Memory C Dynamic Xbar TLoE TX TLoE TX TLoE TX TLoE TX TLoE TX TLoE TX TLoE RX TLoE RX TLoE RX PacketPickerTX (add Ethernet Hdr) MAC TX Network MAC TX PacketPicker RX (strip Ethernet Hdr) TLoE RX TLoE RX TLoE RX Configuration MAC RX Cacheable (M)emory space > 0x4000_0000_0000 (C)ontrol MMIO space < 0x3FFF-FFFF_FFFF
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 8 Node 0 Single OS (SO) Model Similar to a regular SMP/NUMA system CPU0 CPU1 memory0 CPU2 CPU3 Node 1 CPU0 CPU1 memory1 CPU2 CPU3 System CPU0 CPU1 memory0 CPU2 CPU3 CPU4 CPU5 memory1 CPU6 CPU7 NUMA node 0 NUMA node 1 OmniXtend fabric
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 9 Current Status • Up to 4 OmniXtend nodes with NUMA enabled • Dynamic topology reconfiguration with DIP switch • Point-to-Point via SN2010 switch • Based on latest upstream Linux kernel (v5.9) Working NUMA setup
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 10 Can we do better ? • NUMA model doesn’t scale beyond few nodes – Expensive – Power management issues – Reliability • Wide adoption of distributed application model over clustered network servers – Scales well but higher latency – New programming model in absence of cache coherency • We need a cache coherent system that can scale!! To achieve better scale out models
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 11 Independent Nodes (IN) Model • Each node boots independent instance of Linux • Access shared memory exposed by a kernel driver Experimental approach that can scale Node 0 CPU0 CPU1 memory0 CPU2 CPU3 Node 1 CPU0 CPU1 memory1 CPU2 CPU3 OmniXtend fabric shared memory
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 12 OmniXtend Unified Boot Protocol • Allows to run either SO or IN model on the same hardware after reset • OnDemand reconfiguration based on the application requirement • Many possibilities of experimentation and exploration of research problems • Performance benchmarking with ease • No designated leader required Single boot process for all system models
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 13 Management Server • Management server can be implemented in – A separate host machine that is connected to the OmniXtend network – One of the Compute Node • Does not require to participate in all OmniXtend traffic but should be capable of communicating with all nodes via raw ethernet frames • Assigns individual node’s role and memory ranges – Every node boots up independently and wait for role to be assigned by the management server – Implementation is flexible and can be used to create different topologies Manages dynamic discovery and initialization Management server Node 0 (compute & memory) Node 2 (compute & memory) Node 3 (compute & memory) Node N (memory only) Node N (hardware accelerator)
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 14 Boot process (SO system) • BSP (bootstrap processor == first HART to run) wakes up other HARTs once low-level memory initialization is completed in OpenSBI Similar to a regular NUMA system boot flow Linux (S-mode) (OS) ZSBL (M-mode) (ROM) FSBL (M-mode) (LOADER) OpenSBI (M-mode) ZSBL (M-mode) (ROM) FSBL (M-mode) (LOADER) OpenSBI (M-mode) ZSBL (M-mode) (ROM) FSBL (M-mode) (LOADER) OpenSBI (M-mode) All remote memory is accessible by kernel and user space Board1 Board2 Board3
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 15 Boot process (IN system) A distributed system with shared memory Linux (S-mode) (OS) ZSBL (M-mode) (ROM) FSBL (M-mode) (LOADER) OpenSBI (M-mode) ZSBL (M-mode) (ROM) FSBL (M-mode) (LOADER) OpenSBI (M-mode) ZSBL (M-mode) (ROM) FSBL (M-mode) (LOADER) OpenSBI (M-mode) Board1 Board2 Board3 Linux (S-mode) (OS) Linux (S-mode) (OS) On-demand shared memory • Shared memory implemented on demand (e.g. mmap()) with remapping of dynamically allocated node-local and remote DRAM pages • Can scale up to hundreds of nodes
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 16 Can we develop faster ? A software centric approach Simulation/Emulation can fast track research Easy to experiment and explore as successive quick iteration in software Reduce CapEx for Research and Development Early verify the correctness and scalability of the protocol Easy to extend other ISAs (x86, ARM64) in future
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 17 What is protocol simulation? • Simulate the interaction of OmniXtend endpoints • Implemented as a C library • Include unit tests for verification of the protocol • Quickly adapt to the protocol update Protocol Simulation
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 18 Extendable plugins in the library • Plugins are independent from each other • Developers can provide their own implementation • Current implementation – OX_CACHE: Configurable LRU cache – OX_PROTOCOL: TileLink over Ethernet (TLoE) – OX_TRANSPORT: Raw Socket Allowing configurability and extendibility TRANSPORT CACHE PROTOCOL (rawsock, IP/TCP, RDMA) Snoopy, Directory-based Direct Mapped, LRU, FIFO DEVICE
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 19 Protocol simulation overview User input Protocol Simulation library OXSE_DEV QTest System emulation Test Application OXPS_DEV GoogleTest OXPS_APP OX_PROTOCOL OX_TRANSPORT OX_CACHE Address checker memory ops callback
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 20 What is system emulation? • An OmniXtend system consists of distributed compute, memory, storage and optional accelerator nodes • Nodes are connected to a traditional switch or programmable switch with enhanced features Implement the entire OmniXtend protocol in Qemu Switch OmniXtend Node 0 phy kernel Qemu emulation OmniXtend Node 1 phy kernel Qemu emulation NVM NVM NVM phy FPGA/ASIC OmniXtend Node 2 phy ML Accelerator phy
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 21 System Emulation Overview A Qemu device emulates OmniXtend device Kernel User space Qemu process phy virtio net- device Guest Kernel Guest User space Hardware Virtio disk OmniXtend device Read/Write to non-cacheable remote memory Access ethernet packets using a raw socket Implement the OmniXtend protocol Address checker memory ops callback Memory backend LRU Cache Read/Write to cacheable memory Local Remote Remote cacheable memory request mmio device
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 22 They co-exist & complement each other • Complete system level emulation • Verify the initialization protocol • Bring up large number of nodes Pros • Low performance • Can’t emulate hardware accelerator Cons • Simple to setup • Test suites to ensure correctness • Doesn’t require special hardware Pros • Cannot boot Operating Systems • Low performance Cons System emulation Protocol Simulation Simulation & Emulation Summary
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 23 Future work • Implement kernel support for Independent Node model – User space applications can mmap & use the shared memory – Kernel may choose to protect its own page table memory from remote access via PMP – Kernel driver may handle the remote memory page faults – A user space library adds to support to existing distributed application framework • Experiment different topologies with a different rules in management server • Improve Simulation/Emulation using extensive test suites • Hook up with the Qemu instances with OmniXtend FPGAs • Exercise the scalability test with many nodes • Opensource the simulation library and Qemu To infinity and beyond!
© 2020 Western Digital Corporation or its affiliates. All rights reserved. 11/18/20
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 25 Backup
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 26 Types of messages Type Sender Intended Receiver Mode of transmission Payload Intended action HELLO OE MS Broadcast None send and wait WELCOME MS OE Unicast None Continue MY_INFO OE MS Unicast Local OE DTB send and wait DTB_INFO MS OE Unicast Full OmniXtend system DTB parse OmniXtend dt node and configure channels CONFIG_DONE OE MS Unicast None send and wait ENABLE MS OE Unicast None enable TLoE in given order DT ENABLE_DONE OE MS Unicast None send and wait BOOT MS OE Unicast None Reconfigure memory and boot • OmniXtend Endpoint (OE) • Management server (MS)
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 27 Booting SO system Before TLoE is enabled Configure TX/RX Channels Enable TLoE end points in given order OE(Non-Boot) MS OE (Boot) Boot hart Configure ethernet Send HELLO and wait for WELCOME HELLO Unicast WELCOME packet WELCOME MY_INFO DTB_INFO Send MY_INFO and wait for DTB_INFO Multicast full DTB with OmniXtend topology to Non-booting node Parses OmniXtend topology ENABLE Send CONFIG_DONE and wait for ENABLE Configure TX/RX Channels CONFIG_DONE Enable TLoE end points in given order Send ENABLE message after CONFIG_DONE is received from every node Configure ethernet Send HELLO and wait for WELCOME HELLO WELCOME MY_INFO DTB_INFO Send MY_INFO and wait for DTB_INFO CONFIG_DONE Parses OmniXtend topology Multicast full DTB with OmniXtend topology to booting node ENABLE Boot hart
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 28 Booting SO system After TLoE is enabled Send ENABLE_DONE and wait for BOOT Configure MMIO devices OE(Non-Boot) MS OE (Boot) Boot hart Send ENABLE_DONE and wait for BOOT ENABLE_DONE BOOT Reconfigure MMIO address Send BOOT message after ENABLE_DONE is received from every node ENABLE_DONE BOOT Boot hart Reconfigure cacheable memory address Jump Non-boot hart Parse full DT Non-boot hart Warm boot Linux Jump to Linux Normal OpenSBI boot
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 29 Booting IN system Before TLoE is enabled OE(Boot) MS OE (Boot) Boot hart Configure ethernet Send HELLO and wait for WELCOME HELLO Unicast WELCOME packet WELCOME MY_INFO DTB_INFO Send MY_INFO and wait for DTB_INFO Multicast full DTB with OmniXtend topology to Non-booting node Parses OmniXtend topology ENABLE Send CONFIG_DONE and wait for ENABLE Configure TX/RX Channels CONFIG_DONE Enable TLoE end points in given order Send ENABLE message after CONFIG_DONE is received from every node Configure ethernet Send HELLO and wait for WELCOME HELLO WELCOME MY_INFO DTB_INFO Send MY_INFO and wait for DTB_INFO CONFIG_DONE Parses OmniXtend topology Configure TX/RX Channels ENABLE Enable TLoE end points in given order Send CONFIG_DONE and wait for ENABLE
11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 30 Booting IN system After TLoE is enabled OE(Boot) MS OE (Boot) Boot hart Send ENABLE_DONE and wait for BOOT ENABLE_DONE BOOT Reconfigure MMIO address Send BOOT message after ENABLE_DONE is received from every node Send ENABLE_DONE and wait for BOOT ENABLE_DONE BOOT Boot hart Reconfigure cacheable memory address Non-boot hart Configure MMIO devices Non-boot hart Warm boot Linux Jump to Linux Normal OpenSBI boot Warm boot Normal OpenSBI boot Linux Jump to Linux On-demand shared memory using a Linux OmniXtend driver Reconfigure cacheable memory address

Recomendados

RISC-V Zce Extension
RISC-V Zce ExtensionRISC-V Zce Extension
RISC-V Zce Extension
RISC-V International
 
Static partitioning virtualization on RISC-V
Static partitioning virtualization on RISC-VStatic partitioning virtualization on RISC-V
Static partitioning virtualization on RISC-V
RISC-V International
 
RISC-V NOEL-V - A new high performance RISC-V Processor Family
RISC-V NOEL-V - A new high performance RISC-V Processor FamilyRISC-V NOEL-V - A new high performance RISC-V Processor Family
RISC-V NOEL-V - A new high performance RISC-V Processor Family
RISC-V International
 
Reverse Engineering of Rocket Chip
Reverse Engineering of Rocket ChipReverse Engineering of Rocket Chip
Reverse Engineering of Rocket Chip
RISC-V International
 
An Open Discussion of RISC-V BitManip, trends, and comparisons _ Cuff
An Open Discussion of RISC-V BitManip, trends, and comparisons _ Cuff An Open Discussion of RISC-V BitManip, trends, and comparisons _ Cuff
An Open Discussion of RISC-V BitManip, trends, and comparisons _ Cuff
RISC-V International
 
Online test program generator for RISC-V processors
Online test program generator for RISC-V processorsOnline test program generator for RISC-V processors
Online test program generator for RISC-V processors
RISC-V International
 
Andes RISC-V processor solutions
Andes RISC-V processor solutionsAndes RISC-V processor solutions
Andes RISC-V processor solutions
RISC-V International
 
Tech talk with lampro mellon an open source solution for accelerating verific...
Tech talk with lampro mellon an open source solution for accelerating verific...Tech talk with lampro mellon an open source solution for accelerating verific...
Tech talk with lampro mellon an open source solution for accelerating verific...
RISC-V International
 

Recomendados

RISC-V Zce Extension
RISC-V Zce ExtensionRISC-V Zce Extension
RISC-V Zce Extension
RISC-V International
 
Static partitioning virtualization on RISC-V
Static partitioning virtualization on RISC-VStatic partitioning virtualization on RISC-V
Static partitioning virtualization on RISC-V
RISC-V International
 
RISC-V NOEL-V - A new high performance RISC-V Processor Family
RISC-V NOEL-V - A new high performance RISC-V Processor FamilyRISC-V NOEL-V - A new high performance RISC-V Processor Family
RISC-V NOEL-V - A new high performance RISC-V Processor Family
RISC-V International
 
Reverse Engineering of Rocket Chip
Reverse Engineering of Rocket ChipReverse Engineering of Rocket Chip
Reverse Engineering of Rocket Chip
RISC-V International
 
An Open Discussion of RISC-V BitManip, trends, and comparisons _ Cuff
An Open Discussion of RISC-V BitManip, trends, and comparisons _ Cuff An Open Discussion of RISC-V BitManip, trends, and comparisons _ Cuff
An Open Discussion of RISC-V BitManip, trends, and comparisons _ Cuff
RISC-V International
 
Online test program generator for RISC-V processors
Online test program generator for RISC-V processorsOnline test program generator for RISC-V processors
Online test program generator for RISC-V processors
RISC-V International
 
Andes RISC-V processor solutions
Andes RISC-V processor solutionsAndes RISC-V processor solutions
Andes RISC-V processor solutions
RISC-V International
 
Tech talk with lampro mellon an open source solution for accelerating verific...
Tech talk with lampro mellon an open source solution for accelerating verific...Tech talk with lampro mellon an open source solution for accelerating verific...
Tech talk with lampro mellon an open source solution for accelerating verific...
RISC-V International
 
Fueling the datasphere how RISC-V enables the storage ecosystem
Fueling the datasphere how RISC-V enables the storage ecosystemFueling the datasphere how RISC-V enables the storage ecosystem
Fueling the datasphere how RISC-V enables the storage ecosystem
RISC-V International
 
An open flow for dn ns on ultra low-power RISC-V cores
An open flow for dn ns on ultra low-power RISC-V coresAn open flow for dn ns on ultra low-power RISC-V cores
An open flow for dn ns on ultra low-power RISC-V cores
RISC-V International
 
RISC-V growth and successes in technology and industry - embedded world 2021
RISC-V growth and successes in technology and industry - embedded world 2021RISC-V growth and successes in technology and industry - embedded world 2021
RISC-V growth and successes in technology and industry - embedded world 2021
RISC-V International
 
RISC-V 30907 summit 2020 joint picocom_mentor
RISC-V 30907 summit 2020 joint picocom_mentorRISC-V 30907 summit 2020 joint picocom_mentor
RISC-V 30907 summit 2020 joint picocom_mentor
RISC-V International
 
Secure IoT Firmware for RISC-V
Secure IoT Firmware for RISC-VSecure IoT Firmware for RISC-V
Secure IoT Firmware for RISC-V
RISC-V International
 
Semi dynamics high bandwidth vector capable RISC-V cores
Semi dynamics high bandwidth vector capable RISC-V coresSemi dynamics high bandwidth vector capable RISC-V cores
Semi dynamics high bandwidth vector capable RISC-V cores
RISC-V International
 
Getting started with RISC-V verification what's next after compliance testing
Getting started with RISC-V verification what's next after compliance testingGetting started with RISC-V verification what's next after compliance testing
Getting started with RISC-V verification what's next after compliance testing
RISC-V International
 
Andes building a secure platform with the enhanced iopmp
Andes building a secure platform with the enhanced iopmpAndes building a secure platform with the enhanced iopmp
Andes building a secure platform with the enhanced iopmp
RISC-V International
 
Andes andes clarity for risc-v vector processor
Andes andes clarity for risc-v vector processorAndes andes clarity for risc-v vector processor
Andes andes clarity for risc-v vector processor
RISC-V International
 
RISC-V assembly
RISC-V assemblyRISC-V assembly
RISC-V assembly
Peter Cheung
 
Klessydra t - designing vector coprocessors for multi-threaded edge-computing...
Klessydra t - designing vector coprocessors for multi-threaded edge-computing...Klessydra t - designing vector coprocessors for multi-threaded edge-computing...
Klessydra t - designing vector coprocessors for multi-threaded edge-computing...
RISC-V International
 
Andes open cl for RISC-V
Andes open cl for RISC-VAndes open cl for RISC-V
Andes open cl for RISC-V
RISC-V International
 
An Open Discussion of RISC-V BitManip, trends, and comparisons _ Claire
An Open Discussion of RISC-V BitManip, trends, and comparisons _ Claire An Open Discussion of RISC-V BitManip, trends, and comparisons _ Claire
An Open Discussion of RISC-V BitManip, trends, and comparisons _ Claire
RISC-V International
 
Klessydra-T: Designing Configurable Vector Co-Processors for Multi-Threaded E...
Klessydra-T: Designing Configurable Vector Co-Processors for Multi-Threaded E...Klessydra-T: Designing Configurable Vector Co-Processors for Multi-Threaded E...
Klessydra-T: Designing Configurable Vector Co-Processors for Multi-Threaded E...
RISC-V International
 
Closing the RISC-V compliance gap via fuzzing
Closing the RISC-V compliance gap via fuzzingClosing the RISC-V compliance gap via fuzzing
Closing the RISC-V compliance gap via fuzzing
RISC-V International
 
RISC-V & SoC Architectural Exploration for AI and ML Accelerators
RISC-V & SoC Architectural Exploration for AI and ML AcceleratorsRISC-V & SoC Architectural Exploration for AI and ML Accelerators
RISC-V & SoC Architectural Exploration for AI and ML Accelerators
RISC-V International
 
SemiDynamics new family of High Bandwidth Vector-capable Cores
SemiDynamics new family of High Bandwidth Vector-capable CoresSemiDynamics new family of High Bandwidth Vector-capable Cores
SemiDynamics new family of High Bandwidth Vector-capable Cores
RISC-V International
 
System Design on Zynq using SDSoC
System Design on Zynq using SDSoCSystem Design on Zynq using SDSoC
System Design on Zynq using SDSoC
Sundance Multiprocessor Technology Ltd.
 
Embedded Recipes 2019 - Herd your socs become a matchmaker
Embedded Recipes 2019 - Herd your socs become a matchmakerEmbedded Recipes 2019 - Herd your socs become a matchmaker
Embedded Recipes 2019 - Herd your socs become a matchmaker
Anne Nicolas
 
Esperanto accelerates machine learning with 1000+ low power RISC-V cores on a...
Esperanto accelerates machine learning with 1000+ low power RISC-V cores on a...Esperanto accelerates machine learning with 1000+ low power RISC-V cores on a...
Esperanto accelerates machine learning with 1000+ low power RISC-V cores on a...
RISC-V International
 
HKG18-110 - net_mdev: Fast path user space I/O
HKG18-110 - net_mdev: Fast path user space I/OHKG18-110 - net_mdev: Fast path user space I/O
HKG18-110 - net_mdev: Fast path user space I/O
Linaro
 
Porting_uClinux_CELF2008_Griffin
Porting_uClinux_CELF2008_GriffinPorting_uClinux_CELF2008_Griffin
Porting_uClinux_CELF2008_Griffin
Peter Griffin
 

Mais conteúdo relacionado

Mais procurados

RISC-V growth and successes in technology and industry - embedded world 2021
RISC-V growth and successes in technology and industry - embedded world 2021RISC-V growth and successes in technology and industry - embedded world 2021
RISC-V growth and successes in technology and industry - embedded world 2021
RISC-V International
 
Embedded Recipes 2019 - Herd your socs become a matchmaker
Embedded Recipes 2019 - Herd your socs become a matchmakerEmbedded Recipes 2019 - Herd your socs become a matchmaker
Embedded Recipes 2019 - Herd your socs become a matchmaker
Anne Nicolas
 

Mais procurados (20)

Fueling the datasphere how RISC-V enables the storage ecosystem
Fueling the datasphere how RISC-V enables the storage ecosystemFueling the datasphere how RISC-V enables the storage ecosystem
Fueling the datasphere how RISC-V enables the storage ecosystem
 
An open flow for dn ns on ultra low-power RISC-V cores
An open flow for dn ns on ultra low-power RISC-V coresAn open flow for dn ns on ultra low-power RISC-V cores
An open flow for dn ns on ultra low-power RISC-V cores
 
RISC-V growth and successes in technology and industry - embedded world 2021
RISC-V growth and successes in technology and industry - embedded world 2021RISC-V growth and successes in technology and industry - embedded world 2021
RISC-V growth and successes in technology and industry - embedded world 2021
 
RISC-V 30907 summit 2020 joint picocom_mentor
RISC-V 30907 summit 2020 joint picocom_mentorRISC-V 30907 summit 2020 joint picocom_mentor
RISC-V 30907 summit 2020 joint picocom_mentor
 
Secure IoT Firmware for RISC-V
Secure IoT Firmware for RISC-VSecure IoT Firmware for RISC-V
Secure IoT Firmware for RISC-V
 
Semi dynamics high bandwidth vector capable RISC-V cores
Semi dynamics high bandwidth vector capable RISC-V coresSemi dynamics high bandwidth vector capable RISC-V cores
Semi dynamics high bandwidth vector capable RISC-V cores
 
Getting started with RISC-V verification what's next after compliance testing
Getting started with RISC-V verification what's next after compliance testingGetting started with RISC-V verification what's next after compliance testing
Getting started with RISC-V verification what's next after compliance testing
 
Andes building a secure platform with the enhanced iopmp
Andes building a secure platform with the enhanced iopmpAndes building a secure platform with the enhanced iopmp
Andes building a secure platform with the enhanced iopmp
 
Andes andes clarity for risc-v vector processor
Andes andes clarity for risc-v vector processorAndes andes clarity for risc-v vector processor
Andes andes clarity for risc-v vector processor
 
RISC-V assembly
RISC-V assemblyRISC-V assembly
RISC-V assembly
 
Klessydra t - designing vector coprocessors for multi-threaded edge-computing...
Klessydra t - designing vector coprocessors for multi-threaded edge-computing...Klessydra t - designing vector coprocessors for multi-threaded edge-computing...
Klessydra t - designing vector coprocessors for multi-threaded edge-computing...
 
Andes open cl for RISC-V
Andes open cl for RISC-VAndes open cl for RISC-V
Andes open cl for RISC-V
 
An Open Discussion of RISC-V BitManip, trends, and comparisons _ Claire
An Open Discussion of RISC-V BitManip, trends, and comparisons _ Claire An Open Discussion of RISC-V BitManip, trends, and comparisons _ Claire
An Open Discussion of RISC-V BitManip, trends, and comparisons _ Claire
 
Klessydra-T: Designing Configurable Vector Co-Processors for Multi-Threaded E...
Klessydra-T: Designing Configurable Vector Co-Processors for Multi-Threaded E...Klessydra-T: Designing Configurable Vector Co-Processors for Multi-Threaded E...
Klessydra-T: Designing Configurable Vector Co-Processors for Multi-Threaded E...
 
Closing the RISC-V compliance gap via fuzzing
Closing the RISC-V compliance gap via fuzzingClosing the RISC-V compliance gap via fuzzing
Closing the RISC-V compliance gap via fuzzing
 
RISC-V & SoC Architectural Exploration for AI and ML Accelerators
RISC-V & SoC Architectural Exploration for AI and ML AcceleratorsRISC-V & SoC Architectural Exploration for AI and ML Accelerators
RISC-V & SoC Architectural Exploration for AI and ML Accelerators
 
SemiDynamics new family of High Bandwidth Vector-capable Cores
SemiDynamics new family of High Bandwidth Vector-capable CoresSemiDynamics new family of High Bandwidth Vector-capable Cores
SemiDynamics new family of High Bandwidth Vector-capable Cores
 
System Design on Zynq using SDSoC
System Design on Zynq using SDSoCSystem Design on Zynq using SDSoC
System Design on Zynq using SDSoC
 
Embedded Recipes 2019 - Herd your socs become a matchmaker
Embedded Recipes 2019 - Herd your socs become a matchmakerEmbedded Recipes 2019 - Herd your socs become a matchmaker
Embedded Recipes 2019 - Herd your socs become a matchmaker
 
Esperanto accelerates machine learning with 1000+ low power RISC-V cores on a...
Esperanto accelerates machine learning with 1000+ low power RISC-V cores on a...Esperanto accelerates machine learning with 1000+ low power RISC-V cores on a...
Esperanto accelerates machine learning with 1000+ low power RISC-V cores on a...
 

Semelhante a RISC-V 30908 patra

HKG18-110 - net_mdev: Fast path user space I/O
HKG18-110 - net_mdev: Fast path user space I/OHKG18-110 - net_mdev: Fast path user space I/O
HKG18-110 - net_mdev: Fast path user space I/O
Linaro
 
Porting_uClinux_CELF2008_Griffin
Porting_uClinux_CELF2008_GriffinPorting_uClinux_CELF2008_Griffin
Porting_uClinux_CELF2008_Griffin
Peter Griffin
 
XPDDS17: Keynote: Shared Coprocessor Framework on ARM - Oleksandr Andrushchen...
XPDDS17: Keynote: Shared Coprocessor Framework on ARM - Oleksandr Andrushchen...XPDDS17: Keynote: Shared Coprocessor Framework on ARM - Oleksandr Andrushchen...
XPDDS17: Keynote: Shared Coprocessor Framework on ARM - Oleksandr Andrushchen...
The Linux Foundation
 
Icg hpc-user
Icg hpc-userIcg hpc-user
Icg hpc-user
gdburton
 
HKG18-411 - Introduction to OpenAMP which is an open source solution for hete...
HKG18-411 - Introduction to OpenAMP which is an open source solution for hete...HKG18-411 - Introduction to OpenAMP which is an open source solution for hete...
HKG18-411 - Introduction to OpenAMP which is an open source solution for hete...
Linaro
 
Enhancing the Open-Source P-Mesh Cache Coherence System for Open ISAs
Enhancing the Open-Source P-Mesh Cache Coherence System for Open ISAsEnhancing the Open-Source P-Mesh Cache Coherence System for Open ISAs
Enhancing the Open-Source P-Mesh Cache Coherence System for Open ISAs
Ganesan Narayanasamy
 

Semelhante a RISC-V 30908 patra (20)

HKG18-110 - net_mdev: Fast path user space I/O
HKG18-110 - net_mdev: Fast path user space I/OHKG18-110 - net_mdev: Fast path user space I/O
HKG18-110 - net_mdev: Fast path user space I/O
 
Porting_uClinux_CELF2008_Griffin
Porting_uClinux_CELF2008_GriffinPorting_uClinux_CELF2008_Griffin
Porting_uClinux_CELF2008_Griffin
 
Vx Rack : L'hyperconvergence avec l'experience VCE
Vx Rack : L'hyperconvergence avec l'experience VCEVx Rack : L'hyperconvergence avec l'experience VCE
Vx Rack : L'hyperconvergence avec l'experience VCE
 
XPDDS17: Keynote: Shared Coprocessor Framework on ARM - Oleksandr Andrushchen...
XPDDS17: Keynote: Shared Coprocessor Framework on ARM - Oleksandr Andrushchen...XPDDS17: Keynote: Shared Coprocessor Framework on ARM - Oleksandr Andrushchen...
XPDDS17: Keynote: Shared Coprocessor Framework on ARM - Oleksandr Andrushchen...
 
[OpenStack Day in Korea 2015] Track 1-6 - 갈라파고스의 이구아나, 인프라에 오픈소스를 올리다. 그래서 보이...
[OpenStack Day in Korea 2015] Track 1-6 - 갈라파고스의 이구아나, 인프라에 오픈소스를 올리다. 그래서 보이...[OpenStack Day in Korea 2015] Track 1-6 - 갈라파고스의 이구아나, 인프라에 오픈소스를 올리다. 그래서 보이...
[OpenStack Day in Korea 2015] Track 1-6 - 갈라파고스의 이구아나, 인프라에 오픈소스를 올리다. 그래서 보이...
 
Phytium 64 core cpu preview
Phytium 64 core cpu previewPhytium 64 core cpu preview
Phytium 64 core cpu preview
 
Current and Future of Non-Volatile Memory on Linux
Current and Future of Non-Volatile Memory on LinuxCurrent and Future of Non-Volatile Memory on Linux
Current and Future of Non-Volatile Memory on Linux
 
Enduro/X Middleware
Enduro/X MiddlewareEnduro/X Middleware
Enduro/X Middleware
 
Towards Software Defined Persistent Memory
Towards Software Defined Persistent MemoryTowards Software Defined Persistent Memory
Towards Software Defined Persistent Memory
 
Icg hpc-user
Icg hpc-userIcg hpc-user
Icg hpc-user
 
openPOWERLINK over Xenomai
openPOWERLINK over XenomaiopenPOWERLINK over Xenomai
openPOWERLINK over Xenomai
 
2014/09/02 Cisco UCS HPC @ ANL
2014/09/02 Cisco UCS HPC @ ANL2014/09/02 Cisco UCS HPC @ ANL
2014/09/02 Cisco UCS HPC @ ANL
 
Massively Parallel RISC-V Processing with Transactional Memory
Massively Parallel RISC-V Processing with Transactional MemoryMassively Parallel RISC-V Processing with Transactional Memory
Massively Parallel RISC-V Processing with Transactional Memory
 
HKG18-411 - Introduction to OpenAMP which is an open source solution for hete...
HKG18-411 - Introduction to OpenAMP which is an open source solution for hete...HKG18-411 - Introduction to OpenAMP which is an open source solution for hete...
HKG18-411 - Introduction to OpenAMP which is an open source solution for hete...
 
Technical sales education enterprise- svc and ibm flash best practices update
Technical sales education enterprise- svc and ibm flash best practices updateTechnical sales education enterprise- svc and ibm flash best practices update
Technical sales education enterprise- svc and ibm flash best practices update
 
ONIE: Open Network Install Environment @ OSDC 2014 Netways, Berlin
ONIE: Open Network Install Environment @ OSDC 2014 Netways, BerlinONIE: Open Network Install Environment @ OSDC 2014 Netways, Berlin
ONIE: Open Network Install Environment @ OSDC 2014 Netways, Berlin
 
Enhancing the Open-Source P-Mesh Cache Coherence System for Open ISAs
Enhancing the Open-Source P-Mesh Cache Coherence System for Open ISAsEnhancing the Open-Source P-Mesh Cache Coherence System for Open ISAs
Enhancing the Open-Source P-Mesh Cache Coherence System for Open ISAs
 
Presentation vmax hardware deep dive
Presentation vmax hardware deep divePresentation vmax hardware deep dive
Presentation vmax hardware deep dive
 
My First 100 days with an Exadata (PPT)
My First 100 days with an Exadata (PPT)My First 100 days with an Exadata (PPT)
My First 100 days with an Exadata (PPT)
 
The Cell Processor
The Cell ProcessorThe Cell Processor
The Cell Processor
 

Mais de RISC-V International

Mais de RISC-V International (20)

WD RISC-V inliner work effort
WD RISC-V inliner work effortWD RISC-V inliner work effort
WD RISC-V inliner work effort
 
RISC-V Online Tutor
RISC-V Online TutorRISC-V Online Tutor
RISC-V Online Tutor
 
London Open Source Meetup for RISC-V
London Open Source Meetup for RISC-VLondon Open Source Meetup for RISC-V
London Open Source Meetup for RISC-V
 
RISC-V Introduction
RISC-V IntroductionRISC-V Introduction
RISC-V Introduction
 
Ziptillion boosting RISC-V with an efficient and os transparent memory comp...
Ziptillion boosting RISC-V with an efficient and os transparent memory comp...Ziptillion boosting RISC-V with an efficient and os transparent memory comp...
Ziptillion boosting RISC-V with an efficient and os transparent memory comp...
 
Standardizing the tee with global platform and RISC-V
Standardizing the tee with global platform and RISC-VStandardizing the tee with global platform and RISC-V
Standardizing the tee with global platform and RISC-V
 
Security and functional safety
Security and functional safetySecurity and functional safety
Security and functional safety
 
RISC-V 30910 kassem_ summit 2020 - so_c_gen
RISC-V 30910 kassem_ summit 2020 - so_c_genRISC-V 30910 kassem_ summit 2020 - so_c_gen
RISC-V 30910 kassem_ summit 2020 - so_c_gen
 
RISC-V 30906 hex five multi_zone iot firmware
RISC-V 30906 hex five multi_zone iot firmwareRISC-V 30906 hex five multi_zone iot firmware
RISC-V 30906 hex five multi_zone iot firmware
 
RISC-V 30946 manuel_offenberg_v3_notes
RISC-V 30946 manuel_offenberg_v3_notesRISC-V 30946 manuel_offenberg_v3_notes
RISC-V 30946 manuel_offenberg_v3_notes
 
RISC-V software state of the union
RISC-V software state of the unionRISC-V software state of the union
RISC-V software state of the union
 
Ripes tracking computer architecture throught visual and interactive simula...
Ripes tracking computer architecture throught visual and interactive simula...Ripes tracking computer architecture throught visual and interactive simula...
Ripes tracking computer architecture throught visual and interactive simula...
 
Porting tock to open titan
Porting tock to open titanPorting tock to open titan
Porting tock to open titan
 
Open j9 jdk on RISC-V
Open j9 jdk on RISC-VOpen j9 jdk on RISC-V
Open j9 jdk on RISC-V
 
Open source manufacturable pdk for sky water 130nm process node
Open source manufacturable pdk for sky water 130nm process nodeOpen source manufacturable pdk for sky water 130nm process node
Open source manufacturable pdk for sky water 130nm process node
 
Gernot heiser unsw sydney and se l4 foundation
Gernot heiser unsw sydney and se l4 foundationGernot heiser unsw sydney and se l4 foundation
Gernot heiser unsw sydney and se l4 foundation
 
Educating the computer architects of tomorrow's critical systems with RISC-V
Educating the computer architects of tomorrow's critical systems with RISC-VEducating the computer architects of tomorrow's critical systems with RISC-V
Educating the computer architects of tomorrow's critical systems with RISC-V
 
Easily emulating full systems on amazon fpg as
Easily emulating full systems on amazon fpg asEasily emulating full systems on amazon fpg as
Easily emulating full systems on amazon fpg as
 
Developing for polar fire soc
Developing for polar fire socDeveloping for polar fire soc
Developing for polar fire soc
 
Data trustworthiness at the edge
Data trustworthiness at the edgeData trustworthiness at the edge
Data trustworthiness at the edge
 

Último

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc
 
Ransomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdfRansomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdf
Overkill Security
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
WSO2
 

Último (20)

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot ModelNavi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
 
A Beginners Guide to Building a RAG App Using Open Source Milvus
A Beginners Guide to Building a RAG App Using Open Source MilvusA Beginners Guide to Building a RAG App Using Open Source Milvus
A Beginners Guide to Building a RAG App Using Open Source Milvus
 
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu SubbuApidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
Ransomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdfRansomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdf
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 

RISC-V 30908 patra

  • 1. © 2020 Western Digital Corporation or its affiliates. All rights reserved. 11/18/20 Building Cache-Coherent Scaleout Systems With OmniXtend Atish Patra, Tu Dang, Anup Patel, Damien Le Moal, Dejan Vucinic
  • 2. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 2 Agenda Compute Node Architecture System Models Unified boot process Protocol Simulation System emulation Conclusion
  • 3. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 3 OmniXtend Architectures Source: An Open and Coherent Memory Centric Architecture Enabled by RISC-V, Dejan Vucinic OmniXtend is a fully open cache-coherence protocol that works over ethernet layer (L2)
  • 4. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 4 OmniXtend Reference Design Memory Fabric Innovation Platform Standardize RISC-V coherency bus leveraging OmniXtend
  • 5. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 5 An OmniXtend Compute Node High-level view of each compute node Local DRAM Interconnect (TileLink) HARTs OmniXtend Bus Network Local Devices PLIC CLINT PCIe Ethernet SD/eMMC SPI Other Devices Remote Memory
  • 6. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 6 Compute Node Address Space High-level view of physical address space Local MMIO (Non-cacheable) Global RAM (Cacheable) Global MMIO (Non-cacheable) • At reset time, node’s own RAM will be mapped to different part of Global RAM space • RAM from other nodes can be mapped in Global RAM space • Local MMIO space always maps to a node’s own MMIO devices • MMIO devices of other nodes, will be mapped in Global MMIO • Node’s own MMIO devices can also be mapped in Global MMIO space Local RAM (Cacheable but not shared) • Local RAM is only accessible to local node
  • 7. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 7 OmniXtend Hardware Design High level view M Dynamic Xbar (Cacheable channels) • Reads/Writes to RAM of other nodes use both Tx and Rx channels • Outgoing/Incoming cache coherency messages use Tx/Rx channels • Dedicated TLoE endpoint similar to C dynamic Xbar C Dynamic Xbar (Non-cacheable channels) • Reads/Writes to MMIO of other nodes use both Tx and Rx channels • Each Tx/Rx channel has a dedicated TLoE endpoint Local MMIO (Non-cacheable) Global RAM (Cacheable) Global MMIO (Non-cacheable) Local RAM (Cacheable but not shared) Cores Cacheable (M)emory Space >= 0x20_0000_000 (C)ontrol/MMIO Space < 0x20_0000_0000 AddressAduster Local/Remote AddressAduster Local/Remote M Dynamic Xbar Local Devices AddressControl HartId Prefix M Prefix C Prefix Local Memory C Dynamic Xbar TLoE TX TLoE TX TLoE TX TLoE TX TLoE TX TLoE TX TLoE RX TLoE RX TLoE RX PacketPickerTX (add Ethernet Hdr) MAC TX Network MAC TX PacketPicker RX (strip Ethernet Hdr) TLoE RX TLoE RX TLoE RX Configuration MAC RX Cacheable (M)emory space > 0x4000_0000_0000 (C)ontrol MMIO space < 0x3FFF-FFFF_FFFF
  • 8. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 8 Node 0 Single OS (SO) Model Similar to a regular SMP/NUMA system CPU0 CPU1 memory0 CPU2 CPU3 Node 1 CPU0 CPU1 memory1 CPU2 CPU3 System CPU0 CPU1 memory0 CPU2 CPU3 CPU4 CPU5 memory1 CPU6 CPU7 NUMA node 0 NUMA node 1 OmniXtend fabric
  • 9. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 9 Current Status • Up to 4 OmniXtend nodes with NUMA enabled • Dynamic topology reconfiguration with DIP switch • Point-to-Point via SN2010 switch • Based on latest upstream Linux kernel (v5.9) Working NUMA setup
  • 10. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 10 Can we do better ? • NUMA model doesn’t scale beyond few nodes – Expensive – Power management issues – Reliability • Wide adoption of distributed application model over clustered network servers – Scales well but higher latency – New programming model in absence of cache coherency • We need a cache coherent system that can scale!! To achieve better scale out models
  • 11. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 11 Independent Nodes (IN) Model • Each node boots independent instance of Linux • Access shared memory exposed by a kernel driver Experimental approach that can scale Node 0 CPU0 CPU1 memory0 CPU2 CPU3 Node 1 CPU0 CPU1 memory1 CPU2 CPU3 OmniXtend fabric shared memory
  • 12. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 12 OmniXtend Unified Boot Protocol • Allows to run either SO or IN model on the same hardware after reset • OnDemand reconfiguration based on the application requirement • Many possibilities of experimentation and exploration of research problems • Performance benchmarking with ease • No designated leader required Single boot process for all system models
  • 13. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 13 Management Server • Management server can be implemented in – A separate host machine that is connected to the OmniXtend network – One of the Compute Node • Does not require to participate in all OmniXtend traffic but should be capable of communicating with all nodes via raw ethernet frames • Assigns individual node’s role and memory ranges – Every node boots up independently and wait for role to be assigned by the management server – Implementation is flexible and can be used to create different topologies Manages dynamic discovery and initialization Management server Node 0 (compute & memory) Node 2 (compute & memory) Node 3 (compute & memory) Node N (memory only) Node N (hardware accelerator)
  • 14. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 14 Boot process (SO system) • BSP (bootstrap processor == first HART to run) wakes up other HARTs once low-level memory initialization is completed in OpenSBI Similar to a regular NUMA system boot flow Linux (S-mode) (OS) ZSBL (M-mode) (ROM) FSBL (M-mode) (LOADER) OpenSBI (M-mode) ZSBL (M-mode) (ROM) FSBL (M-mode) (LOADER) OpenSBI (M-mode) ZSBL (M-mode) (ROM) FSBL (M-mode) (LOADER) OpenSBI (M-mode) All remote memory is accessible by kernel and user space Board1 Board2 Board3
  • 15. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 15 Boot process (IN system) A distributed system with shared memory Linux (S-mode) (OS) ZSBL (M-mode) (ROM) FSBL (M-mode) (LOADER) OpenSBI (M-mode) ZSBL (M-mode) (ROM) FSBL (M-mode) (LOADER) OpenSBI (M-mode) ZSBL (M-mode) (ROM) FSBL (M-mode) (LOADER) OpenSBI (M-mode) Board1 Board2 Board3 Linux (S-mode) (OS) Linux (S-mode) (OS) On-demand shared memory • Shared memory implemented on demand (e.g. mmap()) with remapping of dynamically allocated node-local and remote DRAM pages • Can scale up to hundreds of nodes
  • 16. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 16 Can we develop faster ? A software centric approach Simulation/Emulation can fast track research Easy to experiment and explore as successive quick iteration in software Reduce CapEx for Research and Development Early verify the correctness and scalability of the protocol Easy to extend other ISAs (x86, ARM64) in future
  • 17. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 17 What is protocol simulation? • Simulate the interaction of OmniXtend endpoints • Implemented as a C library • Include unit tests for verification of the protocol • Quickly adapt to the protocol update Protocol Simulation
  • 18. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 18 Extendable plugins in the library • Plugins are independent from each other • Developers can provide their own implementation • Current implementation – OX_CACHE: Configurable LRU cache – OX_PROTOCOL: TileLink over Ethernet (TLoE) – OX_TRANSPORT: Raw Socket Allowing configurability and extendibility TRANSPORT CACHE PROTOCOL (rawsock, IP/TCP, RDMA) Snoopy, Directory-based Direct Mapped, LRU, FIFO DEVICE
  • 19. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 19 Protocol simulation overview User input Protocol Simulation library OXSE_DEV QTest System emulation Test Application OXPS_DEV GoogleTest OXPS_APP OX_PROTOCOL OX_TRANSPORT OX_CACHE Address checker memory ops callback
  • 20. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 20 What is system emulation? • An OmniXtend system consists of distributed compute, memory, storage and optional accelerator nodes • Nodes are connected to a traditional switch or programmable switch with enhanced features Implement the entire OmniXtend protocol in Qemu Switch OmniXtend Node 0 phy kernel Qemu emulation OmniXtend Node 1 phy kernel Qemu emulation NVM NVM NVM phy FPGA/ASIC OmniXtend Node 2 phy ML Accelerator phy
  • 21. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 21 System Emulation Overview A Qemu device emulates OmniXtend device Kernel User space Qemu process phy virtio net- device Guest Kernel Guest User space Hardware Virtio disk OmniXtend device Read/Write to non-cacheable remote memory Access ethernet packets using a raw socket Implement the OmniXtend protocol Address checker memory ops callback Memory backend LRU Cache Read/Write to cacheable memory Local Remote Remote cacheable memory request mmio device
  • 22. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 22 They co-exist & complement each other • Complete system level emulation • Verify the initialization protocol • Bring up large number of nodes Pros • Low performance • Can’t emulate hardware accelerator Cons • Simple to setup • Test suites to ensure correctness • Doesn’t require special hardware Pros • Cannot boot Operating Systems • Low performance Cons System emulation Protocol Simulation Simulation & Emulation Summary
  • 23. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 23 Future work • Implement kernel support for Independent Node model – User space applications can mmap & use the shared memory – Kernel may choose to protect its own page table memory from remote access via PMP – Kernel driver may handle the remote memory page faults – A user space library adds to support to existing distributed application framework • Experiment different topologies with a different rules in management server • Improve Simulation/Emulation using extensive test suites • Hook up with the Qemu instances with OmniXtend FPGAs • Exercise the scalability test with many nodes • Opensource the simulation library and Qemu To infinity and beyond!
  • 24. © 2020 Western Digital Corporation or its affiliates. All rights reserved. 11/18/20
  • 25. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 25 Backup
  • 26. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 26 Types of messages Type Sender Intended Receiver Mode of transmission Payload Intended action HELLO OE MS Broadcast None send and wait WELCOME MS OE Unicast None Continue MY_INFO OE MS Unicast Local OE DTB send and wait DTB_INFO MS OE Unicast Full OmniXtend system DTB parse OmniXtend dt node and configure channels CONFIG_DONE OE MS Unicast None send and wait ENABLE MS OE Unicast None enable TLoE in given order DT ENABLE_DONE OE MS Unicast None send and wait BOOT MS OE Unicast None Reconfigure memory and boot • OmniXtend Endpoint (OE) • Management server (MS)
  • 27. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 27 Booting SO system Before TLoE is enabled Configure TX/RX Channels Enable TLoE end points in given order OE(Non-Boot) MS OE (Boot) Boot hart Configure ethernet Send HELLO and wait for WELCOME HELLO Unicast WELCOME packet WELCOME MY_INFO DTB_INFO Send MY_INFO and wait for DTB_INFO Multicast full DTB with OmniXtend topology to Non-booting node Parses OmniXtend topology ENABLE Send CONFIG_DONE and wait for ENABLE Configure TX/RX Channels CONFIG_DONE Enable TLoE end points in given order Send ENABLE message after CONFIG_DONE is received from every node Configure ethernet Send HELLO and wait for WELCOME HELLO WELCOME MY_INFO DTB_INFO Send MY_INFO and wait for DTB_INFO CONFIG_DONE Parses OmniXtend topology Multicast full DTB with OmniXtend topology to booting node ENABLE Boot hart
  • 28. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 28 Booting SO system After TLoE is enabled Send ENABLE_DONE and wait for BOOT Configure MMIO devices OE(Non-Boot) MS OE (Boot) Boot hart Send ENABLE_DONE and wait for BOOT ENABLE_DONE BOOT Reconfigure MMIO address Send BOOT message after ENABLE_DONE is received from every node ENABLE_DONE BOOT Boot hart Reconfigure cacheable memory address Jump Non-boot hart Parse full DT Non-boot hart Warm boot Linux Jump to Linux Normal OpenSBI boot
  • 29. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 29 Booting IN system Before TLoE is enabled OE(Boot) MS OE (Boot) Boot hart Configure ethernet Send HELLO and wait for WELCOME HELLO Unicast WELCOME packet WELCOME MY_INFO DTB_INFO Send MY_INFO and wait for DTB_INFO Multicast full DTB with OmniXtend topology to Non-booting node Parses OmniXtend topology ENABLE Send CONFIG_DONE and wait for ENABLE Configure TX/RX Channels CONFIG_DONE Enable TLoE end points in given order Send ENABLE message after CONFIG_DONE is received from every node Configure ethernet Send HELLO and wait for WELCOME HELLO WELCOME MY_INFO DTB_INFO Send MY_INFO and wait for DTB_INFO CONFIG_DONE Parses OmniXtend topology Configure TX/RX Channels ENABLE Enable TLoE end points in given order Send CONFIG_DONE and wait for ENABLE
  • 30. 11/18/20 © 2020 Western Digital Corporation or its affiliates. All rights reserved. 30 Booting IN system After TLoE is enabled OE(Boot) MS OE (Boot) Boot hart Send ENABLE_DONE and wait for BOOT ENABLE_DONE BOOT Reconfigure MMIO address Send BOOT message after ENABLE_DONE is received from every node Send ENABLE_DONE and wait for BOOT ENABLE_DONE BOOT Boot hart Reconfigure cacheable memory address Non-boot hart Configure MMIO devices Non-boot hart Warm boot Linux Jump to Linux Normal OpenSBI boot Warm boot Normal OpenSBI boot Linux Jump to Linux On-demand shared memory using a Linux OmniXtend driver Reconfigure cacheable memory address