Go Lang migrating billions of documents

•

3 likes•597 views

My own story with go. On these story I share how to migrate a lot of data between two MongoDb servers using GO. Some learns and details that makes all the difference.

Technology

GoLang:learning
bydoing
Case: Migrating nearly 2 billion documents
@jonatasdp

thechallenge:
migrate~1.8billion
fromamongoserver
toanother

Migrationalgorithm
● Twomongoconnections:<origin>and<destiny>
● <origin>receiverecords
● <destiny>insertrecords
● Whenrecordison<destiny>
● Removefrom<origin>

FIRSTTASTE
● STARTWITHOUTTESTS:(
● LEARNINGABOUTREFERENCESVSVARS
● Structsforeverything
● InterfaceComprehension
● GORoutines
● LocksandMutex
● Channels

WhatIlearned
● simplicity
● naturallyfast
● easytowriteasynccode
● easytomaintain(mongo-migration#1)

OPENSOURCE
https://github.com/ResultadosDigitais/mongo-migration

Thankyou!
http://shipit.resultadosdigitais.com.br
@jonatasdp
http://ideia.me

Viewers also liked

Quando descobri que era programador

Jônatas Paganini

XP - eXtreme Programming - 2010

Jônatas Paganini

ZendFramework2 & Symfony2

Wesley Victhor Mendes

4 ferramentas para acompanhar seu app em produção pelo slack

Jônatas Paganini

Concurrency in Ruby

Jônatas Paganini

"RD Talk" sobre tecnologia para o pessoal das outras áreas da Resultados Digitais. A ocasião foi pra desmistifica de maneira divertida alguns termos utilizados no ambiente de desenvolvimento de software e que o público em geral desconhece. O conteúdo da palestra foi desenvolvido a partir das questões trazidas pelos colaboradores da Resultados Digitais então tem várias partes que são específicas ou relacionadas ao contexto da RD e não só a tecnologia.

Tech for non techs

Jônatas Paganini

Testes em go

Eduardo Felipe Ewert Bonet

Testing in go

Eduardo Felipe Ewert Bonet

Life hacks for productivity

Jônatas Paganini

7Masters jQuery - Eventos em jQuery, com Felquis Gimenes

iMasters

Docker na vida real

Fernando Ike

Docker: uma visão geral e exemplos de uso

Tobias Sette

Viewers also liked (12)

Quando descobri que era programador

XP - eXtreme Programming - 2010

ZendFramework2 & Symfony2

4 ferramentas para acompanhar seu app em produção pelo slack

Concurrency in Ruby

Tech for non techs

Testes em go

Testing in go

Life hacks for productivity

7Masters jQuery - Eventos em jQuery, com Felquis Gimenes

Docker na vida real

Docker: uma visão geral e exemplos de uso

Similar to Go Lang migrating billions of documents

Reactive applications are becoming a de-facto industry standard and, if employed correctly, toolkits like Lightbend Reactive Platform make the implementation easier than ever. But design of these systems might be challenging as it requires particular mindset shift to tackle problems we might not be used to. In this talk we’re going to discuss the most common things I’ve seen in the field that prevented applications to work as expected. I’d like to talk about typical pitfalls that might cause troubles, about trade-offs that might not be fully understood or important choices that might be overlooked including persistent actors pitfalls, tackling of network partitions, proper implementations of graceful shutdown or distributed transactions, trade-offs of micro-services or actors and more. This talk should be interesting for anyone who is thinking about, implementing, or have already deployed reactive application. My goal is to provide a comprehensive explanation of common problems to be sure they won’t be repeated by fellow developers. The talk is a little bit more focused on Lightbend platform but understanding of the concepts we are going to talk about should be beneficial for everyone interested in this field.

Reactive mistakes - ScalaDays Chicago 2017

Petr Zapletal

Creando microservicios con Java MicroProfile y TomEE - OGBT

César Hernández

Barcamp presentation

Vachagan Balayan

Creando microservicios con Java y Microprofile - Nicaragua JUG

César Hernández

More instances of SQLite are used every day, by more people, than all other database engines combined. An yet, SQLite does not get much attention. Many developers hardly know anything about it. This session will review the features of SQLite, how it is different from other database engines, its strengths and its weaknesses, and when SQLite is an appropriate technology and when some other database engine might be a better choice.

[db tech showcase Tokyo 2017] A11: SQLite - The most used yet least appreciat...

Insight Technology, Inc.

Creando microservicios con java micro profile y tomee - CUNORI 2020

César Hernández

Log Management Systems

Mehdi Hamidi

Identifying Hotspots in Software Build Processes

Shane McIntosh

ekbpy'2012 - Антон Патрушев - ZeroMQ

it-people

Learn TypeScript from scratch

Mohd Manzoor Ahmed

Paper_An Efficient Garbage Collection in Java Virtual Machine via Swap I/O O...

Hyo jeong Lee

This is the speech Max Liu gave at Percona Live Open Source Database Conference 2016. Max Liu: Co-founder and CEO, a hacker with a free soul The slide covered the following topics: - Why another database? - What kind of database we want to build? - How to design such a database, including the principles, the architecture, and design decisions? - How to develop such a database, including the architecture and the core technologies for TiKV and TiDB? - How to test the database to ensure the quality and stability?

How to build TiDB

PingCAP

Creando microservicios con Java, Microprofile y TomEE - Baranquilla JUG

César Hernández

Serverless is great for web applications and APIs, but this does not mean it cannot be used successfully for other use cases. In this talk, we will discuss a successful application of serverless in the field of High Performance Computing. Specifically we will discuss how Lambda, Fargate, Kinesis and other serverless technologies are being used to run sophisticated financial models at one of the major reinsurance companies in the World. We we learn about the architecture, the tradeoffs, some challenges and some unresolved pain points. Most importantly, we'll find out if serverless can be a great fit for HPC and if we can finally stop managing those boring EC2 instances!

Serverless for High Performance Computing

Luciano Mammino

Lrz kurs: big data analysis

Ferdinand Jamitzky

Working with software means working with bugs. Bugs in software, bugs in hardware; bugs in Open Source code, bugs in proprietary code. If software is eating the world, bugs might end up taking the first bite. We will present a few typical bugs, some of them famous, some of them infamous (including bugs that actually killed people). Since one can never be too well-prepared to fend off the next infestation, we will give tools, tips, and best practices to fix bugs in Open Source software. We will give real world examples of Really Mysterious Bugs (sometimes nicknamed "Heisenbugs" because they tend to disappear when you try to observe them), and how they were fixed, in Node.js, Docker, and the Linux Kernel.

Killer Bugs From Outer Space

Jérôme Petazzoni

Serverless for High Performance Computing

Luciano Mammino

"Is your team looking to bring the power of full, end-to-end stream processing with Apache Flink to your organization but are concerned about the time, resources or skills required? In this talk, Sharon Xie, Decodable Founding Engineer and Apache Flink PMC Member, Robert Metzger, will reveal the biggest lessons learned, and how to avoid common mistakes when adopting Apache Flink. If you have any plans on implementing Apache Flink, then this is a session you do not want to miss. We will talk about avoiding data-loss with Flink’s Kafka exactly-once producer, configuring Flink for getting the most bang for the buck out of your memory configuration and tuning for efficient checkpointing."

3 Flink Mistakes We Made So You Won't Have To

HostedbyConfluent

萬事萬物皆是 LOG - 系統架構也來點科普

Poga Po

Kubernetes Native Java and Eclipse MicroProfile | EclipseCon Europe 2019

Jakarta_EE

Similar to Go Lang migrating billions of documents (20)

Reactive mistakes - ScalaDays Chicago 2017

Creando microservicios con Java MicroProfile y TomEE - OGBT

Barcamp presentation

Creando microservicios con Java y Microprofile - Nicaragua JUG

[db tech showcase Tokyo 2017] A11: SQLite - The most used yet least appreciat...

Creando microservicios con java micro profile y tomee - CUNORI 2020

Log Management Systems

Identifying Hotspots in Software Build Processes

ekbpy'2012 - Антон Патрушев - ZeroMQ

Learn TypeScript from scratch

Paper_An Efficient Garbage Collection in Java Virtual Machine via Swap I/O O...

How to build TiDB

Creando microservicios con Java, Microprofile y TomEE - Baranquilla JUG

Serverless for High Performance Computing

Lrz kurs: big data analysis

Killer Bugs From Outer Space

Serverless for High Performance Computing

3 Flink Mistakes We Made So You Won't Have To

萬事萬物皆是 LOG - 系統架構也來點科普

Kubernetes Native Java and Eclipse MicroProfile | EclipseCon Europe 2019

More from Jônatas Paganini

As a Rails Application grows, there is a need to decouple heavy systems from the monolithic applications. Several teams in different companies are doing the same: extracting (micro) services from their monolithic applications to give the engineering teams more flexibility to speed up the workflow. From the separation of the business logic to the server's setup, every change should respect the zero-downtime approach. This talk shares the automated steps and exercises we created to have a smooth transition to the new system. I'll share the context of the tool that is automatically extracting an entire rails engine from a project and moving it to a separate service.

Extracting a Rails Engine to a separated application

Jônatas Paganini

Onboarding developers on a ful remote environment

Jônatas Paganini

TDC Floripa 2018 Dev Tests - weak specs

Jônatas Paganini

Floripa on Rails - dev workflow

Jônatas Paganini

ø Downtime migrations - Ruby Conf 2015

Jônatas Paganini

Tdc cloud computing - RDStation experiences

Jônatas Paganini

Life hacks for productivity

Jônatas Paganini

Dicas para uma boa impressão 3D

Jônatas Paganini

ø Downtime migrations

Jônatas Paganini

More from Jônatas Paganini (9)

Extracting a Rails Engine to a separated application

Onboarding developers on a ful remote environment

TDC Floripa 2018 Dev Tests - weak specs

Floripa on Rails - dev workflow

ø Downtime migrations - Ruby Conf 2015

Tdc cloud computing - RDStation experiences

Life hacks for productivity

Dicas para uma boa impressão 3D

ø Downtime migrations

Recently uploaded

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

Explore 'The Codex of Business: Writing Software for Real-World Solutions,' a compelling SlideShare presentation that delves into digital transformation in healthcare. Discover through a detailed case study how Agile methodologies empower healthcare providers to develop, iterate, and refine digital solutions that address real-world challenges. Learn how strategic planning, user feedback, and continuous improvement drive success in deploying technologies that enhance patient care and operational efficiency. Ideal for healthcare professionals, IT specialists, and digital transformation advocates seeking actionable insights and practical examples of technology making a real difference.

The Codex of Business Writing Software for Real-World Solutions 2.pptx

Malak Abu Hammad

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

BooK Now Call us at +918448380779 to hire a gorgeous and seductive call girl for sex. Take a Delhi Escort Service. The help of our escort agency is mostly meant for men who want sexual Indian Escorts In Delhi NCR. It should be noted that any impersonator will get 100 attention from our Young Girls Escorts in Delhi. They will assume the position of reliable allies. VIP Call Girl With Original Photos Book Tonight +918448380779 Our Cheap Price 1 Hour not available 2 Hours 5000 Full Night 8000 TAG: Call Girls in Delhi, Noida, Gurgaon, Ghaziabad, Connaught Place, Greater Kailash Delhi, Lajpat Nagar Delhi, Mayur Vihar Delhi, Chanakyapuri Delhi, New Friends Colony Delhi, Majnu Ka Tilla, Karol Bagh, Malviya Nagar, Saket, Khan Market, Noida Sector 18, Noida Sector 76, Noida Sector 51, Gurgaon Mg Road, Iffco Chowk Gurgaon, Rajiv Chowk Gurgaon All Delhi Ncr Free Home Deliver

08448380779 Call Girls In Friends Colony Women Seeking Men

Delhi Call girls

🐬 The future of MySQL is Postgres 🐘

RTylerCroy

Microsoft's Threat Matrix for Kubernetes helps organizations understand the attack surface a Kubernetes deployment introduces to their environments. This ensures that adequate detections and mitigations are in place. By covering over 40 different attacker techniques, defenders can learn about Kubernetes-specific mitigations and controls to deploy to their environments. In this session, we will explore the MS-TA9013 Host Path Mount technique, which is commonly used by attackers to perform privilege escalation in a Kubernetes cluster. Attendees will learn how attackers and defenders can: * Escape the container's host volume mount to gain persistence on an underlying node * Move laterally from the underlying node into the customer's cloud environment * Analyze Kubernetes audit logs to detect pods deployed with a hostPath mount * Deploy an admission controller that prevents new pods from using a hostPath mount

Breaking the Kubernetes Kill Chain: Host Path Mount

Puma Security, LLC

Presentation on how to chat with PDF using ChatGPT code interpreter

naman860154

[2024]Digital Global Overview Report 2024 Meltwater.pdf

hans926745

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

08448380779 Call Girls In Civil Lines Women Seeking Men

Delhi Call girls

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Safe Software

A Call to Action for Generative AI in 2024

Results

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

The Raspberry Pi 5 was announced on October 2023. This new version of the popular embedded device comes with a new iteration of Broadcom’s VideoCore GPU platform, and was released with a fully open source driver stack, developed by Igalia. The presentation will discuss some of the major changes required to support this new Video Core iteration, the challenges we faced in the process and the solutions we provided in order to deliver conformant OpenGL ES and Vulkan drivers. The talk will also cover the next steps for the open source Raspberry Pi 5 graphics stack. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://eoss24.sched.com/event/1aBEx

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...

Igalia

Handwritten Text Recognition for manuscripts and early printed texts

Maria Levchenko

In an era where artificial intelligence (AI) stands at the forefront of business innovation, Information Architecture (IA) is at the core of functionality. See “There’s No AI Without IA” – (from 2016 but even more relevant today) Understanding and leveraging how Information Architecture (IA) supports AI synergies between knowledge engineering and prompt engineering is critical for senior leaders looking to successfully deploy AI for internal and externally facing knowledge processes. This webinar be a high-level overview of the methodologies that can elevate AI-driven knowledge processes supporting both employees and customers. Core Insights Include: Strategic Knowledge Engineering: Delve into how structuring AI's knowledge base is required to prevent hallucinations, enable contextual retrieval of accurate information. This will include discussion of gold standard libraries of use cases support testing various LLMs and structures and configurations of knowledge base. Precision in Prompt Engineering: Learn the art of crafting prompts that direct AI to deliver targeted, relevant responses, thereby optimizing customer experiences and business outcomes. Unified Approach for Enhanced AI Performance: Explore the intersection of knowledge and prompt engineering to develop AI systems that are not only more responsive but also aligned with overarching business strategies. Guiding Principles for Implementation: Equip yourself with best practices, ethical guidelines, and strategic considerations for embedding these technologies into your business ecosystem effectively. This webinar is designed to empower business and technology leaders with the knowledge to harness the full potential of AI, ensuring their organizations not only keep pace with digital transformation but lead the charge. Join us to map a roadmap to fully leverage Information Architecture (IA) and AI chart a course towards a future where AI is a key pillar of strategic innovation and business success.

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

Earley Information Science

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Scaling API-first – The story of a global engineering organization

Radu Cotescu

Recently uploaded (20)

A Domino Admins Adventures (Engage 2024)

The Codex of Business Writing Software for Real-World Solutions 2.pptx

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

08448380779 Call Girls In Friends Colony Women Seeking Men

🐬 The future of MySQL is Postgres 🐘

Breaking the Kubernetes Kill Chain: Host Path Mount

Presentation on how to chat with PDF using ChatGPT code interpreter

[2024]Digital Global Overview Report 2024 Meltwater.pdf

Axa Assurance Maroc - Insurer Innovation Award 2024

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

08448380779 Call Girls In Civil Lines Women Seeking Men

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

A Call to Action for Generative AI in 2024

2024: Domino Containers - The Next Step. News from the Domino Container commu...

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...

Handwritten Text Recognition for manuscripts and early printed texts

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

Boost Fertility New Invention Ups Success Rates.pdf

Scaling API-first – The story of a global engineering organization