Apache flume介绍

•

5 gostaram•2,967 visualizações

haosdent huang

the introduction document of Apache flume

Tecnologia

Flume结点的⾓角⾊色
•Node节点，按⾓角⼜又分Agent和Collector
•Master节点

Node结点
• 指逻辑节点。⼀一个物理
节点（JVM实例）上可以
包含多个逻辑节点，并
且可以绑定或解绑逻辑
节点到某个特定的物理
节点。
• 每个逻辑节点会向master
报告执⾏行状态，配置，
发送⼼心跳。
JVM
Agent Node
Collector Node

Agent
•agent通过指定的
source来接受数据
流，并通过指定
的sink转发给
collector。
Agent
source
sink
Collector
source
sink

Collector
•collector聚合多个
agent的数据流，
并传输到指定的
输出
Agent
source
sink
Collector
source
sink

Collector与Agent的对应
•可⼿手动指定，也
可⾃自动匹配
•⾃自动匹配的情况
下，master会平衡
collector之间的负
载

Event
•event是Flume中数
据的最⼩小表⽰示单
元。
Agent
metadata
body
Unix timestamp
Nanosecond timestamp
Priority
Source host

流之间的隔离措施
•为将所有的source加标签，使⽤用⽤用相同的管
道同时进⾏行传输，最终经过分离后传送到不
同的⺫⽬目的地。
•另⼀一种为阻塞型，任意时刻只能有⼀一个数据
流在传输。

Source种类
•Tiered Event Source
• 由master或配置⽂文件提供参数，⽤用于node之间传输
•Logical Source
• 信息不完全⽽而且抽象的source，由master⾃自动管理，当
master获取到⾜足够的信息时会将其转换为rpcSource
•Basic Source

Source机制
•Push数据到Flume
•令Flume不断Polling数据
•Embedding，即在应⽤用程序中嵌⼊入Flume组件

Sink的种类
•Tiered Event Sink
• 由master或配置⽂文件提供参数，⽤用于node之间传输
•Logical Sink
• 信息不完全⽽而且抽象的sink，由master⾃自动管理，当master
获取到⾜足够的信息时会将其转换为rpcSink
•Basic Sink

Sink的种类
•Special Sink
• Fan out
• [sinkA, sinkB] 表⽰示同时往sinkA，sinkB写数据
• Fail over
• [sinkA ? sinkB] 表⽰示写sinkA失败时尝试写sinkB
• Roll
• roll(A) text("ﬁle-%{rolltag}")
• 表⽰示每隔A秒关闭当前sink实例，再开另外⼀一个sink实例继续往另⼀一个⽂文件写内容

Sink的输出格式
•有avro，json，log4j等
•可先设置压缩，再写⼊入HDFS

Sink的装饰器
•可通过sink decorators对sink中的event进⾏行控
制。
•如增⼤大，过滤或压缩其中的event。
•可⾃自定义装饰器。

Master结点
•管理所有结点的配置
•跟踪数据流的最后确认信息，并通知agent

多个Master
•在多个master的情况下，当有⼀一个master失败
后，其他master会接管它的⼯工作并保证所有
正在传输的数据流正常运⾏行。
•借助zookeeper管理

Master⽂文件存储
•临时存于内存
• 只能单Master
• 重启Master及Master错误退出均会配置⽂文件丢失
•基于zookeeper持久化地存储

Agent的可靠性
•两种可靠的Silk
•agentE2ESink
• 使⽤用预先⽇日志和确认信息来保证数据正确传输
•agentDFOSink
• 检测到collector失败的时候，先将数据写到磁盘
• 当collector恢复或有其他可⽤用collector时，继续传输

Agent的可靠性
•故障转移，由
master控制
•autoE2EChain
•autoDFOChain
•autoBEChain

HDFS冗余
•collector在⼀一定时间间隔后（默认30s）会关
闭HDFS⽂文件，并创建⼀一个新的⽂文件继续写数
据
•当HDFS⽂文件为关闭的情况下，HDFS的冗余
策略才能正常实施

安全控制
•可借助kerberized HDFS服务，实现HDFS的读
写安全控制

Mais conteúdo relacionado

Destaque

https://www.hubspot.com/state-of-marketing · Scaling relationships and proving ROI · Social media is the place for search, sales, and service · Authentic influencer partnerships fuel brand growth · The strongest connections happen via call, click, chat, and camera. · Time saved with AI leads to more creative work · Seeking: A single source of truth · TLDR; Get on social, try AI, and align your systems. · More human marketing, powered by robots

2024 State of Marketing Report – by Hubspot

Marius Sescu

ChatGPT is a revolutionary addition to the world since its introduction in 2022. A big shift in the sector of information gathering and processing happened because of this chatbot. What is the story of ChatGPT? How is the bot responding to prompts and generating contents? Swipe through these slides prepared by Expeed Software, a web development company regarding the development and technical intricacies of ChatGPT!

Everything You Need To Know About ChatGPT

Expeed Software

Product Design Trends in 2024 | Teenage Engineerings

Pixeldarts

Mental health has been in the news quite a bit lately. Dozens of U.S. states are currently suing Meta for contributing to the youth mental health crisis by inserting addictive features into their products, while the U.S. Surgeon General is touring the nation to bring awareness to the growing epidemic of loneliness and isolation. The country has endured periods of low national morale, such as in the 1970s when high inflation and the energy crisis worsened public sentiment following the Vietnam War. The current mood, however, feels different. Gallup recently reported that national mental health is at an all-time low, with few bright spots to lift spirits. To better understand how Americans are feeling and their attitudes towards mental health in general, ThinkNow conducted a nationally representative quantitative survey of 1,500 respondents and found some interesting differences among ethnic, age and gender groups. Technology For example, 52% agree that technology and social media have a negative impact on mental health, but when broken out by race, 61% of Whites felt technology had a negative effect, and only 48% of Hispanics thought it did. While technology has helped us keep in touch with friends and family in faraway places, it appears to have degraded our ability to connect in person. Staying connected online is a double-edged sword since the same news feed that brings us pictures of the grandkids and fluffy kittens also feeds us news about the wars in Israel and Ukraine, the dysfunction in Washington, the latest mass shooting and the climate crisis. Hispanics may have a built-in defense against the isolation technology breeds, owing to their large, multigenerational households, strong social support systems, and tendency to use social media to stay connected with relatives abroad. Age and Gender When asked how individuals rate their mental health, men rate it higher than women by 11 percentage points, and Baby Boomers rank it highest at 83%, saying it’s good or excellent vs. 57% of Gen Z saying the same. Gen Z spends the most amount of time on social media, so the notion that social media negatively affects mental health appears to be correlated. Unfortunately, Gen Z is also the generation that’s least comfortable discussing mental health concerns with healthcare professionals. Only 40% of them state they’re comfortable discussing their issues with a professional compared to 60% of Millennials and 65% of Boomers. Race Affects Attitudes As seen in previous research conducted by ThinkNow, Asian Americans lag other groups when it comes to awareness of mental health issues. Twenty-four percent of Asian Americans believe that having a mental health issue is a sign of weakness compared to the 16% average for all groups. Asians are also considerably less likely to be aware of mental health services in their communities (42% vs. 55%) and most likely to seek out information on social media (51% vs. 35%).

How Race, Age and Gender Shape Attitudes Towards Mental Health

ThinkNow

AI Trends in Creative Operations 2024 by Artwork Flow.pdf

marketingartwork

Skeleton Culture Code

Skeleton Technologies

PEPSICO Presentation to CAGNY Conference Feb 2024

Neil Kimberley

Content Methodology: A Best Practices Report (Webinar)

contently

How to Prepare For a Successful Job Search for 2024

Albert Qian

A report by thenetworkone and Kurio. The contributing experts and agencies are (in an alphabetical order): Sylwia Rytel, Social Media Supervisor, 180heartbeats + JUNG v MATT (PL), Sharlene Jenner, Vice President - Director of Engagement Strategy, Abelson Taylor (USA), Alex Casanovas, Digital Director, Atrevia (ES), Dora Beilin, Senior Social Strategist, Barrett Hoffher (USA), Min Seo, Campaign Director, Brand New Agency (KR), Deshé M. Gully, Associate Strategist, Day One Agency (USA), Francesca Trevisan, Strategist, Different (IT), Trevor Crossman, CX and Digital Transformation Director; Olivia Hussey, Strategic Planner; Simi Srinarula, Social Media Manager, The Hallway (AUS), James Hebbert, Managing Director, Hylink (CN / UK), Mundy Álvarez, Planning Director; Pedro Rojas, Social Media Manager; Pancho González, CCO, Inbrax (CH), Oana Oprea, Head of Digital Planning, Jam Session Agency (RO), Amy Bottrill, Social Account Director, Launch (UK), Gaby Arriaga, Founder, Leonardo1452 (MX), Shantesh S Row, Creative Director, Liwa (UAE), Rajesh Mehta, Chief Strategy Officer; Dhruv Gaur, Digital Planning Lead; Leonie Mergulhao, Account Supervisor - Social Media & PR, Medulla (IN), Aurelija Plioplytė, Head of Digital & Social, Not Perfect (LI), Daiana Khaidargaliyeva, Account Manager, Osaka Labs (UK / USA), Stefanie Söhnchen, Vice President Digital, PIABO Communications (DE), Elisabeth Winiartati, Managing Consultant, Head of Global Integrated Communications; Lydia Aprina, Account Manager, Integrated Marketing and Communications; Nita Prabowo, Account Manager, Integrated Marketing and Communications; Okhi, Web Developer, PNTR Group (ID), Kei Obusan, Insights Director; Daffi Ranandi, Insights Manager, Radarr (SG), Gautam Reghunath, Co-founder & CEO, Talented (IN), Donagh Humphreys, Head of Social and Digital Innovation, THINKHOUSE (IRE), Sarah Yim, Strategy Director, Zulu Alpha Kilo (CA).

Social Media Marketing Trends 2024 // The Global Indie Insights

Kurio // The Social Media Age(ncy)

The search marketing landscape is evolving rapidly with new technologies, and professionals, like you, rely on innovative paid search strategies to meet changing demands. It’s important that you’re ready to implement new strategies in 2024. Check this out and learn the top trends in paid search advertising that are expected to gain traction, so you can drive higher ROI more efficiently in 2024. You’ll learn: - The latest trends in AI and automation, and what this means for an evolving paid search ecosystem. - New developments in privacy and data regulation. - Emerging ad formats that are expected to make an impact next year. Watch Sreekant Lanka from iQuanti and Irina Klein from OneMain Financial as they dive into the future of paid search and explore the trends, strategies, and technologies that will shape the search marketing landscape. If you’re looking to assess your paid search strategy and design an industry-aligned plan for 2024, then this webinar is for you.

Trends In Paid Search: Navigating The Digital Landscape In 2024

Search Engine Journal

From their humble beginnings in 1984, TED has grown into the world’s most powerful amplifier for speakers and thought-leaders to share their ideas. They have over 2,400 filmed talks (not including the 30,000+ TEDx videos) freely available online, and have hosted over 17,500 events around the world. With over one billion views in a year, it’s no wonder that so many speakers are looking to TED for ideas on how to share their message more effectively. The article “5 Public-Speaking Tips TED Gives Its Speakers”, by Carmine Gallo for Forbes, gives speakers five practical ways to connect with their audience, and effectively share their ideas on stage. Whether you are gearing up to get on a TED stage yourself, or just want to master the skills that so many of their speakers possess, these tips and quotes from Chris Anderson, the TED Talks Curator, will encourage you to make the most impactful impression on your audience. See the full article and more summaries like this on SpeakerHub here: https://speakerhub.com/blog/5-presentation-tips-ted-gives-its-speakers See the original article on Forbes here: http://www.forbes.com/forbes/welcome/?toURL=http://www.forbes.com/sites/carminegallo/2016/05/06/5-public-speaking-tips-ted-gives-its-speakers/&refURL=&referrer=#5c07a8221d9b

5 Public speaking tips from TED - Visualized summary

SpeakerHub

Everyone is in agreement that ChatGPT (and other generative AI tools) will shape the future of work. Yet there is little consensus on exactly how, when, and to what extent this technology will change our world. Businesses that extract maximum value from ChatGPT will use it as a collaborative tool for everything from brainstorming to technical maintenance. For individuals, now is the time to pinpoint the skills the future professional will need to thrive in the AI age. Check out this presentation to understand what ChatGPT is, how it will shape the future of work, and how you can prepare to take advantage.

ChatGPT and the Future of Work - Clark Boyd

Clark Boyd

Getting into the tech field. what next

Tessa Mero

Google's Just Not That Into You: Understanding Core Updates & Search Intent

Lily Ray

How to have difficult conversations

Rajiv Jayarajah, MAppComm, ACC

Introduction to Data Science

Christy Abraham Joy

Time Management & Productivity - Best Practices

Vit Horky

The six step guide to practical project management If you think managing projects is too difficult, think again. We’ve stripped back project management processes to the basics – to make it quicker and easier, without sacrificing the vital ingredients for success. “If you’re looking for some real-world guidance, then The Six Step Guide to Practical Project Management will help.” Dr Andrew Makar, Tactical Project Management

The six step guide to practical project management

MindGenius

Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...

RachelPearson36

Destaque (20)

2024 State of Marketing Report – by Hubspot

Everything You Need To Know About ChatGPT

Product Design Trends in 2024 | Teenage Engineerings

How Race, Age and Gender Shape Attitudes Towards Mental Health

AI Trends in Creative Operations 2024 by Artwork Flow.pdf

Skeleton Culture Code

PEPSICO Presentation to CAGNY Conference Feb 2024

Content Methodology: A Best Practices Report (Webinar)

How to Prepare For a Successful Job Search for 2024

Social Media Marketing Trends 2024 // The Global Indie Insights

Trends In Paid Search: Navigating The Digital Landscape In 2024

5 Public speaking tips from TED - Visualized summary

ChatGPT and the Future of Work - Clark Boyd

Getting into the tech field. what next

Google's Just Not That Into You: Understanding Core Updates & Search Intent

How to have difficult conversations

Introduction to Data Science

Time Management & Productivity - Best Practices

The six step guide to practical project management

Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...

Apache flume介绍

1. Apache Flume 2013.5.29 ⻩黄浩松

2. Flume的架构

3. Flume结点的⾓角⾊色 •Node节点，按⾓角⼜又分Agent和Collector •Master节点

4. Node结点 • 指逻辑节点。⼀一个物理节点（JVM实例）上可以包含多个逻辑节点，并且可以绑定或解绑逻辑节点到某个特定的物理节点。 • 每个逻辑节点会向master 报告执⾏行状态，配置，发送⼼心跳。 JVM Agent Node Collector Node

5. Agent •agent通过指定的 source来接受数据流，并通过指定的sink转发给 collector。 Agent source sink Collector source sink

6. Collector •collector聚合多个 agent的数据流，并传输到指定的输出 Agent source sink Collector source sink

7. Collector与Agent的对应 •可⼿手动指定，也可⾃自动匹配 •⾃自动匹配的情况下，master会平衡 collector之间的负载

8. Event •event是Flume中数据的最⼩小表⽰示单元。 Agent metadata body Unix timestamp Nanosecond timestamp Priority Source host

9. Event

10. 流之间的隔离措施 •为将所有的source加标签，使⽤用⽤用相同的管道同时进⾏行传输，最终经过分离后传送到不同的⺫⽬目的地。 •另⼀一种为阻塞型，任意时刻只能有⼀一个数据流在传输。

11. Source种类 •Tiered Event Source • 由master或配置⽂文件提供参数，⽤用于node之间传输 •Logical Source • 信息不完全⽽而且抽象的source，由master⾃自动管理，当 master获取到⾜足够的信息时会将其转换为rpcSource •Basic Source

12. Source机制 •Push数据到Flume •令Flume不断Polling数据 •Embedding，即在应⽤用程序中嵌⼊入Flume组件

13. Sink的种类 •Tiered Event Sink • 由master或配置⽂文件提供参数，⽤用于node之间传输 •Logical Sink • 信息不完全⽽而且抽象的sink，由master⾃自动管理，当master 获取到⾜足够的信息时会将其转换为rpcSink •Basic Sink

14. Sink的种类 •Special Sink • Fan out • [sinkA, sinkB] 表⽰示同时往sinkA，sinkB写数据 • Fail over • [sinkA ? sinkB] 表⽰示写sinkA失败时尝试写sinkB • Roll • roll(A) text("ﬁle-%{rolltag}") • 表⽰示每隔A秒关闭当前sink实例，再开另外⼀一个sink实例继续往另⼀一个⽂文件写内容

15. Sink的输出格式 •有avro，json，log4j等 •可先设置压缩，再写⼊入HDFS

16. Sink的装饰器 •可通过sink decorators对sink中的event进⾏行控制。 •如增⼤大，过滤或压缩其中的event。 •可⾃自定义装饰器。

17. Master结点 •管理所有结点的配置 •跟踪数据流的最后确认信息，并通知agent

18. 多个Master •在多个master的情况下，当有⼀一个master失败后，其他master会接管它的⼯工作并保证所有正在传输的数据流正常运⾏行。 •借助zookeeper管理

19. Master⽂文件存储 •临时存于内存 • 只能单Master • 重启Master及Master错误退出均会配置⽂文件丢失 •基于zookeeper持久化地存储

20. Flume的可靠性

21. Agent的可靠性 •两种可靠的Silk •agentE2ESink • 使⽤用预先⽇日志和确认信息来保证数据正确传输 •agentDFOSink • 检测到collector失败的时候，先将数据写到磁盘 • 当collector恢复或有其他可⽤用collector时，继续传输

22. Agent的可靠性 •故障转移，由 master控制 •autoE2EChain •autoDFOChain •autoBEChain

23. HDFS冗余 •collector在⼀一定时间间隔后（默认30s）会关闭HDFS⽂文件，并创建⼀一个新的⽂文件继续写数据 •当HDFS⽂文件为关闭的情况下，HDFS的冗余策略才能正常实施

24. Flume的可管理性

25. ⺴⽹网⻚页

26. 控制台