Mais conteúdo relacionado Semelhante a Thrift+scribe实现分布式日志收集,并与log4j集成 (20) Thrift+scribe实现分布式日志收集,并与log4j集成7. bool true, false
byte 8位的有符号整数
i16 16位的有符号整数
i32 32位的有符号整数
i64 64位的有符号整数
double 64位的浮点数
string UTF-8编码的字符串
binary 字符数组
struct 结构体
list<type> 有序的元素列表,类似于STL的vector
set<type> 无序的不重复元素集,类似于STL的set
map<type1,type2> key-value型的映射,类似于STL的map
exception 是一个继承于本地语言的exception基类
service 服务。包含多个函数接口(纯虚函数)
8. 定义数据类型
struct User {
1: i32 uid,
2: string name
}
定义服务接口
service UserService {
void store(1: User user),
User retrieve(1: i32 uid)
}
9. 导出命令
thrift –r –gen py service.thrift
可导出语言配置
cocoa(Cocoa)、cpp(C++)、csharp(C#)、
erl(Erlang)、hs(Haskell)、html(HTML)、
java(Java)、ocaml(OCaml)、perl(Perl)、
php(PHP)、py(Python)、rb(Ruby)、st(Smalltalk)、
xsd(XSD)
12. enum ResultCode
{
OK,
TRY_LATER
}
struct LogEntry
{
1: string category,
2: string message
}
service scribe extends fb303.FacebookService
{
ResultCode Log(1: list<LogEntry> messages);
}
13. 启动Scribe Server:
scribed config.conf
停止Scribe Server:
scribe_ctrl stop 11810
监控Scribe Server:
scribe_ctrl {command} 11810
▪ status – ALIVE表示运行
▪ version – 版本
▪ alive – 服务器启动时的时间(s)
▪ reload – 重载配置文件
▪ counters – 统计项
▪ received good, received bad, sent, denied for queue size, denied for rate, retries,
requeue, lost, received blank category
14. 配置项:
port: 端口,默认0
max_msg_per_second: 默认100,000
max_queue_size: 默认5,000,000 bytes
check_interval: 检查每个store间隔,默认5s
new_thread_per_category: 默认yes,为每一个catagory创建一个新
的线程
num_thrift_server_threads: 收取消息的监听线程数,默认1
例如:
port=1463
max_msg_per_second=2000000
max_queue_size=10000000
check_interval=3
15. file – 写入文件local or nfs.
network – 发送到Scribe Server
buffer – 主从存储,primary and a secondary
store
bucket – 按照消息头分类hash存储
null – 忽略
thriftfile – 类似于file的ThriftTFileTransport
multi – 同时写多个
参考:http://wiki.github.com/facebook/scribe/scribe-configuration
21. thrift-0.2.0(http://incubator.apache.org/thrift/)
./bootstrap.sh
./configure --with-boost=/usr/local/boost_1_36_0
make (较长时间)
make install
ll /usr/lib/python2.6/site-packages/thrift
fb303(在thrift里:cd contrib/fb303)
./bootstrap.sh
./configure --with-boost=/usr/local/boost_1_36_0
make
make install
ll /usr/lib/python2.6/site-packages/fb303
MONO错误需要pkg-config0.20
22. scribe-2.2(http://github.com/facebook/scribe)
vi /etc/profile
▪ export BOOST_ROOT=/usr/local/boost_1_36_0
▪ export LD_LIBRARY_PATH=/usr/local/boost_1_36_0/lib::/usr/lib:/usr/local/lib
. /etc/profile
./bootstrap.sh
./configure --with-boost=/usr/local/boost_1_36_0
make
make install
ll /usr/lib/python2.6/site-packages/scribe
拷贝python库
cp -r /usr/lib/python2.6/site-packages/* /usr/local/lib/python2.6/site-
packages
cp /usr/local/bin/python /usr/bin/python
验证:python, import thrift, import scribe
23. 启动服务端: /opt/soft/scribe
scribed examples/example1.conf
启动客户端: /opt/soft/scribe
echo "hello world" | ./examples/scribe_cat test
查看日志
more /tmp/scribetest/test/test_current
hello world
26. 配置文件:/usr/liuzhongbing
struct User {
1: i32 uid,
2: string name
}
service UserService {
void store(1: User user),
User retrieve(1: i32 uid)
}
生成:/usr/liuzhongbing
thrift -r -gen java user.thrift
产生:User.java、UserService.java
27. 编译:/usr/liuzhongbing/gen-java
javac -classpath /usr/local/lib/libthrift.jar:/opt/soft/slf4j-1.6.0/slf4j-api-
1.6.0.jar:./ *.java
启动:/usr/liuzhongbing/gen-java
服务端:java -classpath /usr/local/lib/libthrift.jar:/opt/soft/slf4j-
1.6.0/slf4j-api-1.6.0.jar:./ JavaServer 9091
▪ Starting the server...
▪ store: 1=liuzhongbing
▪ store: 2=guest
▪ retrieve: 1=liuzhongbing
客户端:java -classpath /usr/local/lib/libthrift.jar:/opt/soft/slf4j-
1.6.0/slf4j-api-1.6.0.jar:./ JavaClient localhost 9091
▪ retrieve: liuzhongbing
28. 生成:/usr/liuzhongbing
thrift -r -gen py user.thrift
启动:/usr/liuzhongbing/gen-py
服务端:python PythonServer.py 9091
▪ Starting the server...
▪ store: 1=liuzhongbing
▪ store: 2=guest
▪ retrieve: 1=liuzhongbing
客户端:python PythonClient.py localhost 9091
▪ retrieve: liuzhongbing
29. 生成:/opt/soft/scribe/if
thrift -r -I /opt/soft/thrift-0.2.0/contrib -gen java
scribe.thrift
thrift -r -I /opt/soft/thrift-0.2.0/contrib -gen py
scribe.thrift
编译:/opt/soft/scribe/if/gen-java
javac -classpath
/usr/local/lib/libthrift.jar:/opt/soft/slf4j-1.6.0/slf4j-
api-1.6.0.jar:./ *.java
30. 启动Python服务端:/opt/soft/scribe
scribed examples/example1.conf
▪ [Thu May 13 15:39:29 2010] "[test] Creating new category from model default"
▪ [Thu May 13 15:39:29 2010] "store thread starting"
▪ [Thu May 13 15:39:29 2010] "[test] Opened file </tmp/scribetest/test/test_00000> for
writing"
▪ [Thu May 13 15:39:29 2010] "[test] Opened file </tmp/test/test_00000> for writing"
▪ [Thu May 13 15:39:29 2010] "[test] Changing state from <DISCONNECTED> to
<SENDING_BUFFER>"
▪ [Thu May 13 15:39:29 2010] "[test] successfully read <0> entries from file
</tmp/test/test_00000>"
▪ [Thu May 13 15:39:29 2010] "[test] No more buffer files to send, switching to streaming
mode"
▪ [Thu May 13 15:39:29 2010] "[test] Changing state from <SENDING_BUFFER> to
<STREAMING>"
启动Java客户端:/opt/soft/scribe/if/gen-java
java -classpath /usr/local/lib/libthrift.jar:/opt/soft/slf4j-1.6.0/slf4j-api-
1.6.0.jar:./ JavaClient localhost 1463
33. Central Server:192.168.0.1:11810
启动:/usr/local/scribe/conf/central_start.sh
停止:/usr/local/scribe/conf/central_stop.sh
查询:/usr/local/scribe/conf/central_status.sh
统计:/usr/local/scribe/conf/central_counters.sh
日志目录:/opt/scribelog/central, 暂存
/opt/scribelog/tmpcentral
Local Server:192.168.0.2:11811(App所在主机)
启动:/usr/local/scribe/conf/local_start.sh
停止:/usr/local/scribe/conf/local_stop.sh
查询:/usr/local/scribe/conf/local_status.sh
统计:/usr/local/scribe/conf/local_counters.sh
远程主机:192.168.0.1:11810,暂存/opt/scribelog/tmplocal
配置文件参见:192.168.0.1:/usr/local/scribe/conf/
34. 引用scribe-log4j
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>scribe-log4j</artifactId>
<version>${scribe-log4j.version}</version>
</dependency>
log4j.properties中增加了几个功能配置:
# scribe
log4j.logger.com.mycompany=DEBUG,scribeAppender
log4j.appender.scribeAppender=org.apache.scribe_log4j.ScribeAppender
log4j.appender.scribeAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.scribeAppender.layout.ConversionPattern=[%p] - [%d{yyyy-MM-dd
HH:mm:ss}] - [%c.%t(%L)]: %m%n
log4j.appender.scribeAppender.host=localhost
log4j.appender.scribeAppender.port=11811
log4j.appender.scribeAppender.reconnectInterval=10000
log4j.appender.scribeAppender.category=scribe-service
注意:自动重连信息输出在当前App标准stderr.log中
35. 修改host指向Local Server
log4j.appender.scribeAppender.host=192.168.0.1
注意:App中必须host=localhost
运行scribe-log4j测试类
Test.java
查看服务端收到的日志:
tail -f /opt/scribelog/central/scribe-service/scribe-
service_current
在/commons/scribe-log4j下进行