Riak CS を使ってみる。s3cmd Fluentd S3plugin
ここで、先行で、試用出来る環境が手に入りました。
Riak CS ですね。
ちょうど、良い機会なので、ちょっと、試してみる事にしました。
ruby 1.9
s3cmd
は、インストールしてあるものとしてます。
試用した、OS は、Mac 10.7.5 です。
S3cmdで試用する
S3互換ということで、S3コマンドで弄ってみます。
~guutara$ s3cmd --configure Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options. Access key and Secret key are your identifiers for Amazon S3 Access Key [xxxxxx]: Secret Key [xxxxxxx]: Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password [zzzz]: Path to GPG program [/usr/local/bin/gpg]: When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP and can't be used if you're behind a proxy Use HTTPS protocol [No]: On some networks all internet access must go through a HTTP proxy. Try setting it here if you can't conect to S3 directly HTTP Proxy server name: New settings: Access Key: xxxxxx Secret Key: xxxxxx Encryption password: zzzz Path to GPG program: /usr/local/bin/gpg Use HTTPS protocol: False HTTP Proxy server name: HTTP Proxy server port: 0 Test access with supplied credentials? [Y/n] Please wait... Success. Your access key and secret key worked fine :-) Now verifying that encryption works... Success. Encryption and decryption worked fine :-) Save settings? [y/N] y Configuration saved to '/Users/guutara/.s3cfg'
こんな感じで、cs3cfg を作成しました。
入れたのは、アクセスキーと、Secret Key です。
そのあと、手で、.s3cfgの host_base あたりを、IDC用に書き換えます。
host_base = xxxxxx host_bucket = %(bucket)xxxxxx
これで、準備は完了です。
http://docs.basho.co.jp/riakcs/latest/tutorials/quick-start-riak-cs/
の確認を流してみます。
guutara$ s3cmd mb s3://test_bucket Bucket 's3://test_bucket/' created guutara$ s3cmd ls 2013-02-xx xx:xx s3://test_bucket guutara$ dd if=/dev/zero of=test_file bs=1024 count=2 2+0 records in 2+0 records out 2048 bytes transferred in 0.000056 secs (36552913 bytes/sec) guutara$ s3cmd put test_file s3://test_bucket test_file -> s3://test_bucket/test_file [1 of 1] 2048 of 2048 100% in 0s xx kB/s done guutara$ s3cmd ls s3://test_bucket 2013-02-xx xx:xx 2048 s3://test_bucket/test_file guutara$ rm test_file guutara$ ls guutara$ s3cmd get s3://test_bucket/test_file s3://test_bucket/test_file -> ./test_file [1 of 1] 2048 of 2048 100% in 0s XXX kB/s done guutara$ ls test_file
簡単ですねぇ。
とりあえず動作は、確認できました。
fluentd + s3
次は、Fluentd の s3 pluginで、Riak CS に書き込めるかテストしてみます。
結果は、簡単なテストですが、オーケーでしたよ。
まずは、fluentd インストール。
~ guutara$ gem install fluentd Fetching: msgpack-0.4.7.gem (100%) Building native extensions. This could take a while... Fetching: yajl-ruby-1.1.0.gem (100%) Building native extensions. This could take a while... Fetching: iobuffer-1.1.2.gem (100%) Building native extensions. This could take a while... Fetching: cool.io-1.1.0.gem (100%) Building native extensions. This could take a while... Fetching: http_parser.rb-0.5.3.gem (100%) Building native extensions. This could take a while... Fetching: fluentd-0.10.31.gem (100%) Successfully installed msgpack-0.4.7 Successfully installed yajl-ruby-1.1.0 Successfully installed iobuffer-1.1.2 Successfully installed cool.io-1.1.0 Successfully installed http_parser.rb-0.5.3 Successfully installed fluentd-0.10.31 6 gems installed Installing ri documentation for msgpack-0.4.7... Installing ri documentation for yajl-ruby-1.1.0... Installing ri documentation for iobuffer-1.1.2... Installing ri documentation for cool.io-1.1.0... Installing ri documentation for http_parser.rb-0.5.3... Installing ri documentation for fluentd-0.10.31... Installing RDoc documentation for msgpack-0.4.7... Installing RDoc documentation for yajl-ruby-1.1.0... Installing RDoc documentation for iobuffer-1.1.2... Installing RDoc documentation for cool.io-1.1.0... Installing RDoc documentation for http_parser.rb-0.5.3... Installing RDoc documentation for fluentd-0.10.31... ~ guutara$ fluentd --setup ./fluent Installed ./fluent/fluent.conf. ~ guutara$ fluentd -c ./fluent/fluent.conf 2013-02-xx xx +0900: starting fluentd-0.10.31 2013-02-xx xx +0900: reading config file path="./fluent/fluent.conf" 2013-02-xx xx +0900: using configuration file: <ROOT> <source> type forward </source> <source> type http port 8888 </source> <source> type debug_agent port 24230 </source> <match debug.**> type stdout </match> </ROOT> 2013-02-xx xx +0900: adding source type="forward" 2013-02-xx xx +0900: adding source type="http" 2013-02-xx xx +0900: adding source type="debug_agent" 2013-02-xx xx +0900: adding match pattern="debug.**" type="stdout" 2013-02-xx xx +0900: listening fluent socket on 0.0.0.0:24224 2013-02-xx xx +0900: listening dRuby uri="druby://0.0.0.0:24230" object="Engine" ~ guutara$ gem install fluent-plugin-s3 Fetching: uuidtools-2.1.3.gem (100%) Fetching: nokogiri-1.5.6.gem (100%) Building native extensions. This could take a while... Fetching: aws-sdk-1.8.1.2.gem (100%) Fetching: fluent-plugin-s3-0.2.6.gem (100%) Successfully installed uuidtools-2.1.3 Successfully installed nokogiri-1.5.6 Successfully installed aws-sdk-1.8.1.2 Successfully installed fluent-plugin-s3-0.2.6 4 gems installed Installing ri documentation for uuidtools-2.1.3... Installing ri documentation for nokogiri-1.5.6... Installing ri documentation for aws-sdk-1.8.1.2... Installing ri documentation for fluent-plugin-s3-0.2.6... Installing RDoc documentation for uuidtools-2.1.3... Installing RDoc documentation for nokogiri-1.5.6... Installing RDoc documentation for aws-sdk-1.8.1.2... Installing RDoc documentation for fluent-plugin-s3-0.2.6...
次は、fluent.conf に、Riak CS に、接続出来る設定をいれます。
今回は、テスト用に、s3.* で、マッチするようにしました。
## match not matched logs and write to file #<match **> # type file # path /var/log/fluent/else # compress gz #</match> ## match s3 #<match pattern> <match s3.*> type s3 aws_key_id ACCESS_KEY aws_sec_key SECRET_KEY s3_bucket fluent_log s3_endpoint HOST_DOMAIN # s3_object_key_format {path}{time_slice}_{index}.{file_extension} path logs/ # path / buffer_path /Users/guutara/log/fluent/s3 time_slice_format %Y%m%d-%H # time_slice_wait 10m utc flush_interval 1s </match>
さて、fluentd を起動します。
~ guutara$ fluentd -c fluent/fluent.conf 2013-02-xx xx +0900: starting fluentd-0.10.31 2013-02-xx xx +0900: reading config file path="fluent/fluent.conf" 2013-02-xx xx +0900: using configuration file: <ROOT> <source> type forward </source> <source> type http port 8888 </source> <source> type debug_agent port 24230 </source> <match debug.**> type stdout </match> <match s3.*> type s3 aws_key_id xxxxxx aws_sec_key xxxxxx s3_bucket fluent_log s3_endpoint xxxxxx path logs/ buffer_path /Users/guutara/log/fluent/s3 time_slice_format %Y%m%d-%H utc flush_interval 1s </match> </ROOT> 2013-02-xx xx +0900: adding source type="forward" 2013-02-xx xx +0900: adding source type="http" 2013-02-xx xx +0900: adding source type="debug_agent" 2013-02-xx xx +0900: adding match pattern="debug.**" type="stdout" 2013-02-xx xx +0900: adding match pattern="s3.*" type="s3" 2013-02-xx xx +0900: listening fluent socket on 0.0.0.0:24224 2013-02-xx xx +0900: listening dRuby uri="druby://0.0.0.0:24230" object="Engine"
では、テストしてみましょう。
http に、メッセージを投げつけて、s3plugin に処理をさせ、Riak CS に書き込ませます。
その内容を、ダウンロードして表示してみます。
~ guutara$ s3cmd ls s3://fluent_log ~ guutara$ curl http://localhost:8888/s3.http -F 'json={"foo":"bar"}' ~ guutara$ s3cmd ls s3://fluent_log 2013-02-xx xx 63 s3://fluent_log/logs/201302xx-10_0.gz ~ guutara$ s3cmd get s3://fluent_log/logs/201302xx-10_0.gz s3://fluent_log/logs/201302xx-10_0.gz -> ./201302xx-10_0.gz [1 of 1] 63 of 63 100% in 0s xx B/s done ~ guutara$ gzcat 201302xx-10_0.gz 2013-02-xxTxxZ s3.http {"foo":"bar"}
はい、うまくいきました。
なかなか、楽しいですね。w