Riak CS を使ってみる。s3cmd Fluentd S3plugin

オブジェクトストレージ|IDCFクラウド

ここで、先行で、試用出来る環境が手に入りました。
Riak CS ですね。
ちょうど、良い機会なので、ちょっと、試してみる事にしました。

ruby 1.9
s3cmd

は、インストールしてあるものとしてます。

試用した、OS は、Mac 10.7.5 です。

S3cmdで試用する

S3互換ということで、S3コマンドで弄ってみます。

~guutara$ s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key [xxxxxx]: 
Secret Key [xxxxxxx]: 

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password [zzzz]: 
Path to GPG program [/usr/local/bin/gpg]: 

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]: 

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't conect to S3 directly
HTTP Proxy server name: 

New settings:
  Access Key: xxxxxx
  Secret Key: xxxxxx
  Encryption password: zzzz
  Path to GPG program: /usr/local/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name: 
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] 
Please wait...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] y
Configuration saved to '/Users/guutara/.s3cfg'

こんな感じで、cs3cfg を作成しました。
入れたのは、アクセスキーと、Secret Key です。

そのあと、手で、.s3cfgの host_base あたりを、IDC用に書き換えます。

host_base = xxxxxx
host_bucket = %(bucket)xxxxxx

これで、準備は完了です。

http://docs.basho.co.jp/riakcs/latest/tutorials/quick-start-riak-cs/

の確認を流してみます。

 guutara$ s3cmd mb s3://test_bucket
Bucket 's3://test_bucket/' created

guutara$ s3cmd ls
2013-02-xx xx:xx  s3://test_bucket

 guutara$ dd if=/dev/zero of=test_file bs=1024 count=2
2+0 records in
2+0 records out
2048 bytes transferred in 0.000056 secs (36552913 bytes/sec)

 guutara$ s3cmd put test_file s3://test_bucket
test_file -> s3://test_bucket/test_file  [1 of 1]
 2048 of 2048   100% in    0s    xx kB/s  done

 guutara$ s3cmd ls s3://test_bucket
2013-02-xx xx:xx      2048   s3://test_bucket/test_file

 guutara$ rm test_file 
 guutara$ ls

 guutara$ s3cmd get s3://test_bucket/test_file
s3://test_bucket/test_file -> ./test_file  [1 of 1]
 2048 of 2048   100% in    0s    XXX kB/s  done

 guutara$ ls
test_file

簡単ですねぇ。

とりあえず動作は、確認できました。

fluentd + s3

次は、Fluentd の s3 pluginで、Riak CS に書き込めるかテストしてみます。
結果は、簡単なテストですが、オーケーでしたよ。

まずは、fluentd インストール。

~ guutara$ gem install fluentd
Fetching: msgpack-0.4.7.gem (100%)
Building native extensions.  This could take a while...
Fetching: yajl-ruby-1.1.0.gem (100%)
Building native extensions.  This could take a while...
Fetching: iobuffer-1.1.2.gem (100%)
Building native extensions.  This could take a while...
Fetching: cool.io-1.1.0.gem (100%)
Building native extensions.  This could take a while...
Fetching: http_parser.rb-0.5.3.gem (100%)
Building native extensions.  This could take a while...
Fetching: fluentd-0.10.31.gem (100%)
Successfully installed msgpack-0.4.7
Successfully installed yajl-ruby-1.1.0
Successfully installed iobuffer-1.1.2
Successfully installed cool.io-1.1.0
Successfully installed http_parser.rb-0.5.3
Successfully installed fluentd-0.10.31
6 gems installed
Installing ri documentation for msgpack-0.4.7...
Installing ri documentation for yajl-ruby-1.1.0...
Installing ri documentation for iobuffer-1.1.2...
Installing ri documentation for cool.io-1.1.0...
Installing ri documentation for http_parser.rb-0.5.3...
Installing ri documentation for fluentd-0.10.31...
Installing RDoc documentation for msgpack-0.4.7...
Installing RDoc documentation for yajl-ruby-1.1.0...
Installing RDoc documentation for iobuffer-1.1.2...
Installing RDoc documentation for cool.io-1.1.0...
Installing RDoc documentation for http_parser.rb-0.5.3...
Installing RDoc documentation for fluentd-0.10.31...

~ guutara$ fluentd --setup ./fluent
Installed ./fluent/fluent.conf.
~ guutara$ fluentd -c ./fluent/fluent.conf
2013-02-xx xx +0900: starting fluentd-0.10.31
2013-02-xx xx +0900: reading config file path="./fluent/fluent.conf"
2013-02-xx xx +0900: using configuration file: <ROOT>
  <source>
    type forward
  </source>
  <source>
    type http
    port 8888
  </source>
  <source>
    type debug_agent
    port 24230
  </source>
  <match debug.**>
    type stdout
  </match>
</ROOT>
2013-02-xx xx +0900: adding source type="forward"
2013-02-xx xx +0900: adding source type="http"
2013-02-xx xx +0900: adding source type="debug_agent"
2013-02-xx xx +0900: adding match pattern="debug.**" type="stdout"
2013-02-xx xx +0900: listening fluent socket on 0.0.0.0:24224
2013-02-xx xx +0900: listening dRuby uri="druby://0.0.0.0:24230" object="Engine"

~ guutara$ gem install fluent-plugin-s3
Fetching: uuidtools-2.1.3.gem (100%)
Fetching: nokogiri-1.5.6.gem (100%)
Building native extensions.  This could take a while...
Fetching: aws-sdk-1.8.1.2.gem (100%)
Fetching: fluent-plugin-s3-0.2.6.gem (100%)
Successfully installed uuidtools-2.1.3
Successfully installed nokogiri-1.5.6
Successfully installed aws-sdk-1.8.1.2
Successfully installed fluent-plugin-s3-0.2.6
4 gems installed
Installing ri documentation for uuidtools-2.1.3...
Installing ri documentation for nokogiri-1.5.6...
Installing ri documentation for aws-sdk-1.8.1.2...
Installing ri documentation for fluent-plugin-s3-0.2.6...
Installing RDoc documentation for uuidtools-2.1.3...
Installing RDoc documentation for nokogiri-1.5.6...
Installing RDoc documentation for aws-sdk-1.8.1.2...
Installing RDoc documentation for fluent-plugin-s3-0.2.6...

次は、fluent.conf に、Riak CS に、接続出来る設定をいれます。
今回は、テスト用に、s3.* で、マッチするようにしました。

## match not matched logs and write to file
#<match **>
#  type file
#  path /var/log/fluent/else
#  compress gz
#</match>

## match s3
#<match pattern>
<match s3.*>
  type s3

  aws_key_id ACCESS_KEY
  aws_sec_key SECRET_KEY
  s3_bucket fluent_log
  s3_endpoint HOST_DOMAIN
#  s3_object_key_format {path}{time_slice}_{index}.{file_extension}
  path logs/
#  path /
  buffer_path /Users/guutara/log/fluent/s3

 time_slice_format %Y%m%d-%H
#  time_slice_wait 10m
  utc
  flush_interval 1s
</match>

さて、fluentd を起動します。

~ guutara$ fluentd -c fluent/fluent.conf
2013-02-xx  xx +0900: starting fluentd-0.10.31
2013-02-xx  xx +0900: reading config file path="fluent/fluent.conf"
2013-02-xx  xx +0900: using configuration file: <ROOT>
  <source>
    type forward
  </source>
  <source>
    type http
    port 8888
  </source>
  <source>
    type debug_agent
    port 24230
  </source>
  <match debug.**>
    type stdout
  </match>
  <match s3.*>
    type s3
    aws_key_id xxxxxx
    aws_sec_key xxxxxx
    s3_bucket fluent_log
    s3_endpoint xxxxxx
    path logs/
    buffer_path /Users/guutara/log/fluent/s3
    time_slice_format %Y%m%d-%H
    utc 
    flush_interval 1s
  </match>
</ROOT>
2013-02-xx  xx +0900: adding source type="forward"
2013-02-xx  xx +0900: adding source type="http"
2013-02-xx  xx +0900: adding source type="debug_agent"
2013-02-xx  xx +0900: adding match pattern="debug.**" type="stdout"
2013-02-xx  xx +0900: adding match pattern="s3.*" type="s3"
2013-02-xx  xx +0900: listening fluent socket on 0.0.0.0:24224
2013-02-xx  xx +0900: listening dRuby uri="druby://0.0.0.0:24230" object="Engine"

では、テストしてみましょう。
http に、メッセージを投げつけて、s3plugin に処理をさせ、Riak CS に書き込ませます。
その内容を、ダウンロードして表示してみます。

~ guutara$ s3cmd ls s3://fluent_log
~ guutara$ curl http://localhost:8888/s3.http -F 'json={"foo":"bar"}'
~ guutara$ s3cmd ls s3://fluent_log
2013-02-xx xx        63   s3://fluent_log/logs/201302xx-10_0.gz
~ guutara$ s3cmd get s3://fluent_log/logs/201302xx-10_0.gz
s3://fluent_log/logs/201302xx-10_0.gz -> ./201302xx-10_0.gz  [1 of 1]
 63 of 63   100% in    0s  xx B/s  done
~ guutara$ gzcat 201302xx-10_0.gz
2013-02-xxTxxZ	s3.http	{"foo":"bar"}

はい、うまくいきました。

なかなか、楽しいですね。w