NiFiで流れてくるFlowfileの中身をReplace（頭何かデータを入れたい、後ろにデータをつけたいとか）したい場合は、ReplaceTextプロセッサを使えます。
ReplaceText既定の画面
一個例を見てみましょう
例えば流れてくるFlowfileは一行のCSV
aaa,bbb,ccc,ddd
後ろにeeeをつけたい場合、以下のように書きます。
$1,eee
FlowfileのAttribute(filenameとか）を後ろにつけたい場合
＄１をシングルクォーテーションで囲む必要です。
${'$1'},${filename}
$1のdddをfffにreplaceして、eeeを追加したい場合
${'$1':replace('ddd','fff')},eee
一番重要なのは$1をシングルクォーテーションで囲むところかなぁ。囲まないと怒られます。
FreeIPAとAmbariの連携方法をメモしておきます。
HDP3.1.0、Ambari 2.7.3の環境です。
以下のスクリプトでサーバー上にインストールし、いくつかUser、Groupを作成する。
必要に応じて変更してください。
#ホスト名
export NAME=ipa-srv1
#Domain名
export DOMAIN=demotest.com
export REALM=$(echo ${DOMAIN} | awk '{print toupper($0)}')
export IP=$(ip addr | grep 'state UP' -A2 | tail -n1 | awk '{print $2}' | cut -f1 -d'/')
echo "IP is ${IP}"
echo "${IP} ${NAME}.${DOMAIN} ${NAME} $(hostname -f) $(hostname -s)" >> /etc/hosts
hostnamectl set-hostname ${NAME}.${DOMAIN}
hostnamectl --transient set-hostname ${NAME}
hostname ${NAME}.${DOMAIN}
echo 0 > /proc/sys/kernel/hung_task_timeout_secs
ethtool -K eth0 tso off
hostname -f
cat /etc/hosts
#install packages
sudo yum install ipa-server ipa-server-dns -y
#increase entropy
cat /proc/sys/kernel/random/entropy_avail
sudo yum install -y rng-tools
sudo systemctl start rngd
cat /proc/sys/kernel/random/entropy_avail
#sometimes needed to avoid server install failing
sudo service dbus restart
#install IPA server
ipa-server-install \
--realm ${REALM} --domain ${DOMAIN} \
-a BadPass#1 -p BadPass#1 \
--setup-dns \
--forwarder=8.8.8.8 --allow-zone-overlap --no-host-dns \
--auto-forwarders --auto-reverse --unattended
#kinit as admin
#パスワードをBadPass#1に設定する
echo BadPass#1 | kinit admin
# create a new principal to be used for ambari kerberos administration
ipa user-add hadoopadmin --first=Hadoop --last=Admin --shell=/bin/bash
# create a new principal to be used for read only ldab bind (whose password will expire in 90 days)
ipa user-add ldapbind --first=ldap --last=bind
# create a role and and give it privilege to manage users and services
ipa role-add hadoopadminrole
ipa role-add-privilege hadoopadminrole --privileges="User Administrators"
ipa role-add-privilege hadoopadminrole --privileges="Service Administrators"
#add the hadoopadmin user to the role
ipa role-add-member hadoopadminrole --users=hadoopadmin
#create users/groups
ipa group-add analyst --desc analyst
ipa group-add hr --desc hr
ipa group-add legal --desc legal
ipa group-add sales --desc sales
ipa group-add etl --desc etl
ipa group-add us_employee --desc us_employee
ipa group-add eu_employee --desc eu_employee
ipa group-add intern --desc intern
ipa group-add sudoers --desc sudoers
ipa user-add legal1 --first=legal1 --last=legal1 --shell=/bin/bash
ipa user-add legal2 --first=legal2 --last=legal2 --shell=/bin/bash
ipa user-add legal3 --first=legal3 --last=legal3 --shell=/bin/bash
ipa user-add hr1 --first=hr1 --last=hr1 --shell=/bin/bash
ipa user-add hr2 --first=hr2 --last=hr2 --shell=/bin/bash
ipa user-add hr3 --first=hr3 --last=hr3 --shell=/bin/bash
ipa user-add sales1 --first=sales1 --last=sales1 --shell=/bin/bash
ipa user-add sales2 --first=sales2 --last=sales2 --shell=/bin/bash
ipa user-add sales3 --first=sales3 --last=sales3 --shell=/bin/bash
ipa user-add joe_analyst --first=joe --last=analyst --shell=/bin/bash
ipa user-add ivanna_eu_hr --first=ivanna --last=hr --shell=/bin/bash
ipa user-add scott_intern --first=scott --last=intern --shell=/bin/bash
ipa user-add noobie --first=jon --last=snow --shell=/bin/bash
ipa group-add-member legal --users=legal1
ipa group-add-member legal --users=legal2
ipa group-add-member legal --users=legal3
ipa group-add-member hr --users=hr1
ipa group-add-member hr --users=hr2
ipa group-add-member hr --users=hr3
ipa group-add-member hr --users=ivanna_eu_hr
ipa group-add-member sales --users=sales1
ipa group-add-member sales --users=sales2
ipa group-add-member sales --users=sales3
ipa group-add-member analyst --users=joe_analyst
ipa group-add-member intern --users=scott_intern
ipa group-add-member us_employee --users=joe_analyst
ipa group-add-member eu_employee --users=ivanna_eu_hr
# create sudo rule
ipa sudorule-add admin_all_rule
ipa sudorule-mod admin_all_rule --cmdcat=all --hostcat=all
ipa sudorule-add-user admin_all_rule --groups=sudoers
# add noobie to the sudoers user, to enable sudo rules
ipa group-add-member sudoers --users=noobie
echo BadPass#1 > tmp.txt
echo BadPass#1 >> tmp.txt
ipa passwd hadoopadmin < tmp.txt
ipa passwd ldapbind < tmp.txt
ipa passwd legal1 < tmp.txt
ipa passwd legal2 < tmp.txt
ipa passwd legal3 < tmp.txt
ipa passwd hr1 < tmp.txt
ipa passwd hr2 < tmp.txt
ipa passwd hr3 < tmp.txt
ipa passwd sales1 < tmp.txt
ipa passwd sales2 < tmp.txt
ipa passwd sales3 < tmp.txt
ipa passwd joe_analyst < tmp.txt
ipa passwd ivanna_eu_hr < tmp.txt
ipa passwd scott_intern < tmp.txt
rm -f tmp.txt
管理画面でUser,Group,Password Policy等情報を確認設定できます。
ambari-server setup-ldap
Using python /usr/bin/python
Currently 'no auth method' is configured, do you wish to use LDAP instead [y/n] (y)? yes
Enter Ambari Admin login: admin
Enter Ambari Admin password:
Fetching LDAP configuration from DB. No configuration.
Please select the type of LDAP you want to use [AD/IPA/Generic](Generic):IPA
Primary LDAP Host (ipa.ambari.apache.org): ipa-srv1.demotest.com
Primary LDAP Port (636): 389
Secondary LDAP Host <Optional>:
Secondary LDAP Port <Optional>:
Use SSL [true/false] (true): false
User object class (posixAccount): person
User ID attribute (uid): uid
Group object class (posixGroup): groupofnames
Group name attribute (cn): cn
Group member attribute (member): member
Distinguished name attribute (dn): dn
Search Base (cn=accounts,dc=ambari,dc=apache,dc=org): dc=demotest,dc=com
Referral method [follow/ignore] (follow):follow
Bind anonymously [true/false] (false): false
Bind DN (uid=ldapbind,cn=users,cn=accounts,dc=ambari,dc=apache,dc=org): uid=ldapbind,cn=users,cn=accounts,dc=demotest,dc=com
Enter Bind DN Password:
Confirm Bind DN Password:
Handling behavior for username collisions [convert/skip] for LDAP sync (skip): convert
Force lower-case user names [true/false]:false
Results from LDAP are paginated when requested [true/false]:false
====================
Review Settings
====================
Primary LDAP Host (ipa.ambari.apache.org): ipa-srv1.demotest.com
Primary LDAP Port (636): 389
Use SSL [true/false] (true): false
User object class (posixAccount): person
User ID attribute (uid): uid
Group object class (posixGroup): groupofnames
Group name attribute (cn): cn
Group member attribute (member): member
Distinguished name attribute (dn): dn
Search Base (cn=accounts,dc=ambari,dc=apache,dc=org): dc=demotest,dc=com
Referral method [follow/ignore] (follow): follow
Bind anonymously [true/false] (false): false
Handling behavior for username collisions [convert/skip] for LDAP sync (skip): convert
Force lower-case user names [true/false]: false
Results from LDAP are paginated when requested [true/false]: false
ambari.ldap.connectivity.bind_dn: uid=ldapbind,cn=users,cn=accounts,dc=demotest,dc=com
ambari.ldap.connectivity.bind_password: *****
Save settings [y/n] (y)? y
Saving LDAP properties...
Saving LDAP properties finished
Ambari Server 'setup-ldap' completed successfully.
Ambariの既定のID/password(admin/admin)を使用
ambari-server sync-ldap --ldap-sync-admin-name=admin --ldap-sync-admin-password=admin --all
Using python /usr/bin/python
Syncing with LDAP...
Fetching LDAP configuration from DB.
Syncing all...
Completed LDAP Sync.
Summary:
memberships:
removed = 0
created = 33
users:
skipped = 0
removed = 0
updated = 1
created = 15
groups:
updated = 0
removed = 0
created = 292
Ambari Server 'sync-ldap' completed successfully.
ここではhadoopadminというユーザーにambari adminに設定する
NoからYesに変更する
hadoopadminでログインできていることを確認
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_monitor.py", line 78, in
AmsMonitor().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_monitor.py", line 43, in start
action = 'start'
File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunk
return fn(*args, **kwargs)
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/ams_service.py", line 109, in ams_service
user=params.ams_user
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
returns=self.resource.returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/sbin/ambari-metrics-monitor --config /etc/ambari-metrics-monitor/conf start' returned 255. psutil build directory is not empty, continuing...
Verifying Python version compatibility...
Using python /usr/bin/python2.7
Checking for previously running Metric Monitor...
/var/run/ambari-metrics-monitor/ambari-metrics-monitor.pid found with no process. Removing 5237...
Starting ambari-metrics-monitor
Verifying ambari-metrics-monitor process status with PID : 11790
Output of PID check :
ERROR: ambari-metrics-monitor start failed. For more details, see /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out:
====================
server_process_main(stop_handler)
File "/usr/lib/python2.6/site-packages/resource_monitoring/main.py", line 64, in server_process_main
controller = Controller(main_config, stop_handler)
File "/usr/lib/python2.6/site-packages/resource_monitoring/core/controller.py", line 46, in __init__
hostinfo = HostInfo(config)
File "/usr/lib/python2.6/site-packages/resource_monitoring/core/host_info.py", line 49, in __init__
self.__host_static_info = self.get_host_static_info()
File "/usr/lib/python2.6/site-packages/resource_monitoring/core/host_info.py", line 235, in get_host_static_info
boot_time = psutil.boot_time()
AttributeError: 'module' object has no attribute 'boot_time'
====================
Monitor out at: /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out
cd /usr/lib/python2.6/site-packages/resource_monitoring/psutil
make install ambari-metrics-monitor restart
| Evernote はあなたがすべてを記憶し、手間をかけずに整理できるようにお手伝いします。Evernote をダウンロードする |
| Evernote はあなたがすべてを記憶し、手間をかけずに整理できるようにお手伝いします。Evernote をダウンロードする |