Commit d34afbe8 authored by Walter Heck's avatar Walter Heck

Migrated deployment instructions from GDocs

parent 2137bdb3
# Instructions for deploying OpsTheater
1. Get DO access
make sure to upload an ssh key here: https://cloud.digitalocean.com/settings/security
2. Determine hostnames/sizes/ipaddresses
|-
FQDN | IP | Specs (minimum) | role -|
|-
puppet.olindata.com
37.139.31.45
4GB RAM
puppet + foreman
gitlab.olindata.com
2GB RAM
gitlab + mattermost
elastic.olindata.com
2GB RAM
logstash + elastic + kibana / grafana
icinga.olindata.com
2GB RAM
monitoring
mysql.olindata.com
2GB RAM
mysql
Provision the first vm
Manually use the roles.yaml bash script (modified) to setup puppet master
DO specific: Update /etc/hosts with the correct IP, check for DO specific /etc/hosts template setting and remove/update the template.
Topics That needs to be discussed
Public IP vs local IP
local user creation
SSL Certificate
FOSS Master for OpsTheater
Disable firewalld ************************ Enable firewalld again once installation is complete.
/bin/systemctl stop firewalld
/bin/systemctl disable firewalld
Install puppet server
/bin/yum install -y -q https://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm epel-release
/bin/yum install -y -q puppetserver
Install puppet modules
/opt/puppetlabs/bin/puppet module install puppetlabs/puppetdb #v5.1.1
/opt/puppetlabs/bin/puppet module install puppetlabs/vcsrepo #v1.3.2
/opt/puppetlabs/bin/puppet module install zack/r10k #v3.2.0
/opt/puppetlabs/bin/puppet module install abrader/gms #v1.0.2
/opt/puppetlabs/bin/puppet module install ajcrowe/supervisord --ignore-dependencies #v0.6.0
SetHostname, configure autosign cert, enable puppetserver and start it.
/bin/echo '==> Set puppetserver hostname'
/opt/puppetlabs/bin/puppet apply -e 'ini_setting { "set master hostname": ensure => present, section => "main", value => $::fqdn, path => "/etc/puppetlabs/puppet/puppet.conf", setting => "server" }'
/bin/echo '==> Set puppetserver autosign'
/opt/puppetlabs/bin/puppet apply -e 'file { "/etc/puppetlabs/puppet/autosign.conf": ensure => file, content => "$::fqdn\n*\n", }'
/bin/touch /etc/puppetlabs/code/environments/production/manifests/site.pp
/opt/puppetlabs/bin/puppet resource service puppetserver enable=true
/bin/systemctl start puppetserver
Install Development tools.
/bin/yum groupinstall -y -q "Development Tools" "Development Libraries"
Install Foreman.
Note:
Create a directory in /opt/installer and copy foreman_installer.pp in the same.
Create directories as /opt/installer/files/foreman and copy all the files with the same name in it. Location if files: (https://github.com/olindata/opstheater-oscar/tree/master/files/foreman)
Create directories as /opt/installer/files/smart-proxy and copy all the files with the same name in it. Location if files: (https://github.com/olindata/opstheater-oscar/tree/master/files/smart-proxy)
Edit all file in /opt/installer/files to reflect correct name eg: puppet.olindata.com
/opt/puppetlabs/bin/puppet apply /opt/installer/foreman_installation.pp
Set Foreman report
/opt/puppetlabs/bin/puppet apply -e 'ini_setting { "set foreman report ": ensure => present, section => "main", value => "log,foreman", path => "/etc/puppetlabs/puppet/puppet.conf", setting => "reports" }'
Install bundler
/usr/bin/gem install bundler --no-ri --no-rdoc --quiet --no-verbose
cd /opt/foreman; /usr/local/bin/bundle install --without mysql2 sqlite test --path vendor --quiet
cd /opt/foreman; /usr/local/bin/bundle update foreman_default_hostgroup
After this edit /opt/foreman/Gemfile.lock search for rake and change the version from 11.0.1 to 10.5.0.
Migrate the DB
RAILS_ENV=production bundle exec rake db:migrate --quiet
RAILS_ENV=production bundle exec rake assets:precompile locale:pack apipie:cache --quiet
Grep the Credentials
RAILS_ENV=production bundle exec rake db:seed|grep "Login"
Eg:
Login credentials: admin / e3iHDhY7QCiwr47n
Smart Proxy Bundle install
cd /opt/smart-proxy; /usr/local/bin/bundle install --without development test --path vendor --quiet
Above step will install rake 11.0.1. Edit Gemlock.file and change value for 'rake' from 11.0.1 to 10.5.0 and run #bundle install
cd /opt/smart-proxy; /usr/local/bin/bundle update rubocop
Run Foreman Post Install
Copy foreman_post_install.pp from (https://github.com/olindata/opstheater-oscar/blob/master/manifests/foreman_post_install.pp) to /opt/installer/
/opt/puppetlabs/bin/puppet apply /opt/installer/foreman_post_install.pp
Clear IPtables
iptables -F
Add Smart Proxy
/bin/curl -k -s -u admin:PASSWORD_FROM_PREVIOUS_COMMAND -H "Accept: version=2,application/json" -H "Content-Type: application/json" -X POST -d '{ "name": "foreman", "url": "https://puppet.olindata.com:8443" } ' http://puppet.olindata.com:3000/api/smart_proxies
/bin/curl -k -s -u admin:PASSWORD_FROM_PREVIOUS_COMMAND -H "Accept: version=2,application/json" -H "Content-Type: application/json" -X POST -d '{ }' http://puppet.olindata.com:3000/api/smart_proxies/1/import_puppetclasses
Expected Output of above 2 commands:
[root@puppet installer]# /bin/curl -k -s -u admin:e3iHDhY7QCiwr47n -H "Accept: version=2,application/json" -H "Content-Type: application/json" -X POST -d '{ }' http://puppet.olindata.com:3000/api/smart_proxies/1/import_puppetclasses
{
"message": "Successfully updated environment and puppetclasses from the on-disk puppet installation",
"environments_with_new_puppetclasses": 1,
"environments_updated_puppetclasses": 0,
"environments_obsolete": 0,
"results": [{"name":"production","actions":["new"],"new_puppetclasses":["ruby::params","ruby::dev","ruby::gemrc","ruby","ruby::config","gcc::params","gcc","stdlib","stdlib::stages","apt::params","apt","apt::backports","apt::update","puppetdb::database::postgresql","puppetdb::params","puppetdb::globals","puppetdb::master::storeconfigs","puppetdb::master::config","puppetdb::master::report_processor","puppetdb::master::puppetdb_conf","puppetdb::master::routes","puppetdb::server","puppetdb","puppetdb::server::firewall","puppetdb::server::validate_read_db","puppetdb::server::read_database","puppetdb::server::global","puppetdb::server::database","puppetdb::server::jetty","puppetdb::server::puppetdb","puppetdb::server::command_processing","puppetdb::server::validate_db","r10k::params","r10k::postrun_command","r10k::webhook","r10k::mcollective::application","r10k","r10k::install","r10k::mcollective","r10k::config","r10k::install::bundle","r10k::install::pe_gem","r10k::install::puppet_gem","r10k::install::gem","r10k::install::portage","r10k::webhook::package","r10k::webhook::config","r10k::prerun_command","postgresql::params","postgresql::globals","postgresql::repo::apt_postgresql_org","postgresql::repo::yum_postgresql_org","postgresql::client","postgresql::server","postgresql::repo","postgresql::server::initdb","postgresql::server::service","postgresql::server::reload","postgresql::server::plpython","postgresql::server::install","postgresql::server::plperl","postgresql::server::config","postgresql::server::passwd","postgresql::server::postgis","postgresql::server::contrib","postgresql::lib::python","postgresql::lib::devel","postgresql::lib::java","postgresql::lib::perl","postgresql::lib::docs","supervisord::params","supervisord::service","supervisord::reload","supervisord","supervisord::install","supervisord::config","supervisord::pip","git","git::subtree","git::gitosis","portage::params","portage","portage::install","make::params","make","make::install","firewall::params","firewall::linux","firewall","firewall::linux::debian","firewall::linux::redhat","firewall::linux::archlinux","firewall::linux::gentoo"]}]
}
[root@puppet installer]# /bin/curl -k -s -u admin:e3iHDhY7QCiwr47n -H "Accept: version=2,application/json" -H "Content-Type: application/json" -X POST -d '{ "name": "OpsTheater Infra", "environment_id": "1", "puppet_ca_proxy_id": "1", "puppet_proxy_id": "1" } ' http://puppet.olindata.com:3000/api/hostgroups
{"subnet_id":null,"subnet_name":null,"operatingsystem_id":null,"operatingsystem_name":null,"domain_id":null,"domain_name":null,"environment_id":1,"environment_name":"production","compute_profile_id":null,"compute_profile_name":null,"ancestry":null,"puppet_proxy_id":1,"puppet_ca_proxy_id":1,"ptable_id":null,"ptable_name":null,"medium_id":null,"medium_name":null,"architecture_id":null,"architecture_name":null,"realm_id":null,"realm_name":null,"created_at":"2016-03-09T11:48:29Z","updated_at":"2016-03-09T11:48:29Z","id":1,"name":"OpsTheater Infra","title":"OpsTheater Infra","parameters":[],"template_combinations":[],"puppetclasses":[],"config_groups":[],"all_puppetclasses":[]}
Restart PuppetServer
/bin/systemctl restart puppetserver
Running r10K
Download Opstheater-Control.tar.gz to the local server.
Untar opstehater-control in /root.
-Take the backup of existing production enviornment. /etc/puppetlabs/code/production as /etc/puppetlabs/code/production_backup
-Move /root/opstheater-control to /etc/puppetlabs/code/production
Install r10K
-/opt/puppetlabs/bin/puppetserver gem install r10k
-gem install 10k
Execute r10k from /etc/puppetlabs/code/production
-r10k puppetfile install -v
THis will install all the required modules mentioned in Puppetfile in current directory
Setup Hiera
/opt/puppetlabs/bin/puppet config set hiera_config /etc/puppetlabs/code/environments/production/hiera.yaml
Restart PuppetServer
/bin/systemctl restart puppetserver
Update Foreman puppet environments
/bin/curl -k -s -u admin:PASSWORD_FROM_PREVIOUS_COMMAND -H "Accept: version=2,application/json" -H "Content-Type: application/json" -X POST -d '{ }' http://puppet.olindata.com:3000/api/smart_proxies/1/import_puppetclasses
Run puppet agent and then stop puppet service
/opt/puppetlabs/bin/puppet agent -t || true
GitLab Server Setup.
Configure repo and install puppet.
/bin/yum install -y epel-release http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server puppet.olindata.com
Configure/adjust Master
Ensure correct filename ymal file exists in bellow locations:
/etc/puppetlabs/code/environments/production/hieradata/10.fqdn
Once this is done. Puppet agent.
/opt/puppetlabs/bin/puppet agent -t
Elastic Server Setup
Configure repo and install puppet.
/bin/yum install -y epel-release http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server puppet.olindata.com
Configure/adjust Master
Ensure correct filename ymal file exists in bellow locations:
/etc/puppetlabs/code/environments/production/hieradata/10.fqdn
Merge elastic.olindata.com.yaml and logstash.olindata.vm.yaml As we have single server or both application.
(Remember to edit the file and remove extra --- and change the server name to elastic from elasticsearch)
Logstash settings needs to be checked for public vs private ip
Ref: - "elastic.olindata.com:9200"
wget needs to be installed before running puppet agent -t.
Add DNS entry for logstash.olindata.com OR update /etc/filebeat/filebeat.yml to talk to elastic.olindata.com
update site.pp as bellow:
==
node 'elastic.olindata.com' {
include opstheater::role::elastic::server
include opstheater::role::logstash::server
}
==
Run Puppet agent.
MYSQL server Setup
Configure repo and install puppet.
/bin/yum install -y epel-release http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server puppet.olindata.com
Configure/adjust Master
update /etc/puppetlabs/code/environment/production/hieradata/60-.opstheater.yaml to reflect myqsl server ip.
==
##########################
## MySQL related settings
##########################
## MySQL related settings
# Variable: opstheater::mysql::fqdn
# Description:
# Default value: "mysql.%{hiera('opstheater::domain')}"
'opstheater::mysql::fqdn': "mysql.%{hiera('opstheater::domain')}"
# Variable: opstheater::mysql::ipaddress
# Description:
# Default value: '10.20.1.60'
'opstheater::mysql::ipaddress': '10.129.2.113'
# Variable: opstheater::mysql::whitelist_range
# Description:
# Default value: '10.20.1.%'
'opstheater::mysql::whitelist_range': '10.129.%'
==
Run Puppet Agent
This will throw error for the 1st time as below but on the second run it works fine.
nfo: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]: Filebucketed /etc/my.cnf to puppet with sum 80e1eb23d5fbd77fc0ff681b0f0df297
Notice: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]/content: content changed '{md5}80e1eb23d5fbd77fc0ff681b0f0df297' to '{md5}3ab13aa54c001fe3966a08ae49c5517d'
Notice: Disabling SSL is evil! You should never ever do this except if you are forced to use a mysql version compiled without SSL support
Notice: /Stage[main]/Mysql::Server::Config/Notify[ssl-disable]/message: defined 'message' as 'Disabling SSL is evil! You should never ever do this except if you are forced to use a mysql version compiled without SSL support'
Error: Could not start Service[mysqld]: Execution of '/usr/bin/systemctl start mysqld' returned 1: Job for mysqld.service failed because the control process exited with error code. See "systemctl status mysqld.service" and "journalctl -xe" for details.
Error: /Stage[main]/Mysql::Server::Service/Service[mysqld]/ensure: change from stopped to running failed: Could not start Service[mysqld]: Execution of '/usr/bin/systemctl start mysqld' returned 1: Job for mysqld.service failed because the control process exited with error code. See "systemctl status mysqld.service" and "journalctl -xe" for details.
Notice: /Stage[main]/Opstheater::Profile::Mysql/File[/var/log/mysql]/ensure: created
Notice: /Stage[main]/Mysql::Server::Service/File[/var/log/mysql/error.log]/ensure: created
Info: Class[Mysql::Server::Service]: Unscheduling all events on Class[Mysql::Server::Service]
Notice: /Stage[main]/Mysql::Server::Root_password/Exec[remove install pass]: Dependency Service[mysqld] has failures: true
Warning: /Stage[main]/Mysql::Server::Root_password/Exec[remove install pass]: Skipping because of failed dependencies
Info: Creating state file /opt/puppetlabs/puppet/cache/state/state.yaml
Error: Failed to apply catalog: Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Icinga Server Setup
Configure repo and install puppet.
/bin/yum install -y epel-release http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server puppet.olindata.com
Configure/adjust Master
Update /etc/puppetlabs/code/environment/production/hieradata/60-.opstheater.yaml in icinga section to reflect correct name of icinga on the below line.
Also update the ipaddress to the public ipaddress.
'opstheater::icinga::fqdn': "icinga.%{hiera('opstheater::domain')}"
'opstheater::icinga::ipaddress': '37.139.31.34'
update site.pp to reflect correct name.
==
# runs standalone monitoring setup with icinga2 and icinga web2
node 'icinga.olindata.com' {
include opstheater::role::monitoring::standalone
}
==
Run puppet agent
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment