Commit 61ba194e authored by Walter Heck's avatar Walter Heck

Remove directory 'latest' and make it a symlink instead

parent 5c0fe5d5
1.5/
\ No newline at end of file
Support guidelines
We can only deliver support if all of the following conditions have been met:
no version upgrades of tools outside of OpsTheater provided versions
no changes outside the 70.opstheater_custom hiera file
# Instructions for deploying OpsTheater
1. Get DO access
make sure to upload an ssh key here: https://cloud.digitalocean.com/settings/security
2. Determine hostnames/sizes/ipaddresses
|-
FQDN | IP | Specs (minimum) | role -|
|-
puppet.olindata.com
37.139.31.45
4GB RAM
puppet + foreman
gitlab.olindata.com
2GB RAM
gitlab + mattermost
elastic.olindata.com
2GB RAM
logstash + elastic + kibana / grafana
icinga.olindata.com
2GB RAM
monitoring
mysql.olindata.com
2GB RAM
mysql
Provision the first vm
Manually use the roles.yaml bash script (modified) to setup puppet master
DO specific: Update /etc/hosts with the correct IP, check for DO specific /etc/hosts template setting and remove/update the template.
Topics That needs to be discussed
Public IP vs local IP
local user creation
SSL Certificate
## FOSS Master for OpsTheater
Disable firewalld ************************ Enable firewalld again once installation is complete.
```bash
/bin/systemctl stop firewalld
/bin/systemctl disable firewalld
```
Install puppet server
```
/bin/yum install -y -q https://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm epel-release
/bin/yum install -y -q puppetserver
```
Install puppet modules
```
/opt/puppetlabs/bin/puppet module install puppetlabs/puppetdb #v5.1.1
/opt/puppetlabs/bin/puppet module install puppetlabs/vcsrepo #v1.3.2
/opt/puppetlabs/bin/puppet module install zack/r10k #v3.2.0
/opt/puppetlabs/bin/puppet module install abrader/gms #v1.0.2
/opt/puppetlabs/bin/puppet module install ajcrowe/supervisord --ignore-dependencies #v0.6.0
```
SetHostname, configure autosign cert, enable puppetserver and start it.
```
/bin/echo '==> Set puppetserver hostname'
/opt/puppetlabs/bin/puppet apply -e 'ini_setting { "set master hostname": ensure => present, section => "main", value => $::fqdn, path => "/etc/puppetlabs/puppet/puppet.conf", setting => "server" }'
/bin/echo '==> Set puppetserver autosign'
/opt/puppetlabs/bin/puppet apply -e 'file { "/etc/puppetlabs/puppet/autosign.conf": ensure => file, content => "$::fqdn\n*\n", }'
/bin/touch /etc/puppetlabs/code/environments/production/manifests/site.pp
/opt/puppetlabs/bin/puppet resource service puppetserver enable=true
/bin/systemctl start puppetserver
```
Install Development tools.
```
/bin/yum groupinstall -y -q "Development Tools" "Development Libraries"
```
Install Foreman.
Note:
Create a directory in /opt/installer and copy foreman_installer.pp in the same.
Create directories as /opt/installer/files/foreman and copy all the files with the same name in it. Location if files: (https://github.com/olindata/opstheater-oscar/tree/master/files/foreman)
Create directories as /opt/installer/files/smart-proxy and copy all the files with the same name in it. Location if files: (https://github.com/olindata/opstheater-oscar/tree/master/files/smart-proxy)
Edit all file in /opt/installer/files to reflect correct name eg: puppet.olindata.com
`/opt/puppetlabs/bin/puppet apply /opt/installer/foreman_installation.pp`
Set Foreman report
```
/opt/puppetlabs/bin/puppet apply -e 'ini_setting { "set foreman report ": ensure => present, section => "main", value => "log,foreman", path => "/etc/puppetlabs/puppet/puppet.conf", setting => "reports" }'
```
Install bundler
```
/usr/bin/gem install bundler --no-ri --no-rdoc --quiet --no-verbose
cd /opt/foreman; /usr/local/bin/bundle install --without mysql2 sqlite test --path vendor --quiet
cd /opt/foreman; /usr/local/bin/bundle update foreman_default_hostgroup
```
After this edit /opt/foreman/Gemfile.lock search for rake and change the version from 11.0.1 to 10.5.0.
Migrate the DB
```
RAILS_ENV=production bundle exec rake db:migrate --quiet
RAILS_ENV=production bundle exec rake assets:precompile locale:pack apipie:cache --quiet
```
Grep the Credentials
RAILS_ENV=production bundle exec rake db:seed|grep "Login"
Eg:
Login credentials: admin / e3iHDhY7QCiwr47n
Smart Proxy Bundle install
cd /opt/smart-proxy; /usr/local/bin/bundle install --without development test --path vendor --quiet
Above step will install rake 11.0.1. Edit Gemlock.file and change value for 'rake' from 11.0.1 to 10.5.0 and run #bundle install
cd /opt/smart-proxy; /usr/local/bin/bundle update rubocop
Run Foreman Post Install
Copy foreman_post_install.pp from (https://github.com/olindata/opstheater-oscar/blob/master/manifests/foreman_post_install.pp) to /opt/installer/
/opt/puppetlabs/bin/puppet apply /opt/installer/foreman_post_install.pp
Clear IPtables
iptables -F
Add Smart Proxy
/bin/curl -k -s -u admin:PASSWORD_FROM_PREVIOUS_COMMAND -H "Accept: version=2,application/json" -H "Content-Type: application/json" -X POST -d '{ "name": "foreman", "url": "https://puppet.olindata.com:8443" } ' http://puppet.olindata.com:3000/api/smart_proxies
/bin/curl -k -s -u admin:PASSWORD_FROM_PREVIOUS_COMMAND -H "Accept: version=2,application/json" -H "Content-Type: application/json" -X POST -d '{ }' http://puppet.olindata.com:3000/api/smart_proxies/1/import_puppetclasses
Expected Output of above 2 commands:
[root@puppet installer]# /bin/curl -k -s -u admin:e3iHDhY7QCiwr47n -H "Accept: version=2,application/json" -H "Content-Type: application/json" -X POST -d '{ }' http://puppet.olindata.com:3000/api/smart_proxies/1/import_puppetclasses
{
"message": "Successfully updated environment and puppetclasses from the on-disk puppet installation",
"environments_with_new_puppetclasses": 1,
"environments_updated_puppetclasses": 0,
"environments_obsolete": 0,
"results": [{"name":"production","actions":["new"],"new_puppetclasses":["ruby::params","ruby::dev","ruby::gemrc","ruby","ruby::config","gcc::params","gcc","stdlib","stdlib::stages","apt::params","apt","apt::backports","apt::update","puppetdb::database::postgresql","puppetdb::params","puppetdb::globals","puppetdb::master::storeconfigs","puppetdb::master::config","puppetdb::master::report_processor","puppetdb::master::puppetdb_conf","puppetdb::master::routes","puppetdb::server","puppetdb","puppetdb::server::firewall","puppetdb::server::validate_read_db","puppetdb::server::read_database","puppetdb::server::global","puppetdb::server::database","puppetdb::server::jetty","puppetdb::server::puppetdb","puppetdb::server::command_processing","puppetdb::server::validate_db","r10k::params","r10k::postrun_command","r10k::webhook","r10k::mcollective::application","r10k","r10k::install","r10k::mcollective","r10k::config","r10k::install::bundle","r10k::install::pe_gem","r10k::install::puppet_gem","r10k::install::gem","r10k::install::portage","r10k::webhook::package","r10k::webhook::config","r10k::prerun_command","postgresql::params","postgresql::globals","postgresql::repo::apt_postgresql_org","postgresql::repo::yum_postgresql_org","postgresql::client","postgresql::server","postgresql::repo","postgresql::server::initdb","postgresql::server::service","postgresql::server::reload","postgresql::server::plpython","postgresql::server::install","postgresql::server::plperl","postgresql::server::config","postgresql::server::passwd","postgresql::server::postgis","postgresql::server::contrib","postgresql::lib::python","postgresql::lib::devel","postgresql::lib::java","postgresql::lib::perl","postgresql::lib::docs","supervisord::params","supervisord::service","supervisord::reload","supervisord","supervisord::install","supervisord::config","supervisord::pip","git","git::subtree","git::gitosis","portage::params","portage","portage::install","make::params","make","make::install","firewall::params","firewall::linux","firewall","firewall::linux::debian","firewall::linux::redhat","firewall::linux::archlinux","firewall::linux::gentoo"]}]
}
[root@puppet installer]# /bin/curl -k -s -u admin:e3iHDhY7QCiwr47n -H "Accept: version=2,application/json" -H "Content-Type: application/json" -X POST -d '{ "name": "OpsTheater Infra", "environment_id": "1", "puppet_ca_proxy_id": "1", "puppet_proxy_id": "1" } ' http://puppet.olindata.com:3000/api/hostgroups
{"subnet_id":null,"subnet_name":null,"operatingsystem_id":null,"operatingsystem_name":null,"domain_id":null,"domain_name":null,"environment_id":1,"environment_name":"production","compute_profile_id":null,"compute_profile_name":null,"ancestry":null,"puppet_proxy_id":1,"puppet_ca_proxy_id":1,"ptable_id":null,"ptable_name":null,"medium_id":null,"medium_name":null,"architecture_id":null,"architecture_name":null,"realm_id":null,"realm_name":null,"created_at":"2016-03-09T11:48:29Z","updated_at":"2016-03-09T11:48:29Z","id":1,"name":"OpsTheater Infra","title":"OpsTheater Infra","parameters":[],"template_combinations":[],"puppetclasses":[],"config_groups":[],"all_puppetclasses":[]}
Restart PuppetServer
/bin/systemctl restart puppetserver
Running r10K
Download Opstheater-Control.tar.gz to the local server.
Untar opstehater-control in /root.
-Take the backup of existing production enviornment. /etc/puppetlabs/code/production as /etc/puppetlabs/code/production_backup
-Move /root/opstheater-control to /etc/puppetlabs/code/production
Install r10K
-/opt/puppetlabs/bin/puppetserver gem install r10k
-gem install 10k
Execute r10k from /etc/puppetlabs/code/production
-r10k puppetfile install -v
THis will install all the required modules mentioned in Puppetfile in current directory
Setup Hiera
/opt/puppetlabs/bin/puppet config set hiera_config /etc/puppetlabs/code/environments/production/hiera.yaml
Restart PuppetServer
/bin/systemctl restart puppetserver
Update Foreman puppet environments
/bin/curl -k -s -u admin:PASSWORD_FROM_PREVIOUS_COMMAND -H "Accept: version=2,application/json" -H "Content-Type: application/json" -X POST -d '{ }' http://puppet.olindata.com:3000/api/smart_proxies/1/import_puppetclasses
Run puppet agent and then stop puppet service
/opt/puppetlabs/bin/puppet agent -t || true
## GitLab Server Setup.
Configure repo and install puppet.
```
/bin/yum install -y epel-release http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server puppet.olindata.com
```
Configure/adjust Master
Ensure correct filename ymal file exists in bellow locations:
```
/etc/puppetlabs/code/environments/production/hieradata/10.fqdn
```
Once this is done. Puppet agent.
```
/opt/puppetlabs/bin/puppet agent -t
```
## Elastic Server Setup
Configure repo and install puppet.
```
/bin/yum install -y epel-release http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server puppet.olindata.com
```
Configure/adjust Master
Ensure correct filename ymal file exists in bellow locations:
```
/etc/puppetlabs/code/environments/production/hieradata/10.fqdn
```
Merge elastic.olindata.com.yaml and logstash.olindata.vm.yaml As we have single server or both application.
(Remember to edit the file and remove extra --- and change the server name to elastic from elasticsearch)
Logstash settings needs to be checked for public vs private ip
Ref: - "elastic.olindata.com:9200"
wget needs to be installed before running puppet agent -t.
Add DNS entry for logstash.olindata.com OR update /etc/filebeat/filebeat.yml to talk to elastic.olindata.com
update site.pp as bellow:
```
node 'elastic.olindata.com' {
include opstheater::role::elastic::server
include opstheater::role::logstash::server
}
```
Run Puppet agent.
## MYSQL server Setup
Configure repo and install puppet.
```
/bin/yum install -y epel-release http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server puppet.olindata.com
```
Configure/adjust Master
update /etc/puppetlabs/code/environment/production/hieradata/60-.opstheater.yaml to reflect myqsl server ip.
```
##########################
## MySQL related settings
##########################
## MySQL related settings
# Variable: opstheater::mysql::fqdn
# Description:
# Default value: "mysql.%{hiera('opstheater::domain')}"
'opstheater::mysql::fqdn': "mysql.%{hiera('opstheater::domain')}"
# Variable: opstheater::mysql::ipaddress
# Description:
# Default value: '10.20.1.60'
'opstheater::mysql::ipaddress': '10.129.2.113'
# Variable: opstheater::mysql::whitelist_range
# Description:
# Default value: '10.20.1.%'
'opstheater::mysql::whitelist_range': '10.129.%'
```
Run Puppet Agent
This will throw error for the 1st time as below but on the second run it works fine.
```
nfo: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]: Filebucketed /etc/my.cnf to puppet with sum 80e1eb23d5fbd77fc0ff681b0f0df297
Notice: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]/content: content changed '{md5}80e1eb23d5fbd77fc0ff681b0f0df297' to '{md5}3ab13aa54c001fe3966a08ae49c5517d'
Notice: Disabling SSL is evil! You should never ever do this except if you are forced to use a mysql version compiled without SSL support
Notice: /Stage[main]/Mysql::Server::Config/Notify[ssl-disable]/message: defined 'message' as 'Disabling SSL is evil! You should never ever do this except if you are forced to use a mysql version compiled without SSL support'
Error: Could not start Service[mysqld]: Execution of '/usr/bin/systemctl start mysqld' returned 1: Job for mysqld.service failed because the control process exited with error code. See "systemctl status mysqld.service" and "journalctl -xe" for details.
Error: /Stage[main]/Mysql::Server::Service/Service[mysqld]/ensure: change from stopped to running failed: Could not start Service[mysqld]: Execution of '/usr/bin/systemctl start mysqld' returned 1: Job for mysqld.service failed because the control process exited with error code. See "systemctl status mysqld.service" and "journalctl -xe" for details.
Notice: /Stage[main]/Opstheater::Profile::Mysql/File[/var/log/mysql]/ensure: created
Notice: /Stage[main]/Mysql::Server::Service/File[/var/log/mysql/error.log]/ensure: created
Info: Class[Mysql::Server::Service]: Unscheduling all events on Class[Mysql::Server::Service]
Notice: /Stage[main]/Mysql::Server::Root_password/Exec[remove install pass]: Dependency Service[mysqld] has failures: true
Warning: /Stage[main]/Mysql::Server::Root_password/Exec[remove install pass]: Skipping because of failed dependencies
Info: Creating state file /opt/puppetlabs/puppet/cache/state/state.yaml
Error: Failed to apply catalog: Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
```
## Icinga Server Setup
Configure repo and install puppet.
```
/bin/yum install -y epel-release http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server puppet.olindata.com
```
Configure/adjust Master
Update /etc/puppetlabs/code/environment/production/hieradata/60-.opstheater.yaml in icinga section to reflect correct name of icinga on the below line.
Also update the ipaddress to the public ipaddress.
'opstheater::icinga::fqdn': "icinga.%{hiera('opstheater::domain')}"
'opstheater::icinga::ipaddress': '37.139.31.34'
update site.pp to reflect correct name.
```
# runs standalone monitoring setup with icinga2 and icinga web2
node 'icinga.olindata.com' {
include opstheater::role::monitoring::standalone
}
```
Run puppet agent
# Installation
1. Requirements gathering phase
1.1 which tools do they want to use of the OpsTheater stack
1.2 hostnames
Recommended…
* master.opstheater.companyname.xxx (puppetmaster / foreman (if foss))
* monitoring.opstheater.companyname.xxx (icinga)
* logging.opstheater.companyname.xxx (kibana / other techs…?)
* code.opstheater.companyname.xxx (gitlab / mattermost)
1.3 resource allocation (either the default scheme or custom if need be)
1.4 HTTPS Yes or No?
If yes, need certificates for above hostnames
1.5 users (name, email address), groups
We need to create a sheet of sorts to collect what types of users and what types of access to be given eg:
Foreman users
Puppet users
Shell users (admin access)
Gitlab / Mattermost users
Icinga users
1.6 SMTP Relay information for our stack being able to send emails
SMTP Server Hostname
SMTP Server Port
Authenticated? If yes...
username:
Password:
Uses TLS?
Uses StartTLS?
1.7 any custom requirements
1.8 any needed migrations from other tools
1.9 Choice of provider(s) (cloud / physical)
2. deploy vms/physical nodes
3. Installation of master server (FOSS or not)
More details…
4. Customization of client-specific opstheater-control repository to include client-specific configuration, SSL certificates, URLs, SMTP provider, etc.
details...
5. Installation of requested OpsTheater-provided servers & services
master.opstheater.companyname.xxx (puppetmaster / foreman (if foss))
monitoring.opstheater.companyname.xxx (icinga)
logging.opstheater.companyname.xxx server (kibana / other techs…?)
code.opstheater.companyname.xxx server (gitlab / mattermost)
others...
6. Any manual configuration of OpsTheater servers not yet automated
Currently gitlab / mattermost integration requires a bit of manual attention post-install including…
Creating opstheater-control repository
Pointing the puppetmaster’s code source to the opstheater-control repository on their gitlab installation
Creating demo repository from git@github.com:olindata/sample-ruby-project.git
Setup mattermost
Enable mattermost team creation
Log into mattermost with root user
Create a team
Make the team a public team
disable team creation
TEST IT :P
Create integration
Copy/paste integration URL into gitlab for build notifications
7. (per-request) Client-specific configuration pre-discussed, such as setting up foreman to be able to deploy specific server types.
8. Creation of requested users in the various systems per-requested by the client.
## to be added to opstheater-control
In order to make deploys easier and upgrades also, we (W+F) propose something along the lines of this:
another hiera level 70.opstheater_custom
rename 60.opstheater to 60.opstheater_defaults
create besides the role and profile module also a module for each client that lives locally in their gitlab instance. We add it to their puppetfile so it gets deployed nicely
from 70.opstheater_custom we can have keys refer to a path in the client specific module
# Release process
1. On the 23rd of each month, we feature-freeze opstheater-control and freeze versions of included tools
2. create a branch pre-release off of that version?
3. list out current stable versions of the tools included in opstheater
4. We test upgrade from previous stable version
5. We test fresh deploys
6. We incorporate any extra changes only if they are otherwise breaking
7. We create release notes
8. on the 28th of each month, we release the new version and help clients with support contracts upgrade
9. a tarball of the pre-release branch named after whatever version tag it then has
For a log of 1.4 release process see: https://github.com/olindata/opstheater-control/issues/55
This borrows heavily and is modeled after: http://doc.gitlab.com/ce/release/monthly.html
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment