Commit a442cd1c authored by Walter Heck's avatar Walter Heck

Create the 1.6 release documentation directory

parent f1854340
Pipeline #104 skipped
# Prerequisites
* Make sure you have installed the following on your host machine:
* virtualbox (http://virtualbox.org)
* vagrant (https://www.vagrantup.com/)
* vagrant plugin oscar (https://github.com/oscar-stack/oscar)
You can install oscar plugin for vagrant with the following command:
```
vagrant plugin install oscar
```
Please note that you need to run vagrant commands from the location of your Vagrantfile. In this case location of Vagrantfile is : `opstheater/deploy/vagrant-oscar/`
## Puppet Enterprise Prerequisites
If you wish to use puppet enterprise instead of opensource for development, you need to make sure that the vagrant oscar plugin knows where to find the PE installation:
* Download the Puppet enterprise installer you will use at least for the master (by default this is PE 2015.2.2 on CentOS 7):
* PE 2015.2.0:
* https://s3.amazonaws.com/pe-builds/released/2015.2.0/puppet-enterprise-2015.2.0-ubuntu-14.04-amd64.tar.gz
* https://s3.amazonaws.com/pe-builds/released/2015.2.0/puppet-enterprise-2015.2.0-el-7-x86_64.tar.gz
* https://s3.amazonaws.com/pe-builds/released/2015.2.0/puppet-enterprise-2015.2.0-el-6-x86_64.tar.gz
* PE 2015.2.2:
* https://s3.amazonaws.com/pe-builds/released/2015.2.2/puppet-enterprise-2015.2.2-ubuntu-14.04-amd64.tar.gz
* https://s3.amazonaws.com/pe-builds/released/2015.2.2/puppet-enterprise-2015.2.2-el-7-x86_64.tar.gz
* https://s3.amazonaws.com/pe-builds/released/2015.2.2/puppet-enterprise-2015.2.2-el-6-x86_64.tar.gz
* PE 2015.3.0:
* https://s3.amazonaws.com/pe-builds/released/2015.3.0/puppet-enterprise-2015.3.0-el-7-x86_64.tar.gz
Use the pe-build plugin that comes with oscar to place the downloaded puppet enterprise installer in the correct location on your host.
`vagrant pe-build copy /path/to/puppet-enterprise-2015.2.0-el-7-x86_64.tar.gz`
Make sure to update the version of PE you are using in the `opstheater/deploy/vagrant-oscar/config/pe_build.yaml` file. It will determine which file in the pe_builds directory oscar searches for.
# Setting up opstheater in an oscar environment
* Clone the opstheater repository on your host machine:
```
git clone git@gitlab.olindata.com:opstheater/opstheater.git
```
* Bring up the puppet master (This may take a looong time!) vm and log into it:
```
vagrant up master
vagrant ssh master
sudo su -
```
* You might also want to install r10k or puppet-librarian on your host machine and install the modules the opstheater repo uses on your host so you can browse through them.
TODO: Add instructions for r10k setup
* Now bring up one or more agents with oscar on your host machine
```
vagrant up gitlab
```
# Installation
TODO: this is a work in progress, it is not complete and might not be fully up to date
1. Requirements gathering phase
1.1 which tools do they want to use of the OpsTheater stack
1.2 hostnames
Recommended…
* puppet.example.com (puppetmaster / foreman (if foss))
* icinga.example.com (icinga + icinga web)
* elastic.example.com (kibana / logstash / elasticsearch / influxdb / grafana)
* gitlab.example.com (gitlab / mattermost)
* mysql.example.com (mysql)
1.3 resource allocation (either the default scheme or custom if need be)
1.4 HTTPS Yes or No?
If yes, need certificates for above hostnames
1.5 users (name, email address), groups
We need to create a sheet of sorts to collect what types of users and what types of access to be given eg:
Foreman users
Puppet users
Shell users (admin access)
Gitlab / Mattermost users
Icinga users
1.6 SMTP Relay information for our stack being able to send emails
SMTP Server Hostname
SMTP Server Port
Authenticated? If yes...
username:
Password:
Uses TLS?
Uses StartTLS?
1.7 any custom requirements
1.8 any needed migrations from other tools
1.9 Choice of provider(s) (cloud / physical)
2. deploy vms/physical nodes
3. Installation of master server (FOSS or not)
More details…
4. Customization of client-specific opstheater-control repository to include client-specific configuration, SSL certificates, URLs, SMTP provider, etc.
details...
5. Installation of requested OpsTheater-provided servers & services
master.opstheater.companyname.xxx (puppetmaster / foreman (if foss))
monitoring.opstheater.companyname.xxx (icinga)
logging.opstheater.companyname.xxx server (kibana / other techs…?)
code.opstheater.companyname.xxx server (gitlab / mattermost)
others...
6. Any manual configuration of OpsTheater servers not yet automated
7. (per-request) Client-specific configuration pre-discussed, such as setting up foreman to be able to deploy specific server types.
8. Creation of requested users in the various systems per-requested by the client.
## Elastic Server Setup
1. Configure repo and install puppet.
```
/bin/yum install -y epel-release http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server puppet.olindata.com
```
2. Configure/adjust Master
Ensure correct filename ymal file exists in bellow locations:
```
/etc/puppetlabs/code/environments/production/hieradata/10.fqdn
```
Merge elastic.olindata.com.yaml and logstash.olindata.vm.yaml As we have single server or both application.
(Remember to edit the file and remove extra --- and change the server name to elastic from elasticsearch)
Logstash settings needs to be checked for public vs private ip
Ref: - "elastic.olindata.com:9200"
3. wget needs to be installed before running puppet agent -t.
4. Add DNS entry for logstash.olindata.com OR update /etc/filebeat/filebeat.yml to talk to elastic.olindata.com
5. update site.pp as bellow:
```
node 'elastic.olindata.com' {
include opstheater::role::elastic::server
include opstheater::role::logstash::server
}
```
6. Run Puppet agent.
## FOSS Master for OpsTheater
*Note*: The process for this node is currently much longer then it should be since the foreman doesn't officially support Puppet 4 yet. Hence we compile from source in order to make things work. This will be changed as soon as upstream foreman supports Puppet 4 properly.
Disable firewalld ************************ Enable firewalld again once installation is complete.
```bash
/bin/systemctl stop firewalld
/bin/systemctl disable firewalld
```
Install puppet server
```
/bin/yum install -y -q https://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm epel-release
/bin/yum install -y -q puppetserver
```
Install puppet modules
```
/opt/puppetlabs/bin/puppet module install puppetlabs/puppetdb #v5.1.1
/opt/puppetlabs/bin/puppet module install puppetlabs/vcsrepo #v1.3.2
/opt/puppetlabs/bin/puppet module install zack/r10k #v3.2.0
/opt/puppetlabs/bin/puppet module install abrader/gms #v1.0.2
/opt/puppetlabs/bin/puppet module install ajcrowe/supervisord --ignore-dependencies #v0.6.0
```
SetHostname, configure autosign cert, enable puppetserver and start it.
```
/bin/echo '==> Set puppetserver hostname'
/opt/puppetlabs/bin/puppet apply -e 'ini_setting { "set master hostname": ensure => present, section => "main", value => $::fqdn, path => "/etc/puppetlabs/puppet/puppet.conf", setting => "server" }'
/bin/echo '==> Set puppetserver autosign'
/opt/puppetlabs/bin/puppet apply -e 'file { "/etc/puppetlabs/puppet/autosign.conf": ensure => file, content => "$::fqdn\n*\n", }'
/bin/touch /etc/puppetlabs/code/environments/production/manifests/site.pp
/opt/puppetlabs/bin/puppet resource service puppetserver enable=true
/bin/systemctl start puppetserver
```
Install Development tools.
```
/bin/yum groupinstall -y -q "Development Tools" "Development Libraries"
```
Install Foreman.
Note:
Create a directory in /opt/installer and copy foreman_installer.pp in the same.
Create directories as /opt/installer/files/foreman and copy all the files with the same name in it. Location if files: (https://github.com/olindata/opstheater-oscar/tree/master/files/foreman)
Create directories as /opt/installer/files/smart-proxy and copy all the files with the same name in it. Location if files: (https://github.com/olindata/opstheater-oscar/tree/master/files/smart-proxy)
Edit all file in /opt/installer/files to reflect correct name eg: puppet.olindata.com
`/opt/puppetlabs/bin/puppet apply /opt/installer/foreman_installation.pp`
Set Foreman report
```
/opt/puppetlabs/bin/puppet apply -e 'ini_setting { "set foreman report ": ensure => present, section => "main", value => "log,foreman", path => "/etc/puppetlabs/puppet/puppet.conf", setting => "reports" }'
```
Install bundler
```
/usr/bin/gem install bundler --no-ri --no-rdoc --quiet --no-verbose
cd /opt/foreman; /usr/local/bin/bundle install --without mysql2 sqlite test --path vendor --quiet
cd /opt/foreman; /usr/local/bin/bundle update foreman_default_hostgroup
```
After this edit /opt/foreman/Gemfile.lock search for rake and change the version from 11.0.1 to 10.5.0.
Migrate the DB
```
RAILS_ENV=production bundle exec rake db:migrate --quiet
RAILS_ENV=production bundle exec rake assets:precompile locale:pack apipie:cache --quiet
```
Grep the Credentials
RAILS_ENV=production bundle exec rake db:seed|grep "Login"
Eg:
Login credentials: admin / e3iHDhY7QCiwr47n
Smart Proxy Bundle install
cd /opt/smart-proxy; /usr/local/bin/bundle install --without development test --path vendor --quiet
Above step will install rake 11.0.1. Edit Gemlock.file and change value for 'rake' from 11.0.1 to 10.5.0 and run #bundle install
cd /opt/smart-proxy; /usr/local/bin/bundle update rubocop
Run Foreman Post Install
Copy foreman_post_install.pp from (https://github.com/olindata/opstheater-oscar/blob/master/manifests/foreman_post_install.pp) to /opt/installer/
/opt/puppetlabs/bin/puppet apply /opt/installer/foreman_post_install.pp
Clear IPtables
iptables -F
Add Smart Proxy
/bin/curl -k -s -u admin:PASSWORD_FROM_PREVIOUS_COMMAND -H "Accept: version=2,application/json" -H "Content-Type: application/json" -X POST -d '{ "name": "foreman", "url": "https://puppet.olindata.com:8443" } ' http://puppet.olindata.com:3000/api/smart_proxies
/bin/curl -k -s -u admin:PASSWORD_FROM_PREVIOUS_COMMAND -H "Accept: version=2,application/json" -H "Content-Type: application/json" -X POST -d '{ }' http://puppet.olindata.com:3000/api/smart_proxies/1/import_puppetclasses
Expected Output of above 2 commands:
[root@puppet installer]# /bin/curl -k -s -u admin:e3iHDhY7QCiwr47n -H "Accept: version=2,application/json" -H "Content-Type: application/json" -X POST -d '{ }' http://puppet.olindata.com:3000/api/smart_proxies/1/import_puppetclasses
{
"message": "Successfully updated environment and puppetclasses from the on-disk puppet installation",
"environments_with_new_puppetclasses": 1,
"environments_updated_puppetclasses": 0,
"environments_obsolete": 0,
"results": [{"name":"production","actions":["new"],"new_puppetclasses":["ruby::params","ruby::dev","ruby::gemrc","ruby","ruby::config","gcc::params","gcc","stdlib","stdlib::stages","apt::params","apt","apt::backports","apt::update","puppetdb::database::postgresql","puppetdb::params","puppetdb::globals","puppetdb::master::storeconfigs","puppetdb::master::config","puppetdb::master::report_processor","puppetdb::master::puppetdb_conf","puppetdb::master::routes","puppetdb::server","puppetdb","puppetdb::server::firewall","puppetdb::server::validate_read_db","puppetdb::server::read_database","puppetdb::server::global","puppetdb::server::database","puppetdb::server::jetty","puppetdb::server::puppetdb","puppetdb::server::command_processing","puppetdb::server::validate_db","r10k::params","r10k::postrun_command","r10k::webhook","r10k::mcollective::application","r10k","r10k::install","r10k::mcollective","r10k::config","r10k::install::bundle","r10k::install::pe_gem","r10k::install::puppet_gem","r10k::install::gem","r10k::install::portage","r10k::webhook::package","r10k::webhook::config","r10k::prerun_command","postgresql::params","postgresql::globals","postgresql::repo::apt_postgresql_org","postgresql::repo::yum_postgresql_org","postgresql::client","postgresql::server","postgresql::repo","postgresql::server::initdb","postgresql::server::service","postgresql::server::reload","postgresql::server::plpython","postgresql::server::install","postgresql::server::plperl","postgresql::server::config","postgresql::server::passwd","postgresql::server::postgis","postgresql::server::contrib","postgresql::lib::python","postgresql::lib::devel","postgresql::lib::java","postgresql::lib::perl","postgresql::lib::docs","supervisord::params","supervisord::service","supervisord::reload","supervisord","supervisord::install","supervisord::config","supervisord::pip","git","git::subtree","git::gitosis","portage::params","portage","portage::install","make::params","make","make::install","firewall::params","firewall::linux","firewall","firewall::linux::debian","firewall::linux::redhat","firewall::linux::archlinux","firewall::linux::gentoo"]}]
}
[root@puppet installer]# /bin/curl -k -s -u admin:e3iHDhY7QCiwr47n -H "Accept: version=2,application/json" -H "Content-Type: application/json" -X POST -d '{ "name": "OpsTheater Infra", "environment_id": "1", "puppet_ca_proxy_id": "1", "puppet_proxy_id": "1" } ' http://puppet.olindata.com:3000/api/hostgroups
{"subnet_id":null,"subnet_name":null,"operatingsystem_id":null,"operatingsystem_name":null,"domain_id":null,"domain_name":null,"environment_id":1,"environment_name":"production","compute_profile_id":null,"compute_profile_name":null,"ancestry":null,"puppet_proxy_id":1,"puppet_ca_proxy_id":1,"ptable_id":null,"ptable_name":null,"medium_id":null,"medium_name":null,"architecture_id":null,"architecture_name":null,"realm_id":null,"realm_name":null,"created_at":"2016-03-09T11:48:29Z","updated_at":"2016-03-09T11:48:29Z","id":1,"name":"OpsTheater Infra","title":"OpsTheater Infra","parameters":[],"template_combinations":[],"puppetclasses":[],"config_groups":[],"all_puppetclasses":[]}
Restart PuppetServer
/bin/systemctl restart puppetserver
Running r10K
Download Opstheater-Control.tar.gz to the local server.
Untar opstehater-control in /root.
-Take the backup of existing production enviornment. /etc/puppetlabs/code/production as /etc/puppetlabs/code/production_backup
-Move /root/opstheater-control to /etc/puppetlabs/code/production
Install r10K
-/opt/puppetlabs/bin/puppetserver gem install r10k
-gem install 10k
Execute r10k from /etc/puppetlabs/code/production
-r10k puppetfile install -v
THis will install all the required modules mentioned in Puppetfile in current directory
Setup Hiera
/opt/puppetlabs/bin/puppet config set hiera_config /etc/puppetlabs/code/environments/production/hiera.yaml
Restart PuppetServer
/bin/systemctl restart puppetserver
Update Foreman puppet environments
/bin/curl -k -s -u admin:PASSWORD_FROM_PREVIOUS_COMMAND -H "Accept: version=2,application/json" -H "Content-Type: application/json" -X POST -d '{ }' http://puppet.olindata.com:3000/api/smart_proxies/1/import_puppetclasses
Run puppet agent and then stop puppet service
/opt/puppetlabs/bin/puppet agent -t || true
## GitLab Server Setup.
1. Configure repo and install puppet.
```
/bin/yum install -y epel-release http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server puppet.olindata.com
```
2. Configure/adjust Master
Ensure correct filename ymal file exists in bellow locations:
```
/etc/puppetlabs/code/environments/production/hieradata/10.fqdn
```
3. Once this is done. Puppet agent.
```
/opt/puppetlabs/bin/puppet agent -t
```
4. Content Setup
Currently gitlab / mattermost integration requires a bit of manual attention post-install including…
Creating opstheater-control repository
Pointing the puppetmaster’s code source to the opstheater-control repository on their gitlab installation
Creating demo repository from git@github.com:olindata/sample-ruby-project.git
Setup mattermost
Enable mattermost team creation
Log into mattermost with root user
Create a team
Make the team a public team
disable team creation
TEST IT :P
Create integration
Copy/paste integration URL into gitlab for build notifications
# Icinga Server Setup
1. Configure repo and install puppet.
```
/bin/yum install -y epel-release http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server puppet.olindata.com
```
2. Configure/adjust Master
* Update `/etc/puppetlabs/code/environment/production/hieradata/60-.opstheater.yaml` in icinga section to reflect correct name of icinga on the below line.
* Also update the ipaddress to the public ipaddress.
```
'opstheater::icinga::fqdn': "icinga.%{hiera('opstheater::domain')}"
'opstheater::icinga::ipaddress': '37.139.31.34'
```
* update site.pp to reflect correct name.
```
# runs standalone monitoring setup with icinga2 and icinga web2
node 'icinga.olindata.com' {
include opstheater::role::monitoring::standalone
}
```
3. Run puppet agent
## MYSQL server Setup
1. Configure repo and install puppet.
```
/bin/yum install -y epel-release http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server puppet.olindata.com
```
2. Configure/adjust Master
update /etc/puppetlabs/code/environment/production/hieradata/60-.opstheater.yaml to reflect myqsl server ip.
```
##########################
## MySQL related settings
##########################
## MySQL related settings
# Variable: opstheater::mysql::fqdn
# Description:
# Default value: "mysql.%{hiera('opstheater::domain')}"
'opstheater::mysql::fqdn': "mysql.%{hiera('opstheater::domain')}"
# Variable: opstheater::mysql::ipaddress
# Description:
# Default value: '10.20.1.60'
'opstheater::mysql::ipaddress': '10.129.2.113'
# Variable: opstheater::mysql::whitelist_range
# Description:
# Default value: '10.20.1.%'
'opstheater::mysql::whitelist_range': '10.129.%'
```
3. Run Puppet Agent
This will throw error for the 1st time as below but on the second run it works fine.
```
nfo: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]: Filebucketed /etc/my.cnf to puppet with sum 80e1eb23d5fbd77fc0ff681b0f0df297
Notice: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]/content: content changed '{md5}80e1eb23d5fbd77fc0ff681b0f0df297' to '{md5}3ab13aa54c001fe3966a08ae49c5517d'
Notice: Disabling SSL is evil! You should never ever do this except if you are forced to use a mysql version compiled without SSL support
Notice: /Stage[main]/Mysql::Server::Config/Notify[ssl-disable]/message: defined 'message' as 'Disabling SSL is evil! You should never ever do this except if you are forced to use a mysql version compiled without SSL support'
Error: Could not start Service[mysqld]: Execution of '/usr/bin/systemctl start mysqld' returned 1: Job for mysqld.service failed because the control process exited with error code. See "systemctl status mysqld.service" and "journalctl -xe" for details.
Error: /Stage[main]/Mysql::Server::Service/Service[mysqld]/ensure: change from stopped to running failed: Could not start Service[mysqld]: Execution of '/usr/bin/systemctl start mysqld' returned 1: Job for mysqld.service failed because the control process exited with error code. See "systemctl status mysqld.service" and "journalctl -xe" for details.
Notice: /Stage[main]/Opstheater::Profile::Mysql/File[/var/log/mysql]/ensure: created
Notice: /Stage[main]/Mysql::Server::Service/File[/var/log/mysql/error.log]/ensure: created
Info: Class[Mysql::Server::Service]: Unscheduling all events on Class[Mysql::Server::Service]
Notice: /Stage[main]/Mysql::Server::Root_password/Exec[remove install pass]: Dependency Service[mysqld] has failures: true
Warning: /Stage[main]/Mysql::Server::Root_password/Exec[remove install pass]: Skipping because of failed dependencies
Info: Creating state file /opt/puppetlabs/puppet/cache/state/state.yaml
Error: Failed to apply catalog: Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
```
# Instructions for attaching an existing node to opstheater
# Instructions for attaching a new node to opstheater
## OpsTheater Release Checklist for Release 1.5.0
**Following checks need to be performed after every release :**
- Check if you can login into following using your web browser :
- [ ] Foreman
- [ ] Gitlab
- [ ] Mattermost
- [ ] Icinga
- [ ] Grafana
- [ ] Kibana
- [ ] Check if you are able to send message using Mattermost.
- [ ] Check if you are able to create a user in Foreman and can login with that user.
- [ ] Check if you can see the logs coming up for OpsTheater components in Icinga Web UI.
- [ ] Try to search for a host in Kibana and check if you are getting logs of that host.
**Release specific Checks :**
*These checks will change on every release and will be more specific for the new changes that we are introducing in this release.*
- [ ] Login to Grafana web UI and check if you can see different metrics in Grafana
- [ ] Login to Kibana web UI and try to search for logs having source as "/var/log/puppetlabs/puppetserver/puppetserver-access.log". Check if the value of field timestamp is same as the time given in message field
# Release process
For a log of 1.4 release process see: https://github.com/olindata/opstheater-control/issues/55
This borrows heavily and is modeled after: http://doc.gitlab.com/ce/release/monthly.html
**NOTE: This is a guide used by OlinData BV for handling releases of OpsTheater. As an end user you do not need to use this guide.**
# Release cycle
Problems notwithstanding we release a new version of OpsTheater every month on the first of the month. Our timing is based on and coincides with a number of the software packaged within OpsTheater. Security releases are published when needed. Our changelog for the release will contain changes, new features, and packaged software version information.
# Release timing
Monthly release, the 1st day of every month.
Patch release, for serious bugs or security problems.
# Monthly Release Internal Process
1. The process starts the 23rd of every month, in which we evaluate the stability and inclusion of newer versions of our included packages. The release manager decides on a version of all supporting packages and creates a ChangeLog to be included with the release and documented in this repository under releases/x.x.x_changelog.md
2. First thing that occurs the release manager opens a issue for the new release and to keep track of the release. You can refer this one for example : https://gitlab.olindata.com/opstheater/opstheater/issues/10
3. After this the release manager creates a release-x.x.x tag from the master branch. Typically monthly releases are sub-releases, so it should increment the Y in x.y.z, making 1.2.3 into 1.3.0. The final version octet is reserved for emergency bug/security patches, and the first octet is reserved for us to designate major evolutions in the software at our own discretion.
4. From this release branch, the release manager attempts a fresh installation of all of the currently supported services on a staging-ish environment, typically using our primary provider DigitalOcean or a local vagrant setup.
5. Form this local installation release manager will make a list out current stable versions of the tools included in OpsTheater.
6. If this installation happened without problem, then (ideally) a fresh installation is setup from the previous release, and an upgrade installation is performed and tested. This above process is expected to take 1-3 days depending on our level of automation and problems occurred during installation/upgrading/testing. Patches can be made to fix any release problems and committed directly to the release branch to be merged back into the master with a PR after released.
7. If the release testing will take more than 3 days, we evaluate the progress, changes, and situation and consider postponing deploy or proceeding with it.
8. Create release notes and a blog post for the olindata.com blog.
9. Once the upgrade installation is succeeded then release manager need to create a new directory in opstheater-docs reposiroty cloning previous release directory and need to update the documents in the new directory according to the new release for example release manager need to update upgrade.md with the steps that need to be performed for upgrade installation.
10. After this release manager need to update releasechecklist.md in current release branch of opstheater-docs. releasechecklist.md will contains the basic steps that release manager needs to perform once the upgrade is finised to make sure that all opstheater components are working fine.
11. Schedule tweets and a mail to the mailing list also.
12. After a release is tested for installation and upgrading, we upgrade our internal core OpsTheater installation, so we are consuming what we're creating. This needs to be done at least 4 days before the end of the month, so that we can have at least two working days with our platform upgraded to settle any potential longer-tailed issues with the upgrade. If any issues arise, the release manager can either opt to fix them and commit to the release branch, or to rollback the platform to the previous release depending on the severity and estimated time to fix.
13. When the first of the month rolls around, if there were no problems with the release, a release is created in github from the release branch, and any changes made are PR'd back to the master branch for approval by another core team member. This tag is not to be purged after the PR is merged to master, but to remain in git forever.
14. Now release manager can close the issue which he created for this new release.
15. At this point, we send an update notification email to any subscribers to OpsTheater mailers and begin the process to upgrade all of our OpsTheater installations that are paid subscribers. As part of the signup process with our customers, we decide on a day-of-the-week and time to perform maintenance on their OpsTheater. Following their requested schedule, we perform the necessary upgrade, and inform our customers of this 2 days before, and if there is no objection then also notify them as we begin the upgrade procedure, and after its completion along with a changelog of any platform changes/improvements/etc.
# OpsTheater 1.5.0 Upgrade instructions (from 1.4.0)
For this release OpsTheater components will need to be upgraded manually.
OpsTheater components you need to upgrade in this release:
* Gitlab will be upgraded from version 8.5 to 8.6
* Mattermost will be upgraded from version 2.0 to 2.1
* Percona server from 5.6.28 to 5.6.29
* Elasticsearch from 2.2. To 2.3
* Logstash stays the same
* Kibana stays the same
* Puppet from 4.3.2 to 4.4.0
* Foreman stays the same
* Icinga 2 from 2.4.3 to 2.4.4
* Icinga web 2 from 2.1.2 to 2.2.0
## Steps
1. Disable puppet on all OpsTheater nodes
---
Login to the following servers one by one and disable puppet agent by command `puppet agent --disable`
* puppet.example.com
* icinga.example.com
* mysql.example.com
* gitlab.example.com
* elastic.example.com
2. Upgrade MySQL
---
To upgrade Percona server from 5.6.28 to 5.6.29 login to mysql server and run the following command:
```bash
# if you want to update all packages (not just OpsTheater packages) feel free
# to run `yum update` without specifying packages
yum update icinga2 puppet-agent filebeat Percona-Server-server-56.x86_64
```
After that is done, check that the installed versions are correct:
```bash
[root@mysql ~]# rpm -qa | egrep -i 'icinga2|puppet-agent|filebeat|^percona-server' | sort
filebeat-1.2.0-1.x86_64
icinga2-2.4.4-1.el7.centos.x86_64
icinga2-bin-2.4.4-1.el7.centos.x86_64
icinga2-common-2.4.4-1.el7.centos.x86_64
Percona-Server-client-56-5.6.29-rel76.2.el7.x86_64
Percona-Server-server-56-5.6.29-rel76.2.el7.x86_64
Percona-Server-shared-56-5.6.29-rel76.2.el7.x86_64
puppet-agent-1.4.1-1.el7.x86_64
```
3. Upgrading elasticsearch
---
To upgrade Elasticsearch from 2.2 to 2.3 login to elasticsearch server and run the following commands:
```bash
# check the current versions of OpsTheater components
[root@elastic ~]# rpm -qa | egrep -i 'icinga2|puppet-agent|filebeat|elastic' | sort
elasticsearch-2.2.0-1.noarch
filebeat-1.1.1-1.x86_64
icinga2-2.4.3-1.el6.x86_64
icinga2-bin-2.4.3-1.el6.x86_64
icinga2-common-2.4.3-1.el6.x86_64
puppet-agent-1.3.5-1.el6.x86_64
```
```bash
# if you want to update all packages (not just OpsTheater packages) feel free
# to run `yum update` without specifying packages
yum update icinga2 puppet-agent filebeat elasticsearch
```
After that is done, check that the installed versions are correct:
```bash
[root@elastic ~]# rpm -qa | egrep -i 'icinga2|puppet-agent|filebeat|elastic' | sort
elasticsearch-2.3.0-1.noarch
filebeat-1.2.0-1.x86_64
icinga2-2.4.4-1.el6.x86_64
icinga2-bin-2.4.4-1.el6.x86_64
icinga2-common-2.4.4-1.el6.x86_64
puppet-agent-1.4.1-1.el6.x86_64
```
4. Upgrading icinga 2
---
To upgrade Icinga 2 from 2.4.3 to 2.4.4 and Icinga web 2 from 2.1.2 to 2.2.0 login to the elastic server and run the following commands:
```bash
[root@icinga ~]# rpm -qa | egrep -i 'icinga2|puppet-agent|filebeat|^icingaweb2-' | sort
filebeat-1.1.1-1.x86_64
icinga2-2.4.3-1.el7.centos.x86_64
icinga2-bin-2.4.3-1.el7.centos.x86_64
icinga2-common-2.4.3-1.el7.centos.x86_64
icinga2-ido-mysql-2.4.3-1.el7.centos.x86_64
icingaweb2-2.2.0-1.el7.centos.noarch
icingaweb2-common-2.2.0-1.el7.centos.noarch
icingaweb2-vendor-dompdf-0.6.2-1.el7.centos.noarch
icingaweb2-vendor-HTMLPurifier-4.7.0-1.el7.centos.noarch
icingaweb2-vendor-JShrink-1.0.1-1.el7.centos.noarch
icingaweb2-vendor-lessphp-0.4.0-1.el7.centos.noarch
icingaweb2-vendor-Parsedown-1.0.0-1.el7.centos.noarch
puppet-agent-1.3.5-1.el7.x86_64
```
```bash
# if you want to update all packages (not just OpsTheater packages) feel free
# to run `yum update` without specifying packages
yum update puppet-agent filebeat icinga2 icingaweb2
```
After that is done, check that the installed versions are correct:
```bash
[root@icinga ~]# rpm -qa | egrep -i 'icinga2|puppet-agent|filebeat|^icingaweb2-' | sort
filebeat-1.2.0-1.x86_64
icinga2-2.4.4-1.el7.centos.x86_64
icinga2-bin-2.4.4-1.el7.centos.x86_64
icinga2-common-2.4.4-1.el7.centos.x86_64
icinga2-ido-mysql-2.4.4-1.el7.centos.x86_64
icingaweb2-2.2.0-1.el7.centos.noarch
icingaweb2-common-2.2.0-1.el7.centos.noarch
icingaweb2-vendor-dompdf-0.6.2-1.el7.centos.noarch
icingaweb2-vendor-HTMLPurifier-4.7.0-1.el7.centos.noarch
icingaweb2-vendor-JShrink-1.0.1-1.el7.centos.noarch
icingaweb2-vendor-lessphp-0.4.0-1.el7.centos.noarch
icingaweb2-vendor-Parsedown-1.0.0-1.el7.centos.noarch
puppet-agent-1.4.1-1.el7.x86_64
```
5. Upgrading gitlab
---
Login on gitlab.example.com server do the following :
```bash
[root@gitlab ~]# rpm -qa | egrep -i 'icinga2|puppet-agent|filebeat|gitlab' | sort
filebeat-1.1.1-1.x86_64
gitlab-ce-8.5.5-ce.0.el7.x86_64
gitlab-ci-multi-runner-1.0.4-1.x86_64
icinga2-2.4.3-1.el7.centos.x86_64
icinga2-bin-2.4.3-1.el7.centos.x86_64
icinga2-common-2.4.3-1.el7.centos.x86_64
puppet-agent-1.3.5-1.el7.x86_64
```
Run following command to Upgrade Gitlab to 8.6 from 8.5 and Mattermost to 2.1 from 2.0
```bash
# if you want to update all packages (not just OpsTheater packages) feel free
# to run `yum update` without specifying packages
yum update puppet-agent filebeat icinga2 gitlab-ce
```
After that is done, check that the installed versions are correct:
```bash
[root@gitlab ~]# rpm -qa | egrep -i 'icinga2|puppet-agent|filebeat|gitlab' | sort
filebeat-1.2.0-1.x86_64
gitlab-ce-8.6.2-ce.0.el7.x86_64
gitlab-ci-multi-runner-1.0.4-1.x86_64
icinga2-2.4.4-1.el7.centos.x86_64
icinga2-bin-2.4.4-1.el7.centos.x86_64
icinga2-common-2.4.4-1.el7.centos.x86_64
puppet-agent-1.4.1-1.el7.x86_64
```
6. Upgrading puppetserver
---
Login on puppet.example.com server do the following :
```bash
[root@puppet ~]# rpm -qa | egrep -i 'icinga2|puppet|filebeat' | sort
filebeat-1.1.1-1.x86_64
icinga2-2.4.3-1.el7.centos.x86_64
icinga2-bin-2.4.3-1.el7.centos.x86_64
icinga2-common-2.4.3-1.el7.centos.x86_64
puppet-agent-1.3.5-1.el7.x86_64