Commit d419786e authored by Walter Heck's avatar Walter Heck

Initial 1.7 documentation

parent 7c29dbf8
# Prerequisites
* Make sure you have read
* Make sure you have installed the following on your host machine:
* virtualbox (
* vagrant (
* vagrant plugin oscar (
* make sure you have min. 6G RAM available to run the machine
You can install oscar plugin for vagrant with the following command:
`vagrant plugin install oscar`
Please note that you need to run vagrant commands from the location of your Vagrantfile. In this case location of Vagrantfile is : `opstheater/deploy/vagrant-oscar/`
## Puppet Enterprise Prerequisites
If you wish to use puppet enterprise instead of opensource for development, you need to make sure that the vagrant oscar plugin knows where to find the PE installation:
Use the pe-build plugin that comes with oscar to place the downloaded puppet enterprise installer in the correct location on your host.
`vagrant pe-build copy`
Make sure to update the version of PE you are using in the `opstheater/deploy/vagrant-oscar/config/pe_build.yaml` file. It will determine which file in the pe_builds directory oscar searches for.
# Setting up opstheater in an oscar environment
* Clone the opstheater repository on your host machine:
git clone
* Change to ‘develop’ branch to get the latest code:
git checkout -b develop origin/develop
cd opstheater/deploy/vagrant-oscar
* All the deployment files and configurations are in the `opstheater/deploy/vagrant-oscar` directory.
* Bring up the puppet master (This will take a looong time!) vm and log into it:
For Puppet Enterprise:
vagrant up master
vagrant ssh master
sudo su -
For Open Source Puppet:
vagrant up foss-master
vagrant ssh foss-master
sudo su -
At the very end of that process the `vagrant up` output displays the login details for foreman:
==> foss-master: Notice: Applied catalog in 0.25 seconds
==> foss-master: ==> Foreman URL: http://master.opstheater.vm:3000
==> foss-master: ==> Login credentials: admin / zndxYuJbDzasVCpv
Don’t forget to add master.opstheater.vm to your hosts file.
* Now bring up one or more machines with oscar on your host machine:
vagrant up mysql
vagrant up icinga2
vagrant up gitlab
vagrant up elasticsearch
to see which machines are available, simply do an `vagrant status`.
NOTE: you might have to run puppet a number of times on each node to make sure it fully puppetises itself. Some of the initial runs might have errors but they will go away as the puppet runs on all servers start happening.
* You might also want to install r10k or puppet-librarian on your host machine and install the modules the opstheater repo uses on your host so you can browse through them.
TODO: Add instructions for r10k setup
# Installation
TODO: this is a work in progress, it is not complete and might not be fully up to date
* Make sure you have read
1. Requirements gathering phase
1.1 which tools do you want to use of the OpsTheater stack
* Puppet? Open Source or Enterprise?
* The foreman?
* Grafana, Kibana, elasticsearch?
* Icinga?
* Gitlab, Mattermost?
1.2 hostnames
* (puppetmaster / foreman (if foss))
* (icinga + icinga web)
* (kibana / logstash / elasticsearch / influxdb / grafana)
* (gitlab / mattermost)
* (mysql)
1.3 resource allocation (either the default scheme or custom if need be)
1.4 HTTPS Yes or No?
If yes, need certificates for above hostnames
1.5 users (name, email address), groups
We need to create a sheet of sorts to collect what types of users and what types of access to be given eg:
Foreman users
Puppet users
Shell users (admin access)
Gitlab / Mattermost users
Icinga users
1.6 SMTP Relay information for our stack being able to send emails
SMTP Server Hostname
SMTP Server Port
Authenticated? If yes...
Uses TLS?
Uses StartTLS?
1.7 any custom requirements
1.8 any needed migrations from other tools
1.9 Choice of provider(s) (cloud / physical)
2. deploy vms/physical nodes
3. Installation of master server (FOSS or not)
More details…
4. Customization of client-specific opstheater-control repository to include client-specific configuration, SSL certificates, URLs, SMTP provider, etc.
5. Installation of requested OpsTheater-provided servers & services (puppetmaster / foreman (if foss)) (icinga) server (kibana / other techs…?) server (gitlab / mattermost)
6. Any manual configuration of OpsTheater servers not yet automated
7. (per-request) Client-specific configuration pre-discussed, such as setting up foreman to be able to deploy specific server types.
8. Creation of requested users in the various systems per-requested by the client.
## Elastic Server Setup
1. Configure repo and install puppet.
/bin/yum install -y epel-release
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server
2. Configure/adjust Master
Ensure correct filename ymal file exists in bellow locations:
Merge and logstash.olindata.vm.yaml As we have single server or both application.
(Remember to edit the file and remove extra --- and change the server name to elastic from elasticsearch)
Logstash settings needs to be checked for public vs private ip
Ref: - ""
3. wget needs to be installed before running puppet agent -t.
4. Add DNS entry for OR update /etc/filebeat/filebeat.yml to talk to
5. update site.pp as bellow:
node '' {
include opstheater::role::elastic::server
include opstheater::role::logstash::server
6. Run Puppet agent.
## FOSS Master for OpsTheater
*Note*: The process for this node is currently much longer then it should be since the foreman doesn't officially support Puppet 4 yet. Hence we compile from source in order to make things work. This will be changed as soon as upstream foreman supports Puppet 4 properly.
Disable firewalld ************************ Enable firewalld again once installation is complete.
/bin/systemctl stop firewalld
/bin/systemctl disable firewalld
Install puppet server
/bin/yum install -y -q epel-release
/bin/yum install -y -q puppetserver
Install puppet modules
/opt/puppetlabs/bin/puppet module install puppetlabs/puppetdb #v5.1.1
/opt/puppetlabs/bin/puppet module install puppetlabs/vcsrepo #v1.3.2
/opt/puppetlabs/bin/puppet module install zack/r10k #v3.2.0
/opt/puppetlabs/bin/puppet module install abrader/gms #v1.0.2
/opt/puppetlabs/bin/puppet module install ajcrowe/supervisord --ignore-dependencies #v0.6.0
SetHostname, configure autosign cert, enable puppetserver and start it.
/bin/echo '==> Set puppetserver hostname'
/opt/puppetlabs/bin/puppet apply -e 'ini_setting { "set master hostname": ensure => present, section => "main", value => $::fqdn, path => "/etc/puppetlabs/puppet/puppet.conf", setting => "server" }'
/bin/echo '==> Set puppetserver autosign'
/opt/puppetlabs/bin/puppet apply -e 'file { "/etc/puppetlabs/puppet/autosign.conf": ensure => file, content => "$::fqdn\n*\n", }'
/bin/touch /etc/puppetlabs/code/environments/production/manifests/site.pp
/opt/puppetlabs/bin/puppet resource service puppetserver enable=true
/bin/systemctl start puppetserver
Install Development tools.
/bin/yum groupinstall -y -q "Development Tools" "Development Libraries"
Install Foreman.
Create a directory in /opt/installer and copy foreman_installer.pp in the same.
Create directories as /opt/installer/files/foreman and copy all the files with the same name in it. Location if files: (
Create directories as /opt/installer/files/smart-proxy and copy all the files with the same name in it. Location if files: (
Edit all file in /opt/installer/files to reflect correct name eg:
`/opt/puppetlabs/bin/puppet apply /opt/installer/foreman_installation.pp`
Set Foreman report
/opt/puppetlabs/bin/puppet apply -e 'ini_setting { "set foreman report ": ensure => present, section => "main", value => "log,foreman", path => "/etc/puppetlabs/puppet/puppet.conf", setting => "reports" }'
Install bundler
/usr/bin/gem install bundler --no-ri --no-rdoc --quiet --no-verbose
cd /opt/foreman; /usr/local/bin/bundle install --without mysql2 sqlite test --path vendor --quiet
cd /opt/foreman; /usr/local/bin/bundle update foreman_default_hostgroup
After this edit /opt/foreman/Gemfile.lock search for rake and change the version from 11.0.1 to 10.5.0.
Migrate the DB
RAILS_ENV=production bundle exec rake db:migrate --quiet
RAILS_ENV=production bundle exec rake assets:precompile locale:pack apipie:cache --quiet
Grep the Credentials
RAILS_ENV=production bundle exec rake db:seed|grep "Login"
Login credentials: admin / e3iHDhY7QCiwr47n
Smart Proxy Bundle install
cd /opt/smart-proxy; /usr/local/bin/bundle install --without development test --path vendor --quiet
Above step will install rake 11.0.1. Edit Gemlock.file and change value for 'rake' from 11.0.1 to 10.5.0 and run #bundle install
cd /opt/smart-proxy; /usr/local/bin/bundle update rubocop
Run Foreman Post Install
Copy foreman_post_install.pp from ( to /opt/installer/
/opt/puppetlabs/bin/puppet apply /opt/installer/foreman_post_install.pp
Clear IPtables
iptables -F
Add Smart Proxy
/bin/curl -k -s -u admin:PASSWORD_FROM_PREVIOUS_COMMAND -H "Accept: version=2,application/json" -H "Content-Type: application/json" -X POST -d '{ "name": "foreman", "url": "" } '
/bin/curl -k -s -u admin:PASSWORD_FROM_PREVIOUS_COMMAND -H "Accept: version=2,application/json" -H "Content-Type: application/json" -X POST -d '{ }'
Expected Output of above 2 commands:
[root@puppet installer]# /bin/curl -k -s -u admin:e3iHDhY7QCiwr47n -H "Accept: version=2,application/json" -H "Content-Type: application/json" -X POST -d '{ }'
"message": "Successfully updated environment and puppetclasses from the on-disk puppet installation",
"environments_with_new_puppetclasses": 1,
"environments_updated_puppetclasses": 0,
"environments_obsolete": 0,
"results": [{"name":"production","actions":["new"],"new_puppetclasses":["ruby::params","ruby::dev","ruby::gemrc","ruby","ruby::config","gcc::params","gcc","stdlib","stdlib::stages","apt::params","apt","apt::backports","apt::update","puppetdb::database::postgresql","puppetdb::params","puppetdb::globals","puppetdb::master::storeconfigs","puppetdb::master::config","puppetdb::master::report_processor","puppetdb::master::puppetdb_conf","puppetdb::master::routes","puppetdb::server","puppetdb","puppetdb::server::firewall","puppetdb::server::validate_read_db","puppetdb::server::read_database","puppetdb::server::global","puppetdb::server::database","puppetdb::server::jetty","puppetdb::server::puppetdb","puppetdb::server::command_processing","puppetdb::server::validate_db","r10k::params","r10k::postrun_command","r10k::webhook","r10k::mcollective::application","r10k","r10k::install","r10k::mcollective","r10k::config","r10k::install::bundle","r10k::install::pe_gem","r10k::install::puppet_gem","r10k::install::gem","r10k::install::portage","r10k::webhook::package","r10k::webhook::config","r10k::prerun_command","postgresql::params","postgresql::globals","postgresql::repo::apt_postgresql_org","postgresql::repo::yum_postgresql_org","postgresql::client","postgresql::server","postgresql::repo","postgresql::server::initdb","postgresql::server::service","postgresql::server::reload","postgresql::server::plpython","postgresql::server::install","postgresql::server::plperl","postgresql::server::config","postgresql::server::passwd","postgresql::server::postgis","postgresql::server::contrib","postgresql::lib::python","postgresql::lib::devel","postgresql::lib::java","postgresql::lib::perl","postgresql::lib::docs","supervisord::params","supervisord::service","supervisord::reload","supervisord","supervisord::install","supervisord::config","supervisord::pip","git","git::subtree","git::gitosis","portage::params","portage","portage::install","make::params","make","make::install","firewall::params","firewall::linux","firewall","firewall::linux::debian","firewall::linux::redhat","firewall::linux::archlinux","firewall::linux::gentoo"]}]
[root@puppet installer]# /bin/curl -k -s -u admin:e3iHDhY7QCiwr47n -H "Accept: version=2,application/json" -H "Content-Type: application/json" -X POST -d '{ "name": "OpsTheater Infra", "environment_id": "1", "puppet_ca_proxy_id": "1", "puppet_proxy_id": "1" } '
{"subnet_id":null,"subnet_name":null,"operatingsystem_id":null,"operatingsystem_name":null,"domain_id":null,"domain_name":null,"environment_id":1,"environment_name":"production","compute_profile_id":null,"compute_profile_name":null,"ancestry":null,"puppet_proxy_id":1,"puppet_ca_proxy_id":1,"ptable_id":null,"ptable_name":null,"medium_id":null,"medium_name":null,"architecture_id":null,"architecture_name":null,"realm_id":null,"realm_name":null,"created_at":"2016-03-09T11:48:29Z","updated_at":"2016-03-09T11:48:29Z","id":1,"name":"OpsTheater Infra","title":"OpsTheater Infra","parameters":[],"template_combinations":[],"puppetclasses":[],"config_groups":[],"all_puppetclasses":[]}
Restart PuppetServer
/bin/systemctl restart puppetserver
Running r10K
Download Opstheater-Control.tar.gz to the local server.
Untar opstehater-control in /root.
-Take the backup of existing production enviornment. /etc/puppetlabs/code/production as /etc/puppetlabs/code/production_backup
-Move /root/opstheater-control to /etc/puppetlabs/code/production
Install r10K
-/opt/puppetlabs/bin/puppetserver gem install r10k
-gem install 10k
Execute r10k from /etc/puppetlabs/code/production
-r10k puppetfile install -v
THis will install all the required modules mentioned in Puppetfile in current directory
Setup Hiera
/opt/puppetlabs/bin/puppet config set hiera_config /etc/puppetlabs/code/environments/production/hiera.yaml
Restart PuppetServer
/bin/systemctl restart puppetserver
Update Foreman puppet environments
/bin/curl -k -s -u admin:PASSWORD_FROM_PREVIOUS_COMMAND -H "Accept: version=2,application/json" -H "Content-Type: application/json" -X POST -d '{ }'
Run puppet agent and then stop puppet service
/opt/puppetlabs/bin/puppet agent -t || true
## GitLab Server Setup.
1. Configure repo and install puppet.
/bin/yum install -y epel-release
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server
2. Configure/adjust Master
Ensure correct filename ymal file exists in bellow locations:
3. Once this is done. Puppet agent.
/opt/puppetlabs/bin/puppet agent -t
4. Content Setup
Currently gitlab / mattermost integration requires a bit of manual attention post-install including…
Creating opstheater-control repository
Pointing the puppetmaster’s code source to the opstheater-control repository on their gitlab installation
Creating demo repository from
Setup mattermost
Enable mattermost team creation
Log into mattermost with root user
Create a team
Make the team a public team
disable team creation
Create integration
Copy/paste integration URL into gitlab for build notifications
# Icinga Server Setup
1. Configure repo and install puppet.
/bin/yum install -y epel-release
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server
2. Configure/adjust Master
* Update `/etc/puppetlabs/code/environment/production/hieradata/60-.opstheater.yaml` in icinga section to reflect correct name of icinga on the below line.
* Also update the ipaddress to the public ipaddress.
'opstheater::icinga::fqdn': "icinga.%{hiera('opstheater::domain')}"
'opstheater::icinga::ipaddress': ''
* update site.pp to reflect correct name.
# runs standalone monitoring setup with icinga2 and icinga web2
node '' {
include opstheater::role::monitoring::standalone
3. Run puppet agent
4. Open icinga2 web frontend and log in with icingaadmin / icinga
# mattermost Setup
The user management of mattermost is done through gitlab, but the process is currently a bit strange. Follow the following steps to start using mattermost:
1. Login to the gitlab server and edit `/etc/gitlab/gitlab.rb`. Find the setting `mattermost['team_enable_team_creation']` and change it's value to true.
2. run `gitlab-ctl reconfigure` to let these settings take effect.
3. Go to the mattermost address you configured (eg. `http://chat.opstheater.vm`), it'll now prompt you to create a team.
4. Create a team, then revers the setting `mattermost['team_enable_team_creation']` back to false and run `gitlab-ctl reconfigure` once more.
5. Log in to mattermost like normal.
## MYSQL server Setup
1. Configure repo and install puppet.
/bin/yum install -y epel-release
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server
2. Configure/adjust Master
update /etc/puppetlabs/code/environment/production/hieradata/60-.opstheater.yaml to reflect myqsl server ip.
## MySQL related settings
## MySQL related settings
# Variable: opstheater::mysql::fqdn
# Description:
# Default value: "mysql.%{hiera('opstheater::domain')}"
'opstheater::mysql::fqdn': "mysql.%{hiera('opstheater::domain')}"
# Variable: opstheater::mysql::ipaddress
# Description:
# Default value: ''
'opstheater::mysql::ipaddress': ''
# Variable: opstheater::mysql::whitelist_range
# Description:
# Default value: '10.20.1.%'
'opstheater::mysql::whitelist_range': '10.129.%'
3. Run Puppet Agent
This will throw error for the 1st time as below but on the second run it works fine.
nfo: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]: Filebucketed /etc/my.cnf to puppet with sum 80e1eb23d5fbd77fc0ff681b0f0df297
Notice: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]/content: content changed '{md5}80e1eb23d5fbd77fc0ff681b0f0df297' to '{md5}3ab13aa54c001fe3966a08ae49c5517d'
Notice: Disabling SSL is evil! You should never ever do this except if you are forced to use a mysql version compiled without SSL support
Notice: /Stage[main]/Mysql::Server::Config/Notify[ssl-disable]/message: defined 'message' as 'Disabling SSL is evil! You should never ever do this except if you are forced to use a mysql version compiled without SSL support'
Error: Could not start Service[mysqld]: Execution of '/usr/bin/systemctl start mysqld' returned 1: Job for mysqld.service failed because the control process exited with error code. See "systemctl status mysqld.service" and "journalctl -xe" for details.
Error: /Stage[main]/Mysql::Server::Service/Service[mysqld]/ensure: change from stopped to running failed: Could not start Service[mysqld]: Execution of '/usr/bin/systemctl start mysqld' returned 1: Job for mysqld.service failed because the control process exited with error code. See "systemctl status mysqld.service" and "journalctl -xe" for details.
Notice: /Stage[main]/Opstheater::Profile::Mysql/File[/var/log/mysql]/ensure: created
Notice: /Stage[main]/Mysql::Server::Service/File[/var/log/mysql/error.log]/ensure: created
Info: Class[Mysql::Server::Service]: Unscheduling all events on Class[Mysql::Server::Service]
Notice: /Stage[main]/Mysql::Server::Root_password/Exec[remove install pass]: Dependency Service[mysqld] has failures: true
Warning: /Stage[main]/Mysql::Server::Root_password/Exec[remove install pass]: Skipping because of failed dependencies
Info: Creating state file /opt/puppetlabs/puppet/cache/state/state.yaml
Error: Failed to apply catalog: Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
# Deployment scenario's
In short, there are three different ways to deploy OpsTheater, each with their own reason and their own setup process. It is important to understand this and make a right choice before running off and setting up OpsTheater.
## Scenario 1: OpsTheater development setup
Usecase: you want to help in developing OpsTheater further. In this scenario, you don't plan to ever run any of the deployment in production. You are okay to use local virtualbox and vagrant in order to give you maximum flexibility in bringing up and tearing down vm's. Some instructions can be found here:
## Scenario 2: OpsTheater deployment for non-production
Usecase: in this scenario you want to explore OpsTheater and see what it is all about. You want to use your specific hostnames and choose to customise the setup to your liking. You deploy on local virtualbox and vagrant vms as that is easy. You may or may not bring this type of deployment to production eventually but that is not your first goal.
<<instructions coming soon>>
## Scenario 3: OpsTheater deployment for production
Usecase: in this scenario you are planning a (new) production environment and need to make careful choices that fit for your architecture and your usecase. In this scenario we can't do the deployment in an automated fashion, so expect lots of planning work.
Read more:
Some Frequently Asked Questions:
Q) I am having trouble getting the stack up and running
A) We've seen several issues regarding the versions of vagrant and virtualbox in combination with your host OS. Confirmed working combinations:
Host: Ubuntu 16.04/16.10
Vagrant: 1.8.1 (1.8.5: and 1.9.1)
Virtualbox: 5.0.30
# Instructions for attaching an existing node to opstheater
In order to add an existing node to OpsTheater the puppet agent for the operating system needs to be installed. See the website for details.
# Instructions for attaching a new node to opstheater
Make sure the node gets the puppet agent installed as part of it's provisioning and point it at the puppetmaster you have set up. More details on this can be found on the puppetlabs website.
## OpsTheater Release Checklist for Release 1.7.0
**Following checks need to be performed after every release :**
- Check if you can login into following using your web browser :
- [ ] Foreman
- [ ] Gitlab
- [ ] Mattermost
- [ ] Icinga
- [ ] Grafana
- [ ] Kibana
- [ ] Check if you are able to send message using Mattermost.
- [ ] Check if you are able to create a user in Foreman and can login with that user.
- [ ] Check if you can see the logs coming up for OpsTheater components in Icinga Web UI.
- [ ] Try to search for a host in Kibana and check if you are getting logs of that host.
[ ] Check if you can create users and usergroups in Icinga
**Release specific Checks :**
*These checks will change on every release and will be more specific for the new changes that we are introducing in this release.*
- [ ] Login to Grafana web UI and check if you can see different metrics in Grafana
[ ] Login to grafana and check that you can see two datasources. Edit each one of them and press "Test Connection" to make sure they work.
- [ ] Login to Kibana web UI and try to search for logs having source as "/var/log/puppetlabs/puppetserver/puppetserver-access.log". Check if the value of field timestamp is same as the time given in message field
# Release process
This borrows heavily and is modeled after:
**NOTE: This is a guide used by OlinData BV for handling releases of OpsTheater. As an end user you do not need to use this guide.**
# Release cycle
Problems notwithstanding we release a new version of OpsTheater every month on the first working day of the month. Our timing is based on and coincides with a number of the software packaged within OpsTheater. Security releases are published when needed. Our changelog for the release will contain changes, new features, and packaged software version information.
# Release timing
Monthly release, the 1st day of every month.
Patch release, for serious bugs or security problems.
# Monthly Release Internal Process
1. The process starts the 23rd of every month, in which we evaluate the stability and inclusion of newer versions of our included packages. The release manager decides on a version of all supporting packages and creates a ChangeLog to be included with the release and documented in this repository under releases/
2. First thing that occurs the release manager opens a issue for the new release and to keep track of the release. You can refer this one for example :
3. After this the release manager creates a release-x.x.x tag from the master branch. Typically monthly releases are sub-releases, so it should increment the Y in x.y.z, making 1.2.3 into 1.3.0. The final version octet is reserved for emergency bug/security patches, and the first octet is reserved for us to designate major evolutions in the software at our own discretion.
4. From this release branch, the release manager attempts a fresh installation of all of the currently supported services on a staging-ish environment, typically using our primary provider DigitalOcean or a local vagrant setup.
5. Form this local installation release manager will make a list out current stable versions of the tools included in OpsTheater.
6. If this installation happened without problem, then (ideally) a fresh installation is setup from the previous release, and an upgrade installation is performed and tested. This above process is expected to take 1-3 days depending on our level of automation and problems occurred during installation/upgrading/testing. Patches can be made to fix any release problems and committed directly to the release branch to be merged back into the master with a PR after released.
7. If the release testing will take more than 3 days, we evaluate the progress, changes, and situation and consider postponing deploy or proceeding with it.
8. Create release notes and a blog post for the blog.
9. Once the upgrade installation is succeeded then release manager need to create a new directory in opstheater-docs reposiroty cloning previous release directory and need to update the documents in the new directory according to the new release for example release manager need to update with the steps that need to be performed for upgrade installation.
10. After this release manager need to update in current release branch of opstheater-docs. will contains the basic steps that release manager needs to perform once the upgrade is finised to make sure that all opstheater components are working fine.
11. Schedule tweets and a mail to the mailing list also.
12. After a release is tested for installation and upgrading, we upgrade our internal core OpsTheater installation, so we are consuming what we're creating. This needs to be done at least 4 days before the end of the month, so that we can have at least two working days with our platform upgraded to settle any potential longer-tailed issues with the upgrade. If any issues arise, the release manager can either opt to fix them and commit to the release branch, or to rollback the platform to the previous release depending on the severity and estimated time to fix.
13. When the first of the month rolls around, if there were no problems with the release, a release is created in github from the release branch, and any changes made are PR'd back to the master branch for approval by another core team member. This tag is not to be purged after the PR is merged to master, but to remain in git forever.
14. Now release manager can close the issue which he created for this new release.
15. At this point, we send an update notification email to any subscribers to OpsTheater mailers and begin the process to upgrade all of our OpsTheater installations that are paid subscribers. As part of the signup process with our customers, we decide on a day-of-the-week and time to perform maintenance on their OpsTheater. Following their requested schedule, we perform the necessary upgrade, and inform our customers of this 2 days before, and if there is no objection then also notify them as we begin the upgrade procedure, and after its completion along with a changelog of any platform changes/improvements/etc.
# OpsTheater 1.7.0 Upgrade instructions (from 1.6.0)
For this release OpsTheater components will need to be upgraded manually. In the future we aim to automate as much of this as possible, all help is welcome!
* Gitlab will be upgraded from version 8.6 to 8.7
* Mattermost will be upgraded from version 2.1 to 2.2
* Percona server stays at 5.6.29
* Logstash stays at 2.1.1
* Kibana stays at 4.2.1
* Puppet from 4.4.0 to 4.4.2
* Foreman from 1.10.2 to 1.10.3
* Icinga 2 from 2.4.4 to 2.4.7
* Icinga web 2 from 2.2.0 to 2.2.3
* Filebeat from 1.2.0 to 1.2.2
* grafana stays at 2.5.0
Please see the release notes for a list of versions:
## Steps
1. Disable puppet on all OpsTheater nodes
Login to the following servers one by one and disable puppet agent by command `puppet agent --disable`
2. Upgrade MySQL
No MySQL upgrade is needed this time, so we just need to upgrade the icinga, puppet and filebeat packages.
# if you want to update all packages (not just OpsTheater packages) feel free
# to run `yum update` without specifying packages
yum update icinga2 puppet-agent filebeat
After that is done, check that the installed versions are correct:
[root@mysql ~]# rpm -qa | egrep -i 'icinga2|puppet-agent|filebeat' | sort
3. Upgrading elasticsearch
To upgrade Elasticsearch from 2.2 to 2.3 login to elasticsearch server and run the following commands:
# check the current versions of OpsTheater components
[root@elastic ~]# rpm -qa | egrep -i 'icinga2|puppet-agent|filebeat|elastic' | sort
# if you want to update all packages (not just OpsTheater packages) feel free
# to run `yum update` without specifying packages
yum update icinga2 puppet-agent filebeat elasticsearch
After that is done, check that the installed versions are correct:
[root@elastic ~]# rpm -qa | egrep -i 'icinga2|puppet-agent|filebeat|elastic' | sort
4. Upgrading icinga 2
To upgrade Icinga 2 from 2.4.3 to 2.4.4 and Icinga web 2 from 2.1.2 to 2.2.0 login to the elastic server and run the following commands:
[root@icinga ~]# rpm -qa | egrep -i 'icinga2|puppet-agent|filebeat|^icingaweb2-' | sort