Commit 459542cd authored by Walter Heck's avatar Walter Heck

More adjustments and clarifications for the documentation

parent 2a52c239
## Elastic Server Setup
1. Configure repo and install puppet.
```
/bin/yum install -y epel-release http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server puppet.olindata.com
```
Configure repo and install puppet.
2. Configure/adjust Master
```
/bin/yum install -y epel-release http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server puppet.olindata.com
```
Ensure correct filename ymal file exists in bellow locations:
```
/etc/puppetlabs/code/environments/production/hieradata/10.fqdn
```
Merge elastic.olindata.com.yaml and logstash.olindata.vm.yaml As we have single server or both application.
(Remember to edit the file and remove extra --- and change the server name to elastic from elasticsearch)
Configure/adjust Master
Logstash settings needs to be checked for public vs private ip
Ensure correct filename ymal file exists in bellow locations:
Ref: - "elastic.olindata.com:9200"
```
/etc/puppetlabs/code/environments/production/hieradata/10.fqdn
```
3. wget needs to be installed before running puppet agent -t.
Merge elastic.olindata.com.yaml and logstash.olindata.vm.yaml As we have single server or both application.
(Remember to edit the file and remove extra --- and change the server name to elastic from elasticsearch)
4. Add DNS entry for logstash.olindata.com OR update /etc/filebeat/filebeat.yml to talk to elastic.olindata.com
5. update site.pp as bellow:
```
node 'elastic.olindata.com' {
include opstheater::role::elastic::server
include opstheater::role::logstash::server
}
```
Logstash settings needs to be checked for public vs private ip
Ref: - "elastic.olindata.com:9200"
wget needs to be installed before running puppet agent -t.
Add DNS entry for logstash.olindata.com OR update /etc/filebeat/filebeat.yml to talk to elastic.olindata.com
update site.pp as bellow:
```
node 'elastic.olindata.com' {
include opstheater::role::elastic::server
include opstheater::role::logstash::server
}
```
Run Puppet agent.
6. Run Puppet agent.
## GitLab Server Setup.
Configure repo and install puppet.
1. Configure repo and install puppet.
```
/bin/yum install -y epel-release http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server puppet.olindata.com
```
```
/bin/yum install -y epel-release http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server puppet.olindata.com
```
2. Configure/adjust Master
Configure/adjust Master
Ensure correct filename ymal file exists in bellow locations:
Ensure correct filename ymal file exists in bellow locations:
```
/etc/puppetlabs/code/environments/production/hieradata/10.fqdn
```
```
/etc/puppetlabs/code/environments/production/hieradata/10.fqdn
```
3. Once this is done. Puppet agent.
Once this is done. Puppet agent.
```
/opt/puppetlabs/bin/puppet agent -t
```
```
/opt/puppetlabs/bin/puppet agent -t
```
## Icinga Server Setup
# Icinga Server Setup
Configure repo and install puppet.
1. Configure repo and install puppet.
```
/bin/yum install -y epel-release http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server puppet.olindata.com
```
```
/bin/yum install -y epel-release http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server puppet.olindata.com
```
Configure/adjust Master
2. Configure/adjust Master
* Update `/etc/puppetlabs/code/environment/production/hieradata/60-.opstheater.yaml` in icinga section to reflect correct name of icinga on the below line.
* Also update the ipaddress to the public ipaddress.
Update /etc/puppetlabs/code/environment/production/hieradata/60-.opstheater.yaml in icinga section to reflect correct name of icinga on the below line.
Also update the ipaddress to the public ipaddress.
```
'opstheater::icinga::fqdn': "icinga.%{hiera('opstheater::domain')}"
'opstheater::icinga::ipaddress': '37.139.31.34'
```
'opstheater::icinga::fqdn': "icinga.%{hiera('opstheater::domain')}"
* update site.pp to reflect correct name.
'opstheater::icinga::ipaddress': '37.139.31.34'
```
# runs standalone monitoring setup with icinga2 and icinga web2
node 'icinga.olindata.com' {
include opstheater::role::monitoring::standalone
}
```
update site.pp to reflect correct name.
```
# runs standalone monitoring setup with icinga2 and icinga web2
node 'icinga.olindata.com' {
include opstheater::role::monitoring::standalone
}
```
Run puppet agent
3. Run puppet agent
## MYSQL server Setup
Configure repo and install puppet.
```
/bin/yum install -y epel-release http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server puppet.olindata.com
```
Configure/adjust Master
update /etc/puppetlabs/code/environment/production/hieradata/60-.opstheater.yaml to reflect myqsl server ip.
```
##########################
## MySQL related settings
##########################
## MySQL related settings
# Variable: opstheater::mysql::fqdn
# Description:
# Default value: "mysql.%{hiera('opstheater::domain')}"
'opstheater::mysql::fqdn': "mysql.%{hiera('opstheater::domain')}"
# Variable: opstheater::mysql::ipaddress
# Description:
# Default value: '10.20.1.60'
'opstheater::mysql::ipaddress': '10.129.2.113'
# Variable: opstheater::mysql::whitelist_range
# Description:
# Default value: '10.20.1.%'
'opstheater::mysql::whitelist_range': '10.129.%'
```
Run Puppet Agent
This will throw error for the 1st time as below but on the second run it works fine.
```
nfo: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]: Filebucketed /etc/my.cnf to puppet with sum 80e1eb23d5fbd77fc0ff681b0f0df297
Notice: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]/content: content changed '{md5}80e1eb23d5fbd77fc0ff681b0f0df297' to '{md5}3ab13aa54c001fe3966a08ae49c5517d'
Notice: Disabling SSL is evil! You should never ever do this except if you are forced to use a mysql version compiled without SSL support
Notice: /Stage[main]/Mysql::Server::Config/Notify[ssl-disable]/message: defined 'message' as 'Disabling SSL is evil! You should never ever do this except if you are forced to use a mysql version compiled without SSL support'
Error: Could not start Service[mysqld]: Execution of '/usr/bin/systemctl start mysqld' returned 1: Job for mysqld.service failed because the control process exited with error code. See "systemctl status mysqld.service" and "journalctl -xe" for details.
Error: /Stage[main]/Mysql::Server::Service/Service[mysqld]/ensure: change from stopped to running failed: Could not start Service[mysqld]: Execution of '/usr/bin/systemctl start mysqld' returned 1: Job for mysqld.service failed because the control process exited with error code. See "systemctl status mysqld.service" and "journalctl -xe" for details.
Notice: /Stage[main]/Opstheater::Profile::Mysql/File[/var/log/mysql]/ensure: created
Notice: /Stage[main]/Mysql::Server::Service/File[/var/log/mysql/error.log]/ensure: created
Info: Class[Mysql::Server::Service]: Unscheduling all events on Class[Mysql::Server::Service]
Notice: /Stage[main]/Mysql::Server::Root_password/Exec[remove install pass]: Dependency Service[mysqld] has failures: true
Warning: /Stage[main]/Mysql::Server::Root_password/Exec[remove install pass]: Skipping because of failed dependencies
Info: Creating state file /opt/puppetlabs/puppet/cache/state/state.yaml
Error: Failed to apply catalog: Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
```
1. Configure repo and install puppet.
```
/bin/yum install -y epel-release http://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
/bin/yum install -y puppet-agent
/opt/puppetlabs/bin/puppet config set --section main server puppet.olindata.com
```
2. Configure/adjust Master
update /etc/puppetlabs/code/environment/production/hieradata/60-.opstheater.yaml to reflect myqsl server ip.
```
##########################
## MySQL related settings
##########################
## MySQL related settings
# Variable: opstheater::mysql::fqdn
# Description:
# Default value: "mysql.%{hiera('opstheater::domain')}"
'opstheater::mysql::fqdn': "mysql.%{hiera('opstheater::domain')}"
# Variable: opstheater::mysql::ipaddress
# Description:
# Default value: '10.20.1.60'
'opstheater::mysql::ipaddress': '10.129.2.113'
# Variable: opstheater::mysql::whitelist_range
# Description:
# Default value: '10.20.1.%'
'opstheater::mysql::whitelist_range': '10.129.%'
```
3. Run Puppet Agent
This will throw error for the 1st time as below but on the second run it works fine.
```
nfo: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]: Filebucketed /etc/my.cnf to puppet with sum 80e1eb23d5fbd77fc0ff681b0f0df297
Notice: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]/content: content changed '{md5}80e1eb23d5fbd77fc0ff681b0f0df297' to '{md5}3ab13aa54c001fe3966a08ae49c5517d'
Notice: Disabling SSL is evil! You should never ever do this except if you are forced to use a mysql version compiled without SSL support
Notice: /Stage[main]/Mysql::Server::Config/Notify[ssl-disable]/message: defined 'message' as 'Disabling SSL is evil! You should never ever do this except if you are forced to use a mysql version compiled without SSL support'
Error: Could not start Service[mysqld]: Execution of '/usr/bin/systemctl start mysqld' returned 1: Job for mysqld.service failed because the control process exited with error code. See "systemctl status mysqld.service" and "journalctl -xe" for details.
Error: /Stage[main]/Mysql::Server::Service/Service[mysqld]/ensure: change from stopped to running failed: Could not start Service[mysqld]: Execution of '/usr/bin/systemctl start mysqld' returned 1: Job for mysqld.service failed because the control process exited with error code. See "systemctl status mysqld.service" and "journalctl -xe" for details.
Notice: /Stage[main]/Opstheater::Profile::Mysql/File[/var/log/mysql]/ensure: created
Notice: /Stage[main]/Mysql::Server::Service/File[/var/log/mysql/error.log]/ensure: created
Info: Class[Mysql::Server::Service]: Unscheduling all events on Class[Mysql::Server::Service]
Notice: /Stage[main]/Mysql::Server::Root_password/Exec[remove install pass]: Dependency Service[mysqld] has failures: true
Warning: /Stage[main]/Mysql::Server::Root_password/Exec[remove install pass]: Skipping because of failed dependencies
Info: Creating state file /opt/puppetlabs/puppet/cache/state/state.yaml
Error: Failed to apply catalog: Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
```
# Release process
1. On the 23rd of each month, we feature-freeze opstheater-control and freeze versions of included tools
2. create a branch pre-release off of that version?
3. list out current stable versions of the tools included in opstheater
4. We test upgrade from previous stable version
5. We test fresh deploys
6. We incorporate any extra changes only if they are otherwise breaking
7. We create release notes
8. on the 28th of each month, we release the new version and help clients with support contracts upgrade
9. a tarball of the pre-release branch named after whatever version tag it then has
For a log of 1.4 release process see: https://github.com/olindata/opstheater-control/issues/55
This borrows heavily and is modeled after: http://doc.gitlab.com/ce/release/monthly.html
......@@ -31,12 +21,22 @@ Patch release, for serious bugs or security problems.
2. First thing that occurs the release manager creates a release-x.x.x tag from the master branch. Typically monthly releases are sub-releases, so it should increment the Y in x.y.z, making 1.2.3 into 1.3.0. The final version octet is reserved for emergency bug/security patches, and the first octet is reserved for us to designate major evolutions in the software at our own discretion.
3. From this release branch, the release manager attempts a fresh installation of all of the currently supported services on a staging-ish environment, typically using our primary provider DigitalOcean. If this installation happened without problem, then (ideally) a fresh installation is setup from the previous release, and an upgrade installation is performed and tested. This above process is expected to take 1-3 days depending on our level of automation and problems occurred during installation/upgrading/testing. Patches can be made to fix any release problems and committed directly to the release branch to be merged back into the master with a PR after released.
3. From this release branch, the release manager attempts a fresh installation of all of the currently supported services on a staging-ish environment, typically using our primary provider DigitalOcean or a local vagrant setup.
3. list out current stable versions of the tools included in opstheater
4. If this installation happened without problem, then (ideally) a fresh installation is setup from the previous release, and an upgrade installation is performed and tested. This above process is expected to take 1-3 days depending on our level of automation and problems occurred during installation/upgrading/testing. Patches can be made to fix any release problems and committed directly to the release branch to be merged back into the master with a PR after released.
5. If the release testing will take more than 3 days, we evaluate the progress, changes, and situation and consider postponing deploy or proceeding with it.
7. We create release notes and a blog post for the olindata.com blog.
4. If the release testing will take more than 3 days, we evaluate the progress, changes, and situation and consider postponing deploy or proceeding with it.
8. Schedule tweets and a mail to the mailing list also.
5. After a release is tested for installation and upgrading, we upgrade our internal core opstheater installation, so we are consuming what we're creating. This needs to be done at least 4 days before the end of the month, so that we can have at least two working days with our platform upgraded to settle any potential longer-tailed issues with the upgrade. If any issues arise, the release manager can either opt to fix them and commit to the release branch, or to rollback the platform to the previous release depending on the severity and estimated time to fix.
6. After a release is tested for installation and upgrading, we upgrade our internal core opstheater installation, so we are consuming what we're creating. This needs to be done at least 4 days before the end of the month, so that we can have at least two working days with our platform upgraded to settle any potential longer-tailed issues with the upgrade. If any issues arise, the release manager can either opt to fix them and commit to the release branch, or to rollback the platform to the previous release depending on the severity and estimated time to fix.
5. When the first of the month rolls around, if there were no problems with the release, a release is created in github from the release branch, and any changes made are PR'd back to the master branch for approval by another core team member. This tag is not to be purged after the PR is merged to master, but to remain in git forever.
7. When the first of the month rolls around, if there were no problems with the release, a release is created in github from the release branch, and any changes made are PR'd back to the master branch for approval by another core team member. This tag is not to be purged after the PR is merged to master, but to remain in git forever.
6. At this point, we send an update notification email to any subscribers to OpsTheater mailers and begin the process to upgrade all of our OpsTheater installations that are paid subscribers. As part of the signup process with our customers, we decide on a day-of-the-week and time to perform maintenance on their OpsTheater. Following their requested schedule, we perform the necessary upgrade, and inform our customers of this 2 days before, and if there is no objection then also notify them as we begin the upgrade procedure, and after its completion along with a changelog of any platform changes/improvements/etc.
8. At this point, we send an update notification email to any subscribers to OpsTheater mailers and begin the process to upgrade all of our OpsTheater installations that are paid subscribers. As part of the signup process with our customers, we decide on a day-of-the-week and time to perform maintenance on their OpsTheater. Following their requested schedule, we perform the necessary upgrade, and inform our customers of this 2 days before, and if there is no objection then also notify them as we begin the upgrade procedure, and after its completion along with a changelog of any platform changes/improvements/etc.
9. a tarball of the pre-release branch named after whatever version tag it then has
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment