DevOps – Lower project risk

We all wish to be out of the “install” business. How many times has your projects require installing or re-installing OS, solutions, and databases, & then perform post-install configurations, where it seems the same challenges are repeated.

Fortunately, with the SaaS services, virtualization of OS platforms, and solution appliances, the effort/time to install a component is on its way to being a small line item within a project plan.

However, in the enterprise software world, there are still components that have not been fully deployed in one of the above models, or the business/technical requirements do not fit the above models.

The value of DevOps to an enterprise project, is in two (2) areas:

1. Automating deployment & re-deployment of solution components
   • First effort is always a wash for initial deployments, but value gained on the next environment(s)
   • Utilizing automated deployments ensure quality between peer members of components, e.g. install exactly the same way
   • Avoids the “too-many-cooks-in-the-kitchen” challenge when more than two (2) resource deploy in differ ways.
   • Allows rapid scaling of the solution with horizontally cluster integration.
2. Automating deployment of business content from a dev to test to qa to pre-prod to eventually the production environment.
   • Hands on keyboard once
   • Workflow Approval/Reject of changes
   • Avoid downtime for business release cycles.
   • Allow for rollback of business content.
   • Lower project cost due to large maintenance windows.

DevOps does not have to be overwhelming. While customers’ may invest in enterprise DevOps solutions or open-source solution, like Chef, the proof-of-concept process that many follow first is using the embedded command line offerings in solutions to install solution components using the “silent” or “non-interactive” switches. For DevOPS with business content migration, the use of APIs via SOAP/REST or again CLI (command line interfaces) are used with script languages that customer staffs are familiar with.

Example of a command line to install Oracle Java JDK, that was then migrated to an enterprise DevOps solution:

A view of the architecture methodology we use for DEV-OPS of any vendor solution, e.g. CA Technologies, Oracle, RedHat, Microsoft, etc.

DevOps Architecture Methodology:  Bottom-Up Approach 

• To meet the expected use of the CLI processes for migration into a commercial DevOps solution, the approach will attempt to emulate the same behavior.
• Server(s) Acquisition –The servers are assumed to be compatible with the solution’s support matrices and to have the supported OS.
  • Most solution(s) are able to adequate run on 2-4 vCPU with 8-32 GB RAM.
    • Disk space may be from 10-100 GB.
  • A sandbox environment should attempt to run all solutions within a 2 vCPU with 8 GB RAM on 80 GB disk (20 GB for OS and 60 GB for data)
    • OS Mount Point
      • The majority of the solutions will deploy under /opt/<subfolder> on a Linux/UNIX OS.
      • This mount point will be chosen for all vendors and 3^rd party solutions.
    • OS Libraries
      • OS libraries that are identified from literature, the installers, debug steps will be pre-loaded as a separate script.
    • OS Entropy
      • OS Entropy will be requested to be deployed prior to any installation via the OS RGND service or 3^rd party RGND solutions. {Do NOT miss this step}

• 3^rd Party & vendor solution Installations
  • Any software declared as a predecessor solution, will be installed in the correct order to avoid re-work effort.
  • Any software solution that allow peer and/or cluster setup; will be deployed as a cluster member of 1; to allow future “stacks” to be integrated rapidly; and to allow a solution to scale horizontally.
  • Any co-location of software components, on the same server, will be isolated by folder, network ports, and JBOSS/Wildfly/J2EE instances.
  • Any software installation that require pre-installation steps will be identified and a process will be built via CLI to manage the general use-case of deployment and integration.
  • Any software installation that require input for service accounts; passwords; or other variables will be address with an input properties file and/or script variables that will be defined at the header of the script.
  • Any software installation that requires non-root account to install, will be identified and/or updated to use sudo access to execute as the non-root account.
  • Any software wizard installation that is not clear on “changes” or delta to the install base, will be execute several times to capture the deltas.
    • Process to capture install deltas (file based)
      • Install 1^st time with interactive console mode. §tar/zip up the install folder
      • Install 2^nd time with interactive console mode
        • Tar/zip up the install folder
      • Copy both files to a workstation/laptop and use a file/folder compare tool (WinMerge/Beyond Compare) to identify the file /folder deltas
  • Update software installation silent install scripts to use variables to manage the deltas.

Finally, we arrive at the DevOps Architecture Methodology:  Installation Processes

• All solutions will be installed with CLI processes that may be executed as root or a non-root service account, as needed.
• Interaction with Web Server/Web Application Servers will be managed with the CLI process of CuRL/MS Powershell/PDI/etc..
• Bottom-Up approach will be used, where assumption of pre-installed components will be declared.
• Installations scripts will follow the methodology:
  • Declare properties file and/or variables to be used.
  • Uninstall any prior installation
    • Execute shutdown script/process (if exists)
    • Execute OS kill command (search for running processes)
    • Execute uninstall script (provided by solution)
    • Remove installation folder
  • Install solution
    • Update silent install input file based on properties and/or script variables
    • Install solution via silent install input file
  • Perform base validation checks (query on files/folders/running processes)

Now we reach this section: DevOps Architecture Methodology:  Post Install / Integration

• Realized & set/manage the expectation that there is a point-of-diminished returns for value of automating installations.
  • 1^st Question:  Do we understand task A intent (requirements/business logic)?
  • 2nd Question:  Can task A be automated.  
  • 3^rd Question:  Should task A be automated.
• The post install and/or migration integration steps between components where the components reside on separate servers.
  • Otherwise the dev-ops process needs to be aware of the remote IP/hostnames of the cluster members.
    • May be pre-defined in  3^rd Party & Vendor solution Installations
    • Note:  If “dummy hostnames” are to be used, ensure they are mapped to the current hostname to have them resolvable by DNS lookup.
• Business logic unique to each customer may be pre-defined by use-case
  • The assumption is that well-defined business logic unique to each solution has been pre-built for consumption.
• Service Accounts’ Passwords that were hard-coded as part of the silent install scripts will need to be immediately changed to new secure passwords prior to any production roll-out.

The project effort for DevOps above can be considered a “wash” with the expected manual installation effort for the first environment (dev) for OS & a vendors solution. The value will be realized / gained for project timelines due to deployment of the next 2-5 environments, when resources change, re-deployment is required, when there is a need to horizontally scale the solution, or if a disaster recovery site is required. Project timeline risk will be minimized, and the knowledge gained by the resources that build the DevOps processes will lower business risk during future triages of technical challenges.

The second value of DevOps is around business release process, that we will cover in another blog entry, to promote business logic/content from one environment to the next environment, until finally deployment to the production environment.