I recently worked on a new product. Our escalation engineers didn't trust customers to make modifications or customize even though the product was 100% open source. Customers disliked that and the product crashed and burned.
Had our engineering teams trusted our customers and making the needed customizations the product would have been useful to our customers and wouldn't crashed and burned.
I had a discussion with our engineering director. He explained to me that he wanted to bring “DevOps” to the entire organization. Being one of the only Sr. SRE’s for the company, they asked what I would do. I responded, “We need to build a culture where we can make changes to our product safe and offer a culture of experimentation.”
This is the basis for trust. When you can fail fast, fail often and recover quickly, it builds trust in the entire organization.
I have learned over the years that DevOps keeps me sane. I say this because throwing random code over the wall and expecting it to run well in production is the very definition of insanity. The Devops processes of working with a team of mixed developers and Site engineers helps uptime and business productivity... When things do go wrong root causes are identified more quickly. It has been amazing to me to watch cultures shift as developers are responsible for their code running in production.
Our compliance teams have a requirement that all logs be processed by a new SIEM tool for real time security event detection. One of our more advanced DevOps teams was the first to use the new tool, the Security Team provided them a 5 page word document questionnaire, which asked for the locations of all the log files the application created, the format of each log file, samples of messages from the event logs, and finally, provided a 1 month SLA for configuration of the SIEM tool for the application once the questionnaire was completed. We have 5000+ applications, we’ll be done implementing the SIEM tool after my grand children have finally passed away. In DevOps, all log aggregation is configured automatically, not via Word document.
What DevOps is NOT…. Our Operations teams use an enterprise tool for patching of systems. They indicated that the tool does not work and that another team needed to find another solution. The engineering team took the challenge and asked for an Ops team to be involved from the beginning to ensure that any solutions provided would meet their needs. The requirements were provided via a word document and the Ops team promptly disengaged. The engineering team continued to work on a solution involving Chef which used Git/ELK/Chef Compliance for rolling out patches to 50,000 devices in under 48 hours. Sporadic involvement was received from the Operations teams and finally after all testing completed and the solution was documented and multiple trainings completed, the Operations Manager stated that they were “interested” in this new model (which was 5x more efficient than their current model) and would like to discuss it, bi-weekly over a few months to determine what happens next. In DevOps, all teams with accountability are actively engaged on solving problems and "throw it over the wall" mentality is avoided.
I was working with a team of developers for about 4 months when the Sr. Software Engineer asked me to come to a meeting. When I sat down with him, he informed me that all the automation and monitoring that I had set up for them has done something amazing for him. He was now able to take vacation time with his family. They whole team now trusted my team to take care of the systems without fear of having to always be on call. Building that trust is how you practice DevOps.
We've got all the rockstars from our industry in one place just for you! Experienced industry leaders who want to share their experience with you!
NOAH’S of South Jordan is a three-story premier event venue located minutes from downtown Salt Lake City.