Written by Many years ago, when I was a software developer, a very smart boss said to me: “Tanya, it’s always operations first. Projects after.” At first, I was confused, how will I make any progress on my projects if I’m always doing operations? And, what the HECK is “operations” anyway? Reader, this was an incredibly important lesson that has helped me countless times, throughout my entire career.
At We Hack Purple (WHP), we have a rule: “operations first, then everything else”. Operations means all the stuff you already regularly do, that people are counting on you for. For instance, at WHP, every single week there’s a newsletter. If it’s late, our subscribers ask where it is. Subscribers expect it and enjoy receiving it. It is one of the services that we offer, and it’s part of our general operations.
Other examples of WHP operations: running payroll, answering support requests from students in our academy, emails from active clients, accepting/approving new community members, and moderating our online community if someone acts inappropriately. Imagine if my team and I were “too busy” to run payroll, or ensure students could log into their accounts, or to let new people into our community? It would become a huge bottle neck for the business, and WHP would be known for leaving people disappointed.
When I was a software developer, ensuring that all bugs were fixed, customer problems/complaints were addressed, all our apps were up and running, and that my entire team knew what they needed to do (and had the information/access/resources to do it), meant that then I could work on my projects. Ensuring people weren’t waiting on me not only meant I ran a smoothly running shop; it got me promoted. Multiple times!
Examples of Application Security ‘Operations’:
- Attending project kick off meetings to make yourself known to the team
- Provide security requirements for all new projects
- Performing threat modelling sessions, and completing the paperwork afterwards
- Following up on unfixed bugs
- Checking in with your security champions, every month
- Answering questions from… Everyone.
- Running scans, reviewing scan results
- Arranging pentests, reviewing results with dev team
- Reporting up to management
- Being ready, should a security incident occur
Recently I had a conversation with a client who was trying a new project management methodology, and we were talking about how to best implement it at their org. After about 10 minutes of discussing, I said “What about operations? Sounds like you’re not getting your everyday work done. If you can’t even finish up the close out of a security incident from 6 months ago, you don’t need a new project management system. You need to stop over-allocating the people on your team. Ideally, operations should take somewhere between 25-50% of your time, but it sounds like you have a lot of not-quite-finished work items. Get all that done, then start on new projects. And ensure you save time, every day, for operations.”
Note: it was 25-50% of the time *for her team* and the responsibilities they had, to run operations. For your team it might be higher or lower. When I was a software developer, I was told to never allocate a resource above 80%, because something always comes up. And they were right!
Be prepared, always add 20% to your time estimates for software projects. You’ll thank me.
If your team is supposed to review the architecture for every single software project, and you have allocated zero time for it, how do you think that’s going to go? It’s not going to be good, that’s for sure. It sounds obvious when I lay it out like that, and you might think “I would never do that”, but guess what? I see companies do this ALL THE TIME. And I know I have been guilty of this in the past, not realizing I had done it.
Any security activities that you want to do as part of the system development life cycle (SDLC) are part of your team’s operations. If there are documents to review, meetings to attend, scans to run, whatever, you need to ensure you have the capacity to perform these activities as needed. You can’t say “You must complete this 20-page architecture document, then receive our approval, before you start your coding phase” then proceed to make them wait several weeks for feedback from your team. Or… I guess technically you CAN do this, but it will cause a lot of problems, frustration, and delays for other teams. *Note: I would not recommend this strategy to friends.*
Then my client and I got into a discussion about the 3 ways of DevOps (as per The Phoenix Project and The DevOps Handbook), with the 1st way being “Emphasize the efficiency of the entire system”, the 2nd being “Fast Feedback”, and the 3rd way “Taking Time to Improve Your Everyday Work.” I LOVE DevOps, and in my opinion, The Three Ways are rules to live by if you work in IT.
I know I’ve talked to about The Three Ways of DevOps a lot, but they add value in SO MANY situations! I just can’t help myself, they are just SO GOOD.
Unplanned Work: There will always be external forces that you cannot control. A security incident will happen. The big boss decides your team is going to run a tabletop exercise. Suddenly your team is in charge of something it never was before. If you have allocated your team’s resources already at 100%, there’s no space for this. And unplanned work is (unfortunately) a fact of life. One of the things you can do in such a situation and ask ‘the powers that be’ which thing will come off of your plate if you add this new one. If they say nothing comes off, explain that you team is already fully allocated, and you expect them to start slipping on other projects or operational requirements. At least you warned them, right?
This might not make you popular, but it WILL ensure your projects succeed, on time.
The First Way of DevOps: Emphasize the efficiency of the entire system.
If security teams around the world took this to heart, people would like their IT Security co-workers a lot more. I have heard hundreds of times “We make them fill out all these forms, then we don’t have time to read them.” So…. Did you stop making them fill out the forms? “No.“
If instead the security team adopted the model of “operations first”, they would either 1) make those forms way less complicated and time consuming, and 2) allocate enough time for their team to properly review them, promptly. It is my wish that security teams would look at all the inputs (forms, meetings, documents, and so on) that they ask for from other teams, and then ensure they have capacity to use those inputs to their fullest, in efforts to protect their organizations. This might mean reducing risk, adding additional layers of security, preparing for potential disasters, etc. Security teams demanding other teams to perform work, and then not fully utilizing the work they asked for, makes me very upset. I used to be a software developer, and I have been put through the paces by a lot of management in my time, and I’m quite tired of filling out templates that no one ever reads… And I know I am not the only person who feels this way!
Tip: whenever possible, make services from your organization self-service. If you can provide all the guidance on a wiki page, do it. If you can create a portal where all the developers can access your toolset for them, do it. If you can set it up so that you already have a pre-existing contract with a pentesting company, and they just need to call and book the dates, do it. This results in saved time, less friction between teams, and operations moving much more smoothly!
A great tip, from me to you!
The Second Way of DevOps: Fast Feedback
I tend to add onto this phrase and change it from ‘fast feedback’ to ‘feedback, that is accurate, and gets to the right person/people, fast”. Who cares if the feedback is fast if it never gets to its intended destination? Or it’s completely inaccurate so it sends someone on a wild goose chase? That is not helpful.
This is another area where if we do ‘operations first’ that we will see some big benefits. Whenever a project team asks for feedback from the security team, if we turned that around very quickly it would enable the project to finish that part ON TIME. Meaning the rest of the project could potentially also finish ON TIME. Being on time, on budget, and pleasing the customer with the end product, is the trifecta of “this project succeeded”. That’s what we all want, successful projects.
While many teams ask the security team for feedback constantly, there are others who hide stuff from the security team. When other teams hide things from us, it’s often because we take so damn long to provide feedback and/or the feedback we provide is not helpful. I suspect that if security teams that gave fast feedback, regularly, would receive more requests. “Hey, we’re planning on doing XYZ, any chance we could run our design by you?” is a sentence I dream about hearing. When the other IT teams come to us, instead of us chasing them around, we have created a trusting relationship!
Tip: Make a list of all the things your team does. Then split it the list into two: projects and operations. Once you have a list of all the operational activities that your team is responsible for, it’s easier to allocate time and resources. You can also then show your boss just how many initiatives that your team is supporting for your organization. This creates visibility of your work, which is important when it comes time to allocate budget.
Make your work visible to management.
The Third Way of DevOps: Taking time to improve your daily work.
When I think of The Third Way, I try to apply it not only to MY daily work but also to ‘other people’s daily work that I affect’. If I can take an afternoon to tighten up the configuration on a tool, to remove some false positives it was spitting out, this could potentially affect the daily work for several of my co-workers (usually developers). If I spend 4 hours doing this, but it saves about one hour of time for each of my 100 developers that year, that’s a fantastic return on investment (ROI).
Another way in the past that I have applied this principal is by creating 1-page ‘best practice’ documents for various technologies. If one project team is building a serverless app, and I need to give them some guidelines, why not reuse those guidelines next time? Plus, for every new project using that same technology within our org? We could provide it even if they didn’t ask for it, we have the options to provide best practices information by default. And in my case, as an independent consultant, author, community manager, and public speaker, why not share that research in a blog, conference talk, or book? Why not share this work with as many other humans as possible, so that, as an industry, we can ALL move forward? (You don’t have to think that big, but, as usual, I digress.)
When we are putting operations first, before we work on projects, you might think that The Third Way is not in line with this thinking, but it is! It’s about finding better efficiencies for when we are performing operations. Improving what we do, day in and day out, so we do a better job and/or we can do it faster, from then on. It’s an investment in improving our organization’s operations as a whole, going forward. We are improving our our futures.
Tip: Double your time estimates. A boss told me this long ago, and at the time I thought he was nuts. The idea with doubling your estimate is that 1) technical folks are famous for underestimating how long something takes to do and 2) if you finish early you look like a rock star! This only works if you are the only person to double it though, I once had a boss who doubled my estimate, and his boss also doubled it, and by the time it got to the big boss it looked like it was going to take 6 months for me to make two windows forms… That was not so good…
– ME
Back To Operations
Back to the topic at hand: putting operations first. If you are not able to get through your inbox, you probably shouldn’t take on another new project. If you have several other teams waiting on you, for a process that your team is forcing them to go through, you should likely not purchase yet another tool. If you don’t already have your current toolset fully operationalized (for example, having a SAST and SCA scan performed on every PR, as opposed to manually performing 1-off scans from time to time), then you are not ready to add yet another security step to the SDLC. If you aren’t meeting your current operational requirements, you cannot (successfully) take on new projects.
Last tip! Help your teammates, especially those with less experience and seniority, prioritize and reprioritize their work, often. I’ve seen many people become a bit lost, or feel overwhelmed, because their list has gotten out of hand. Often there are things on the list that you, as their boss, are completely unaware of. Take that stuff off, and go talk to the other managers who are trying to offload their responsibilities onto your team…
– Me, again
If every single day you never finish your work, if your inbox has people writing you multiple times asking the same question because you still haven’t answered, if you feel like you are drowning at work; it’s time to look at your operational capacity, and make sure you haven’t over allocated yourself or your team. It’s always better to do a fantastic job of your current responsibilities, than to have several unfinished projects and really frustrated stakeholders.
I hope this helps you prioritize your daily work.
Wow, very well explained