Category: Uncategorized

Operations First!

Posted on by 1 Comment
Man at computer, performing operations

Many years ago, when I was a software developer, a very smart boss said to me: “Tanya, it’s always operations first. Projects after.” At first, I was confused, how will I make any progress on my projects if I’m always doing operations? And, what the HECK is “operations” anyway? Reader, this was an incredibly important lesson that has helped me countless times, throughout my entire career.

Man sitting in front of computer, implying he is performing operations

At We Hack Purple (WHP), we have a rule: “operations first, then everything else”. Operations means all the stuff you already regularly do, that people are counting on you for. For instance, at WHP, every single week there’s a newsletter. If it’s late, our subscribers ask where it is. Subscribers expect it and enjoy receiving it. It is one of the services that we offer, and it’s part of our general operations.

Other examples of WHP operations: running payroll, answering support requests from students in our academy, emails from active clients, accepting/approving new community members, and moderating our online community if someone acts inappropriately. Imagine if my team and I were “too busy” to run payroll, or ensure students could log into their accounts, or to let new people into our community? It would become a huge bottle neck for the business, and WHP would be known for leaving people disappointed.

When I was a software developer, ensuring that all bugs were fixed, customer problems/complaints were addressed, all our apps were up and running, and that my entire team knew what they needed to do (and had the information/access/resources to do it), meant that then I could work on my projects. Ensuring people weren’t waiting on me not only meant I ran a smoothly running shop; it got me promoted. Multiple times!

Examples of Application Security ‘Operations’:

  • Attending project kick off meetings to make yourself known to the team
  • Provide security requirements for all new projects
  • Performing threat modelling sessions, and completing the paperwork afterwards
  • Following up on unfixed bugs
  • Checking in with your security champions, every month
  • Answering questions from… Everyone.
  • Running scans, reviewing scan results
  • Arranging pentests, reviewing results with dev team
  • Reporting up to management
  • Being ready, should a security incident occur

Recently I had a conversation with a client who was trying a new project management methodology, and we were talking about how to best implement it at their org. After about 10 minutes of discussing, I said “What about operations? Sounds like you’re not getting your everyday work done. If you can’t even finish up the close out of a security incident from 6 months ago, you don’t need a new project management system. You need to stop over-allocating the people on your team. Ideally, operations should take somewhere between 25-50% of your time, but it sounds like you have a lot of not-quite-finished work items. Get all that done, then start on new projects. And ensure you save time, every day, for operations.”

Note: it was 25-50% of the time *for her team* and the responsibilities they had, to run operations. For your team it might be higher or lower. When I was a software developer, I was told to never allocate a resource above 80%, because something always comes up. And they were right!

Be prepared, always add 20% to your time estimates for software projects. You’ll thank me.

If your team is supposed to review the architecture for every single software project, and you have allocated zero time for it, how do you think that’s going to go? It’s not going to be good, that’s for sure. It sounds obvious when I lay it out like that, and you might think “I would never do that”, but guess what? I see companies do this ALL THE TIME. And I know I have been guilty of this in the past, not realizing I had done it.

Any security activities that you want to do as part of the system development life cycle (SDLC) are part of your team’s operations. If there are documents to review, meetings to attend, scans to run, whatever, you need to ensure you have the capacity to perform these activities as needed. You can’t say “You must complete this 20-page architecture document, then receive our approval, before you start your coding phase” then proceed to make them wait several weeks for feedback from your team. Or… I guess technically you CAN do this, but it will cause a lot of problems, frustration, and delays for other teams. *Note: I would not recommend this strategy to friends.*

Then my client and I got into a discussion about the 3 ways of DevOps (as per The Phoenix Project and The DevOps Handbook), with the 1st way being “Emphasize the efficiency of the entire system”, the 2nd being “Fast Feedback”, and the 3rd way “Taking Time to Improve Your Everyday Work.” I LOVE DevOps, and in my opinion, The Three Ways are rules to live by if you work in IT.

I know I’ve talked to about The Three Ways of DevOps a lot, but they add value in SO MANY situations! I just can’t help myself, they are just SO GOOD.

Unplanned Work: There will always be external forces that you cannot control. A security incident will happen. The big boss decides your team is going to run a tabletop exercise. Suddenly your team is in charge of something it never was before. If you have allocated your team’s resources already at 100%, there’s no space for this. And unplanned work is (unfortunately) a fact of life. One of the things you can do in such a situation and ask ‘the powers that be’ which thing will come off of your plate if you add this new one. If they say nothing comes off, explain that you team is already fully allocated, and you expect them to start slipping on other projects or operational requirements. At least you warned them, right?

This might not make you popular, but it WILL ensure your projects succeed, on time.

The First Way of DevOps: Emphasize the efficiency of the entire system.

If security teams around the world took this to heart, people would like their IT Security co-workers a lot more. I have heard hundreds of times “We make them fill out all these forms, then we don’t have time to read them.” So…. Did you stop making them fill out the forms? “No.“

If instead the security team adopted the model of “operations first”, they would either 1) make those forms way less complicated and time consuming, and 2) allocate enough time for their team to properly review them, promptly. It is my wish that security teams would look at all the inputs (forms, meetings, documents, and so on) that they ask for from other teams, and then ensure they have capacity to use those inputs to their fullest, in efforts to protect their organizations. This might mean reducing risk, adding additional layers of security, preparing for potential disasters, etc. Security teams demanding other teams to perform work, and then not fully utilizing the work they asked for, makes me very upset. I used to be a software developer, and I have been put through the paces by a lot of management in my time, and I’m quite tired of filling out templates that no one ever reads… And I know I am not the only person who feels this way!

            Tip: whenever possible, make services from your organization self-service. If you can provide all the guidance on a wiki page, do it. If you can create a portal where all the developers can access your toolset for them, do it. If you can set it up so that you already have a pre-existing contract with a pentesting company, and they just need to call and book the dates, do it. This results in saved time, less friction between teams, and operations moving much more smoothly!

A great tip, from me to you!

The Second Way of DevOps: Fast Feedback

I tend to add onto this phrase and change it from ‘fast feedback’ to ‘feedback, that is accurate, and gets to the right person/people, fast”.  Who cares if the feedback is fast if it never gets to its intended destination? Or it’s completely inaccurate so it sends someone on a wild goose chase? That is not helpful.

This is another area where if we do ‘operations first’ that we will see some big benefits. Whenever a project team asks for feedback from the security team, if we turned that around very quickly it would enable the project to finish that part ON TIME. Meaning the rest of the project could potentially also finish ON TIME. Being on time, on budget, and pleasing the customer with the end product, is the trifecta of “this project succeeded”. That’s what we all want, successful projects.

While many teams ask the security team for feedback constantly, there are others who hide stuff from the security team. When other teams hide things from us, it’s often because we take so damn long to provide feedback and/or the feedback we provide is not helpful. I suspect that if security teams that gave fast feedback, regularly, would receive more requests. “Hey, we’re planning on doing XYZ, any chance we could run our design by you?” is a sentence I dream about hearing. When the other IT teams come to us, instead of us chasing them around, we have created a trusting relationship!

            Tip: Make a list of all the things your team does. Then split it the list into two: projects and operations. Once you have a list of all the operational activities that your team is responsible for, it’s easier to allocate time and resources. You can also then show your boss just how many initiatives that your team is supporting for your organization. This creates visibility of your work, which is important when it comes time to allocate budget.

Make your work visible to management.

The Third Way of DevOps: Taking time to improve your daily work.

When I think of The Third Way, I try to apply it not only to MY daily work but also to ‘other people’s daily work that I affect’. If I can take an afternoon to tighten up the configuration on a tool, to remove some false positives it was spitting out, this could potentially affect the daily work for several of my co-workers (usually developers). If I spend 4 hours doing this, but it saves about one hour of time for each of my 100 developers that year, that’s a fantastic return on investment (ROI).

Another way in the past that I have applied this principal is by creating 1-page ‘best practice’ documents for various technologies. If one project team is building a serverless app, and I need to give them some guidelines, why not reuse those guidelines next time? Plus, for every new project using that same technology within our org? We could provide it even if they didn’t ask for it, we have the options to provide best practices information by default. And in my case, as an independent consultant, author, community manager, and public speaker, why not share that research in a blog, conference talk, or book? Why not share this work with as many other humans as possible, so that, as an industry, we can ALL move forward? (You don’t have to think that big, but, as usual, I digress.)

When we are putting operations first, before we work on projects, you might think that The Third Way is not in line with this thinking, but it is! It’s about finding better efficiencies for when we are performing operations. Improving what we do, day in and day out, so we do a better job and/or we can do it faster, from then on. It’s an investment in improving our organization’s operations as a whole, going forward. We are improving our our futures.

            Tip: Double your time estimates. A boss told me this long ago, and at the time I thought he was nuts. The idea with doubling your estimate is that 1) technical folks are famous for underestimating how long something takes to do and 2) if you finish early you look like a rock star! This only works if you are the only person to double it though, I once had a boss who doubled my estimate, and his boss also doubled it, and by the time it got to the big boss it looked like it was going to take 6 months for me to make two windows forms… That was not so good…

– ME

Back To Operations

Back to the topic at hand: putting operations first. If you are not able to get through your inbox, you probably shouldn’t take on another new project. If you have several other teams waiting on you, for a process that your team is forcing them to go through, you should likely not purchase yet another tool. If you don’t already have your current toolset fully operationalized (for example, having a SAST and SCA scan performed on every PR, as opposed to manually performing 1-off scans from time to time), then you are not ready to add yet another security step to the SDLC. If you aren’t meeting your current operational requirements, you cannot (successfully) take on new projects.

            Last tip!  Help your teammates, especially those with less experience and seniority, prioritize and reprioritize their work, often. I’ve seen many people become a bit lost, or feel overwhelmed, because their list has gotten out of hand. Often there are things on the list that you, as their boss, are completely unaware of. Take that stuff off, and go talk to the other managers who are trying to offload their responsibilities onto your team…

– Me, again

If every single day you never finish your work, if your inbox has people writing you multiple times asking the same question because you still haven’t answered, if you feel like you are drowning at work; it’s time to look at your operational capacity, and make sure you haven’t over allocated yourself or your team. It’s always better to do a fantastic job of your current responsibilities, than to have several unfinished projects and really frustrated stakeholders.

I hope this helps you prioritize your daily work.

#WeHackHealth Getting Better Sleep

Posted on by Leave a comment
Tanya building garden beds

If you’ve been following the #WeHackHealth hashtag, quite a few people who work in the field of information security have been sharing health tips, encouraging each other to focus on their own health, and showing progress reports on their efforts. Several people I know have been following it closely, participating, and reaping the benefits of this wonderfully positive movement. Started by @HackingDave, this use of social media to encourage others to live healthier lives is something I have been wanting to contribute to for quite a while, but I wasn’t sure quite how I could add value. That is, until my most recent trip to San Francisco for #RSAC 2023!

A summary of this blog post is available in PDF format here.

Tanya, posing with her freshly planted seedlings.

I’ve been struggling with poor sleep since my teens (that’s 30 years, for those who enjoy math). During my 20’s I used to stay up until at least 1:00 am most weeknights, then getting up at 6:30 to 7:00 am to go to work. The weekends were worse. I was part of the local music scene in Ottawa, playing at live music clubs several times a month, and would often be out until 2:00 am, 3:00 am, or even later on the weekends. Sometimes my fans would wait until the club closed (2:00 am) I would put my drums/guitars/whatever back at my place, then take me out to dinner, at 3:00 am. This whole time I worked full time as a software developer as well, doing the 9-5 routine. My thought was “If I’m not going to be asleep anyway, why bother laying in a bed being bored when I could be out having fun instead???” In my early 30’s I was slightly less ridiculous, until I met a doctor who asked me what my “sleep hygiene” was like. I had no idea what that was. He suggested that if I got better sleep that it could help with other issues I was having, and I set upon a path to get better sleep.

To be quite clear: I walked around like a complete zombie until around 11:00 am every day. I was an auto-pilot, and since I found coding easy and fun… It didn’t bother me. I zoned out, into the code, and worked until lunch…. I can’t imagine how I must have seemed to my co-workers, 1/2 asleep, trying to get work done, I must have looked a mess.

Sleep-deprived-Tanya

Before I get any further down this path, I’d like to inform you that I am not a doctor. I’ve never been to med school, or studied medicine in any way. In fact, I’ve never even played a doctor on TV (I know, so lame!). You should not take any of this as official medical advice, this is just what I have learned through lots of (non-professional) research, trial-and-error, and personal experience. Please talk to your doctor before trying anything with an astrix (*) beside it. Most of this stuff is harmless, but I will put the little * if I think you should ask your doctor first. Feel free to ask your doctor about anything anyway! In summary: I am definitely not a doctor, but just a regular person who hopes sharing her sleep journey might help you get better sleep.

WTF ‘Sleep Hygiene’?

Sleep hygiene means setting a time to wake up and go to sleep, every single day, and sticking to it. If you have children, or remember being a child, they usually have a “bedtime”. For some reason, as we become adults, we tend to throw this idea away. I told the doctor that suggested it that I never slept anyway (literally 2-3 hours a night, but sometimes as much as 5 hours. Yay?) but he insisted I try for it for 3 straight months, and I thought “WTF not?”.

His instructions: go to bed and wake up at the exact same time every single day, weekend or weekday. Give yourself 9 hours or more. Do not deviate, even if there’s a “super cool party”. * I might have asked if it was okay to skip this for parties and he gave me a serious frowny-face…

For the next 3 months, I went and laid in my bed at 11:00 every night, and forced myself to get up at 8:00 every morning. I did not think it would work. But it was SUCH A GIANT IMPROVEMENT (after a few weeks of being diligent). I did more than just this, but I started sleeping more hours. And for the first time, I started to feel drowsy around 11:30. And my other health condition improved noticeably. #WIN

Caffeine, Addiction, and Timing

Caffeine is a drug that a large portion of North American adults are addicted to, but it doesn’t have to be this way. I’ve had lots of friends who drink several coffees a day (multiple pots, in fact), they can’t sleep, but they also tell me “It doesn’t really affect me”. If caffeine doesn’t affect you, why are you always consuming it? This does not add up.

If you feel really tired, sleepy, or have ‘brain fog’ later in the day, it might not be that you are genuinely tired, it *could* be that you are having caffeine withdraw. Once I stopped having caffeine, I noticed I didn’t need it anymore. I didn’t have almost any caffeine (just de-caf tea or herbal teas) for a few years, and felt way better. I do drink it now though, but I stop early afternoon, no matter what.

If you love coffee, tea, diet cola, or whatever, that’s okay. But you need to only consume it at certain times if you’re having sleep issues. If you work the regular 9-5, I suggest no more caffeinated drinks after lunch, or just have decaf from then on. I personally don’t usually even have decaf past 1:00 pm (there is caffeine in decaf drinks, it’s just less!), but you can find your own rhythm that works for you.

Lowering the Lights (Dimmers are the best!)

I read a book called “The Primal Blue Print” By Mark Sission, and I loved it (and yes, eat paleo and do all the stuff he says). Then did what I always do, read every other thing the author ever wrote. “The Primal Connection” is a book about reconnecting with our bodies and nature, and one of his suggestions was lowering the lights, and removing blue light, when the sun sets.

You can get dimmer switches and change out a bunch of your lights in your house or apartment for a couple hundred bucks (I know, not cheap!) but I have found it worth the expense. If you LED or CFL lights, it’s important you buy the ones that are dimmable and “warm” temperature, otherwise they will flicker and be really annoying, or make a buzzing sound (also annoying). I walk around the house at a certain time and lower the lights. Everyone in the house starts chilling out. I usually do this around 9:00 pm, but do what’s best for you.

Also, I don’t mean walk around in the dark. I mean turn them down to 70% or 60%. So that you feel a bit relaxed. It will make sense over time which amount of dimming is best for you.

Amber/warm versus bright blue daylights

When you buy lightbulbs lots will say “bright White” or “daylight”, or “blue white”, those are great for an office, bathroom, or your kitchen, where you want to be fully awake and alert.

Some lights will say “Warm” or “Amber” lights, they are great for your bedroom, living room, dining room, anywhere you want to relax and wind down.

I typically use these to try to get myself more awake or more relaxed/wind down for bed. If I work late in the living room we have a special extra light that my partner setup, to hep me concentrate on my writing. It works like a charm!

Blue lights/screens

TV and phone screens often come into our bedrooms with us. All of them are able to display most colours, including blue, which tells our brain “WAKE UP IT’S DAYTIME”. There’s a setting on your phone where you can have it slightly dim the screen, and remove most of the blue light, when the sun sets. Doing this will help you sleep, and you can automate it easily.

For televisions, this is harder. I’ve seen people who buy funky orange lensed-glasses and wear them in the evening to remove the blue light themselves, but I am not personally a fan. If you watch TV via a computer/stream, some of them have settings that allow you to change and remove the blue, but not all. I used to have a raspberry pi that did this for me, but I just got a roku and I’m not sure if I can do that with it yet. Check your own devices if you have this option.

Complete Darkness

Sleeping in *complete* darkness helps me get very deep sleep. I have blackout blinds on all my bedroom windows, no visible LEDs, and we turn off the lights in other rooms so that they don’t shine through under the door. I take this very seriously, and travel with black electrical tape so I can cover all the lights in hotel rooms. I have received feedback from my significant other that this one change made a huge difference for their sleep. Removing all lights is worth the effort!

Sun Lamps

I am one of many people who are affected by Seasonal Affectiveness Disorder (SAD), sometimes known as “seasonal depression”. If you don’t know what it is, basically I get really bad brain fog every winter. It’s hard to concentrate, and I feel “down”, for months at a time. I remember my grades used to plumet in the winter semester, and soar in the spring… It’s not real depression, it’s much less serious. That said, it still sucks, and I moved across Canada just so I can avoid this situation as much as possible. (You can read more on SAD here)

SAD is caused by not enough sunlight. Our bodies NEED it. You can treat SAD by taking vitamin D, getting lots of sun in the winter (for instance, taking a vacation, or moving to a less-wintery-place if you’re me), and using a Sun Lamp.

A sun lamp generally has to give off 11,000 lumens of light or more, and sometimes they are bright white, or blue light. You need to sit in front of it for 15-30 minutes (depends on the model) every morning, ideally as soon as you wake up. I’ve been doing this every winter, since I was 23, and these lamps changed my entire life.

Note: these lamps are great for treating jet lag, SAD, or just helping you get better sleep. You are telling your body “HEY, this is MORNING”.

Word of caution: do not use these lamps at other times of day. It will just keep you up and mess up your sleep. I’ve seen people say “Oh, I forgot this morning, I will do it after work” NOOOOOOO. Do not do that. It will not make for a fun night of sleep. :-/

Magnesium *

Magnesium is a type of salt that is really good for us and if you are; a woman over 40, someone with chronic pain, someone with (list other symptoms), it’s advised that you take it.

That said, it’s ALSO good for sleep! I take a small amount with water before bed, and it even kinda tastes good to boot.

Note: if you take too much magnesium you will have “exciting” trips to the bathroom. Start small and work your way up to the full dose over several days or even weeks.

Sleep Rituals

You might not realize it, but many of us have rituals we perform every day. We have a “get ready for work” ritual, a list of specific things we do in order to feel “ready”. Often, we also have nighttime rituals, to help us get ready for bed, whether you consciously realize it or not. Most of include brushing our teeth, journaling, turning off all the lights in the house, locking the doors, saying “goodnight” to people you live with, etc.

Years ago, I had a friend who had TERRIBLE nightmares. She feared going to sleep, and would often stay up as late as possible to avoid this vivid and awful dreams. We talked about it and I asked what her ritual was before bed and she said she didn’t know. She didn’t have one.

It turned out she did have a bedtime ritual, but it was the opposite of helpful for her. She would watch TV to try to avoid going to bed, and worry about what she would dream about. She would treats, to try to calm herself. She would “keep herself really busy” until she would fall into bed. This was NOT working for her.

Together we came up with a new one for her:

  • Herbal tea instead of sugary snacks
  • Calling a friend when it’s not too late, so she can have a nice conversation and remember there are a ton of people in her life who love her
  • Journaling all of her worries, then locking it away into a drawer
  • Stretching (I gave her a bed time yoga video I used to do every night)
  • Reading something non-scary in bed, instead of TV or phone
  • Lowering the lights 2 hours before sleep

Although her nightmares did not stop completely, it went from “pretty much every night” to “once a month” and “I don’t always remember them”. She started sleeping WAY better. She was also happier. One top of that, she took the lighting item to a whole new level and redid all the lighting in her entire house and has inspired me in decorating every place I have lived ever since!

Journals/Lists

Some of us don’t sleep because we are worrying. Worrying about work. Worrying we don’t have work. Worrying about money. Worrying about our loved ones. Worried no on loves us. Etc. This is NOT good for sleep. I personally worry about having too many things to do (and that I might forget one) and/or missing a flight. I’m SO WORRIED about flights. Sigh. We all have our hang ups.

Anyway, one way to get around this is to write everything in your head into a journal. It doesn’t even need to make sense. Getting it out is what matters. Whenever I wake up in the night concerned about something, I tried to write it down. Then I always fall back to sleep so easily. I have seen this work for several people in my life, including children! Just the act of writing it down can make us feel better when we are upset, even if we never read it again. Even if you are not upset, just making a list of what’s in your head can help…

  • Gotta sign the kids up for summer camp
  • Can’t forget the gas bill
  • Did my friend apply for that job or not? I’m going to ask. She might need a nudge.
  • Don’t forget to call your mom this weekend!

Bedrooms are only for 2 things, and one is way more fun than the other

“Bedrooms are for sleeping and sex. Nothing else.”

Mark Sission, author of the Primal Connection and other amazing books

I remember reading this in the primal connection, and thinking “Gosh, I want to live at your place.” But I used to do everything in my room… I would play guitar in my room, read in my room, whatever. If I wanted to avoid roommates, my room was the place to be, rather than the common room. Once I changed it to “only two things happen here”, it sounds weird but I go in and I know that’s what I’m doing. I’ve added “Get dressed” and “put away laundry” to the list, but I try to not do general activities there, and instead use other spaces. It helps my family members know I’m not avoiding them, and it sets the mood for sleep.

Note: If you live in a bachelor apartment or have a bunch of roommates, just ignore this one. This is one of those “if you’re lucky enough to have space for” rules.

Jetlag

I have travelled all over the planet and I’m pretty “good” at Jetlag now. Using a sun lamp, and following the eating windows in the section below can really help. I also sometimes “cheat” and take a sleeping pill to make myself sleep at the correct time the first day, and I force myself to have breakfast first thing in the morning in my new time zone even though I hate eating breakfast. But I need to tell my body “I am breaking the fast” and “this is morning now!”.

Diets and Eating Windows *

Our bodies are not meant to eat every moment of every day. When we eat at weird times, it can (negatively) effect our sleep. I am as guilty as the next person of having a snack in the evening lately, but if you are having a lot of trouble sleeping and you snack at night, I suggest reading this book: Your Circadian Code. Although the author jumps over into “this is how you can lose weight” a bunch, if you can ignore that part, the rest is REALLY GOOD. And, if you’re trying to lose weight, this could be a double whammy for you. The guy who read for the audiobook has a really nasal voice, but if you can get over that it’s not very long, and all of it was very helpful for me.

Meditation

Mediation is staying still and attempting to clear your mind and just observe your thoughts, body and breathing. I used to think it meant ‘trying hard to think of a specific thing’, concentrating very hard. But that’s not true, not really.

Meditation has been linked to all sorts of excellent health benefits, both physical and mental, such as lowering stress and anxiety, reduced chronic pain, more patience and calmness, happier outlook on life, etc. AND it can help with sleep!

If you meditate regularly, it can help you clear your mind so you can go to sleep. Start listening to your body, calming down your sympathetic system, and you’re miles ahead in getting to sleep fast. Regular medication can help with your sleep overall as well!

Hypnosis *

There are all sorts of hypnosis recordings and psychologists you can do it live for you, that can help you sleep. They hypnotize you, then tell your that sleep is your friend (not literally, but basically, they make you believe that you can sleep, you should sleep, and you will sleep). I used hypnosis years ago to help me stop drinking cola. It worked. For 5 years. Until I started working at Microsoft and travelling all over the world and I needed caffeine to power through various travels. 5 years is pretty good!

Comfortable bed – note, it might be way harder than you think!

I used to have a really soft bed. When I went to the store to buy it, I laid down on the soft bed with my friend and we both agreed, it was super soft and comfy. But then it made my back hurt, and I was very confused about it.

When Is started travelling for work all the time I got to stay in lots of different beds, in different hotels. I decided the Marriot’s beds were the best! I learned I could buy it cheaper direct from a place that sells beds, rather than the hotel, for about half the price. And I learned they choose FIRM beds. I thought that would hurt… but it’s SO MUCH BETTER. So if you have back pain and a really soft bed, consider trying out a very firm bed. Having a comfortable bed is really, really helpful.

Snoring and Sleep Apnea *

I got tested and I have incredibly mild sleep apnea. The doctor told me to sleep on my side and I would never snore again, and hug a pillow if it hurts my shoulders. I only sleep if I’ve had a glass of wine I swear! But I digress.

If you snore a lot, you likely have sleep apnea. I’m not saying you are doomed, or that you need to immediately get a C-PAC machine. But when you’re snoring it’s because you can’t quite breath exactly how your body needs to breath. This interrupts you sleeping in tiny internals.  The louder and more irregular your snoring, the more likely you are getting CRAPPY sleep. If you know you snore, and you feel really tired when you wake up, even though you should have had “enough” sleep, you likely have this going on. There are a bunch of options, and a doctor or sleep clinic can help you fix this!

Carbs & Sugar near bedtime

I love candy and sugar. I wish it wasn’t true, but it is. I definitely want to have a sugary treat before bed, every night, but it’s not helpful for my sleep. It’s likely not helpful for your sleep either. If you are an evening snacker (and I’m not saying you are!), consider not snacking after dinner for a week or so and see if you’re getting better sleep. Might be a habit worth breaking!

Massage and/or physical affection

Also in the Primal Connection book was the idea of human touch and affection. I come from a very affectionate family; we hug each other all the time. I’ve always been “touchy feely”, but not everyone is. The book pointed out that human touch is actually a need, not a want, like I had thought. In the book the author suggests that the reader “just have sex”, which is all well and great if you have that option available to you at the time, but we don’t all have a special someone just waiting to supply us with all the sweet loving we need, whenever we want. Way to make me feel inadequate Mark! (just kidding, I think the author is awesome)

As an alternative, you can get a massage or acupuncture, you can hug a friend, you can get a pet (not the same, but still helps make humans happier), you can play a high contact sport like ball hockey, do acrobatic yoga, and more. If no one has touched you in months, this is something you might want to look at. I’m not saying this to cast judgement or make anyone feel bad. I’m telling you this because it might improve your life, and every human deserves happiness.

General Health, Weight, Stress, and Happiness

Prepare for some really obvious advice, that I didn’t always understand. If you already know it all, cool! If not, also cool! We do not need to be perfectly healthy every moment of every day, but there are things we can choose to limit, to reap big benefits. When I dropped sugar, alcohol, processed foods and gluten from my life for 5 years, every part of my body was great. My hair was softer. My skin was perfect. My sleep improved. But you don’t need to be very, very strict in order to benefit; I’ve loosened up over the years on some things. Below is a list of places you could “be more healthy” and for each one you do, you will not only sleep better, there will be other great benefits too!

  • Alcohol is very bad for our bodies. I know it’s socially acceptable and “everybody’s doing it”, but you don’t have to, or you can just have some on special occasions. Having less (or none) will make you a healthier person, full stop. Also, all those news articles proclaiming that “having a glass of wine a day is good for you” are complete bullshit and the studies they based it upon where incredibly biased. I will definitely have an internet argument about this if you want!
  • Sugar is bad. Not as bad as alcohol, but it’s also in way more foods and still total garbage for us. It’s SNEAKY, especially in the United States. Read the ingredients. Having less sugar will also help you be healthier.
  • Processed food is bad. It usually also has sugar, salt and chemicals. Having whole foods instead of processed foods will mean way more nutrients (to power your amazing brain and body) and less sugar and salt. Whole foods means eating vegetables with butter and spices, or salad with oil and herbs, or meat that you’ve grilled. It’s not something that has a list of ingredients.
  • Eat LOTS of veggies. LOTS. Eat tons of veggies and your body will thank you.
  • Spices, herbs, especially turmeric, are your friend. They contains tons of stuff that’s good for us (nutrients, vitamins, etc) but they also make food taste better. Then you can have very tasty meals, of unprocessed foods. Spices and herbs are the secret!
  • Lifting heavy things and sprinting is good. The paleo folks have lots of mixed feelings about cardio, but basically every health expert agrees that moving around often, lifting heavy stuff sometimes, and sprinting once in a while, is good for us. Think: playing sports once a week, walking to and from work, and lifting weights once a week. This recipe can be very easy to stick to, be really fun, and keep you lean and trim.
  • Regular cardio can be quite bad, which I found surprising. Instead focus on “movement”, often. Plus play and have fun! Seriously. This is paleo wisdom, and I gotta say, I agree with it. I used to do the “tons of cardio” thing, and it never really worked for me. It spikes your cortisol (I have enough already, thank you) and it’s nowhere near as fun for me as playing sports, doing a yoga or pilates class, ‘playing’ in my garden, or goofing around with my kids at a park. Make your “exercise” a fun part of your life, and you will be fit forever.
  • Me and my weird walking desk: I have a walking desk, I really like it. I use it whenever I have a meeting where I just need to listen (think: team meeting). I used to use it a lot more than right now, because I cannot create content and walk at the same time. But I can listen and walk easily. If you’ve thought about getting one, they are now cheaper than ever before. And you don’t need to walk all day! If you walk one meeting per day, you’re awesome!
  • Grounding is good. I thought this was “total crap” when someone first suggested that I “touch dirt”, but over the years I have grown to love gardening so much that I now own a small hobby farm and grow a lot of the food my family and I eat. It works for me, and might not work for you, but as a self-described ‘city-slicker’ and tech worker who lived downtown and was surrounded by concrete most of her life, clearly it had it’s effect on me.
  • Filling your own cup. Doing things for yourself that bring you joy and comfort. THIS is important. We cannot just work and do things for others. If we do not take care of ourselves, we will have nothing left for anyone else. This can mean making a piece of art, writing a story or blog post, joining a sports team, having a great big laugh with a good friend. Your happiness greatly affects your overall health. No joke!

For those of us that travel on airplanes, often

  • WATER! Drink a lot of water. Note to self: Coffee is not water. Neither is diet soda.
  • Walking: walk around the airport a lot, rather than more sitting. It will help ensure you don’t get swollen ankles on the plane, but also help you feel better later.
  • Compression socks and more: if you are on a plane often, wear compression socks or even compression outfits.
  • More water. Seriously.
  • Do calf raises and any sort of neck/shoulder stretches/movement, at least once, per trip. It prevents blood clots and will make you feel way better.
  • Don’t sit on those crappy chairs at the gates if you can avoid it. They are uneven, so that your hips tilt back, which makes your head no longer even, so then you move your head forward. In summary: they are very bad for your back, neck and posture.
  • Carrying your own food, that doesn’t suck: I often bring protein powder that is high in fat & collagen as well, plus a shaker when I travel. Then I always have a food option. As a person who is sensitive to gluten and all sorts of other stuff, it’s hard to find food in airports for me. Bonus points: put two plastic bags around it. I had mine explode once and I smelled like a chocolate milkshake for the rest of the trip. I was not impressed, although my colleagues found it pretty funny. “Why does this elevator smell like…. A chocolate milkshake?”

Further reading:

Thank you for reading.

Preventing Secrets in Code

Posted on by 1 Comment
Tanya in BC

When I started programming in the 90’s the security of software wasn’t on everyone’s mind like it is now. I took no security classes in my 3-year college computer science program, and it never even came up as a subject. I was taught to save the connection string for each different environment in the comments in your code, so it was easier for the next programmer to find them. It wasn’t until 2012 that someone ran a web app scanner (also known as a DAST – dynamic application security testing tool) on one of my apps. I didn’t understand a word of what I read in the report at the time. When I switched from programming to penetration testing, and then onto application security, there was quite a big learning curve for me.

Tanya Janca, in British Columbia, Malahat

Back to the Secrets

Secrets are what computers use to authenticate to other computers. For instance, an application sending a connection string to a database is its way of asking “I am this specific web app, please let me query your database.” When the database connection works, that’s the database’s way of saying “Sure thing!” Computers don’t have eyes, ears, or brains, so they can’t ‘recognize’ someone like humans can; they have to use secrets.

A secret can be a password, an API secret, a certificate, a hash, a connection string, etc. Most importantly: they should not be shared and should only be saved into your secret management tool. But I am getting ahead of myself.

This is a talk I gave in April 2023 at #Bsides San Francisco, “Hunting Secrets”. Similar topic!

Memories

When we save secrets into our code it is possible for another programmer to come along and use that secret; for better or for worse. They can login into your database, connect to your API, or anything else that the secret can be used for. Sometimes this can seem quite helpful, for instance if a client forgot their password when I was a programmer I used to log into the database, grab a copy of their password, use our decryption tool, and tell it to them over the phone. My whole team used to do it. Now I know that it’s more secure to have the user receive a password link in their email (to validate they are who they say they are), that the client’s password should have been salted and hashed (a one-way cryptographic method), and that the password to the database should have been kept in a secret management tool (making it unretrievable for human beings). Secrets in our code allow for all sorts of potential attacks, breaches, and embarrassments.

Finding Secrets

If you want to find out if you have secrets in your code, you can use a tool called a secret scanner. There are many on the market, and many of them are free. They use a variety of ways to try to find secret, but most commonly they use REGEX (regular expressions) to look for entropy (extremely long and random bunches of characters) and key words (password, secret, key, etc.).

When I work somewhere doing AppSec, I try to get read-only access to the code repositories as soon as possible (for many reasons, not just this). Once I have it, I download all the code, from all the projects I can, in a zip. I unzip it, point my secret scanner at it, and then settle in for a few hours to go hunting around in the code. Putting on music and getting a tasty warm beverage (hot chocolate anyone?) can make this a more enjoyable activity. It’s not exactly riveting.

Start by looking at the first finding. Sometimes it’s something really obviously bad, such as:

Password=”AliceandBobLearnIsMyFavoriteBook”;

That’s a secret for sure! The next step is to rotate that secret. Rotating this secret would mean changing the password to something new on the system this is used for. Then you check that new secret into your secret management tool (more on this soon), and then (the hard part) you update the code in this application to fetch the secret from your secret management tool instead and publish the updated code. Do not, under any circumstance, use the same value as the one you found. That secret has been ‘spoiled’, ‘spilled’, or ‘spilt’. It is no longer usable, as someone malicious might have it saved somewhere, or already be actively using it for malicious purposes.

You are going to need to follow this process for every secret you find. Sometimes it means regenerating a certificate, creating a new API, etc. It’s a bit of a pain, but it’s a lot better than having a data breach or other type of security incident to deal with.

Special Note: when you find a secret in the code, depending upon what you found, you may want to trigger the incident response (IR) process, to investigate as to if this secret has been used improperly. When you find a secret, you can’t know if you were the first, second, or tenth person to find it. Kicking off your IR process is a real-life application of the ‘assume breach’ secure design concept.

Preventing Secrets in the Code

Code repositories (also known as version control or ‘repo’) have several types of ‘events’ that can be used to trigger automation. When someone merges their code back into the main branch, you can automate it to run tests to verify it integrates nicely. When code is checked it, the repo can prompt someone else to review the changes before it is merged into all the other code. The event we are interested in is called a ‘pre-commit hook’.

The moment someone checks code in that contains a secret, they have spilt it. The secret will be in the history and backups and maybe even in the logs. You must rotate it. Even if you realize your mistake only 5 minutes later, the damage is done.

A pre-commit hooks allows you to run your secret scanning tool on only the new or changed code you are checking in, and if it finds a secret, it stops the check-in process. It gives the user an error message, explaining that it thinks it has found a secret, and blocks the code from being checked in. This means the secret has not been spilt; no secret rotation required! If you code does not have a secret in it, your check in continues, and any other events you set up do their thing. The test takes so little time, that is almost unnoticeable to the end user.

Secret Management

Secret Management tools did not exist when I started programming. In fact, they are somewhat ‘new on the scene’ and not widely adopted, yet. Secret management tools manage secrets for machines. They are not password managers, which manage secrets for humans. They are still fantastic though!

When using secret management tools, generally we create a new vault (an instance of encrypted secrets) per system (the application to which those secrets belong). We do this so that if one vault is compromised somehow (perhaps the vault is lost or corrupted), then only one system will be harmed. We also do this to ensure the vault is accessible by whatever system it supports; you wouldn’t want to have to open a hundred holes in your firewall so that all your systems can connect to it.

When we check a secret into a secret management tool, we say goodbye to it forever. We do not keep a copy elsewhere, because we can trust the secret management tool to keep it safe for us. It’s encrypted in the vault, and it is retrieved only programmatically (humans cannot ‘reveal’ the secret in plaintext). Your CI/CD can retrieve it, your application, APIs, etc. This means your secrets are managed in an automated way, leaving zero room for human error. Trust me, it’s a good deal!

Tips

As you follow the process of finding all the secrets, you should take note of false positives, so you can suppress them in the future. An example I ran into myself: there was a license key for a mail merge program, but the company who made the program had gone out of business years ago. This meant that they weren’t breaking any licensing agreement to use it all over the place, and they didn’t need to protect the key because it could be used as many times as they liked. That meant it wasn’t really a secret anymore. We suppressed the license key from then on.

You should create rules to avoid false positives, as it will become annoying over time if you have weird situations like the one mentioned above.

Conclusion

If you work at an organization that has a lot of technical debt, cleaning up all of your secrets can take quite a lot of time. That said, if you have an intern, co-op student, or junior application security person on your team, this is an ideal task for them. It’s lots of work, it’s easy to do, and it looks good on a resume. It also reduces the risk of your organization greatly, which is always a big win.

Happy (secret) hunting!

Why can’t I get over log4j?

Posted on by 4 Comments
Image of Tanya Janca

I haven’t written in my personal blog in a while, and I have good reasons (I moved to a new city, the new place will be a farm, I restarted my international travel, something secret that I can’t announce yet, and also did I mention I was a bit busy?). But I still can’t get over log4j (see previous article 1, article 2, and the parody song). The sheer volume of work involved (one company estimated 100 weeks of work, completed over the course of 8 days of time) in the response was spectacular, and the damage caused is still unknown at this point. We will likely never know the true extend of the cost of this vulnerability. And this bugs me.

Photos make blog posts better. People have told me this, repeatedly. Here’s a photo, I look like this.

I met up last month with a bunch of CISOs and incident responders, to discuss the havoc that was this zero-day threat. What follows are stories, tales, facts and fictions, as well as some of my own observations. I know it’s not the perfect story telling experience you are used to here, bear with me, please.

Short rehash: log4j is a popular java library used for application logging. A vulnerability was discovered in it that allowed any user to paste a short string of characters into the address bar, and if vulnerable, the user would have remote code execution (RCE). No authentication to the system was required, making this the simplest attack of all time to gain the highest possible level of privilege on the victim’s system. In summary: very, very scary.

Most companies had no reason to believe they had been breached, yet they pulled together their entire security team and various other parts of their org to fight against this threat, together. I saw and heard about a lot of teamwork. Many people I spoke to told me they had their security budgets increased my multitudes, being able to hire several extra people and buy new tools. I was told “Never let a good disaster go to waste”, interesting….

I read several articles from various vendors claiming that they could have prevented log4j from happening in the first place, and for some of them it was true, though for many it was just marketing falsehoods. I find it disappointing that any org would publish an outright lie about the ability of their product, but unfortunately this is still common practice for some companies in our industry.

I happened to be on the front line at the time, doing a 3-month full time stint (while still running We Hack Purple). I had *just* deployed an SCA tool that confirmed for me that we were okay. Then I found another repo. And another. And another. In the end they were still safe, but finding out there had been 5 repos full of code, that I was unaware of as their AppSec Lead, made me more than a little uncomfortable, even if it was only my 4th week on the job.

I spoke to more than one individual who told me they didn’t have log4j vulnerabilities because the version they were using was SO OLD they had been spared, and still others who said none of their apps did any logging at all, and thus were also spared. I don’t know about you, but I wouldn’t be bragging about that to anyone…

For the first time ever, I saw customers not only ask if vendors were vulnerable, but they asked “Which version of the patch did you apply?”, “What day did you patch?” and other very specific questions that I had never had to field before.

Some vendors responded very strongly, with Contrast Security giving away a surprise tool (https://www.contrastsecurity.com/security-influencers/instantly-inoculate-your-servers-against-log4j-with-new-open-source-tool ) to help people find log4j on servers. They could likely have charged a small fortune, but they did not. Hats off to them. I also heard of one org that was using the new Wiz.io, apparently it did a very fast inventory for them. I like hearing about good new tools in our industry.

I heard several vendors have their customers demand “Why didn’t you warn us about this? Why can’t your xyz tool prevent this?” when in fact their tool has nothing to do with libraries, and therefore it’s not at all in the scope of the tool. This tells me that customers were quite frightened. I mean, I certainly was….

Several organizations had their incident response process TESTED for the first time. Many of us realized there were improvements to make, especially when it comes to giving updates on the status of the event. Many people learned to improve their patching process. Or at least I hope they did.

Those that had WAF, RASP, or CNDs were able to throw up some fancy REGEX and block most requests. Not a perfect or elegant solution, but it saved quite a few company’s bacon and reduced the risk greatly.

I’ve harped on many clients and students before that if you can’t do quick updates to your apps, that it is a vulnerability in itself. Log4j proved this, as never before. I’m not generally an “I told you so” type of person. But I do want to tell every org “Please prioritize your ability to patch and upgrade frameworks quickly, this is ALWAYS important and valuable as a security activity. It is a worthy investment of your time.”

Again, I apologize for this blog post being a bit disjointed. I wasn’t sure how to string so many different thoughts and facts into the same article. I hope this was helpful.

Parody Songs

Posted on by 1 Comment
Image of Tanya half way through singing this song

For those who are not aware, I used to be a professional musician. I went both under my name (Tanya Janca, folk singer) and was in several different musical groups including Couchwrecked, who wrote the song Hottawa.

I just released another parody video and thought I would share it.

“Open Source Ain’t So Good”

Set to the music “You Know I’m No Good” by Amy Winehouse

“Open Source Ain’t So Good”

Reviewing my dependencies, and it hurt,

My rolled up sleeves, SheHacksPurple shirt

You say “what did I add to my app today?”

And sniffed out insecure log4j

‘Cause you’re my sec champ, my guy

Hand me your code and fly

By the time I scanned your dependencies

My tool lit up like a Christmas tree

I used open source

Like I knew I would

I told you It was trouble

Open Source ain’t so good

Open source is free, like a puppy

You Gotta check for insecurities

Just because the code is there for all to see

Don’t mean that it’s been tested thoroughly

Rush to run my SCA tool

It looks at me and 

says I’m such a fool

This package ain’t supported no more

I cried for us on the kitchen floor

I used open source

Like I knew I would

I told you It was trouble

Open Source ain’t so good

Sweet refactor, Dependencies upgrade

The app is like it was again

I’m testing it all, while you sit and wait

Us PenTesters, we never hesitate

Then I notice the results and it burns

My stomach drop and my guts churn

You shrug and it’s the worst

Who truly stuck the knife in first

I used open source

Like I knew I would

I told you It was trouble

Open Source ain’t so good

I cheated my app

Like I knew I would

I told you It was trouble

Yeah, Open Source ain’t so good

And here is a previous parody I released, last month.

.

Just Release It Anyway

Sung to Backstreet Boy’s “I want it that way”

“Just Release It Anyway”

Lyrics

Yeah

You are, setting fires

In my, applications

Believe when I say

I don’t want it that way

Your app, is falling apart

Security isn’t in your heart

When you say

Release it anyway

Tell me why You didn’t fix the bugs I found

Tell me why You Ignored the PenTest result

Tell me why I never wanna hear you say

Release it anyway

Am I your advisor?

Your one security hire

Yes, I know it’s too late

‘Cause you released it anyway

Tell me why You didn’t fix the bugs I found

Tell me why You Ignored the PenTest report

Tell me why I never wanna hear you say

Release it anyway

Our security program has fallen apart From the way we know it should be, yeah

No matter the software I want you to know It’s safety matters to meeeeeeeeeee

You are, setting fires

In my, applications

Believe when I say I don’t want it that way

Ain’t nothin’ but a heartache

Ain’t nothin’ but a mistake (don’t wanna hear you say)

I never wanna hear you say (oh, yeah)

Just release it anyway

Tell me why

Ain’t nothin’ but a heartache

Tell me why

Ain’t nothing but a mistake

Tell me why I never want to hear you say (never wanna hear you say)

Release it anyway

Tell me why

Ain’t nothin’ but a heartache

Ain’t nothin’ but a mistake

Tell me why I never want to hear you say (don’t want to hear you say)

Just Release it anyway

‘Cause I don’t want it that way

The Difference Between Applications and Infrastructure

Posted on by Leave a comment
Christian Wiediger on Unsplash

Recently someone asked me what the difference was between Applications and Infrastructure. He asked why a Linux operating system wasn’t “software” and I said it was but it’s a perfect copy… I tend to speak about ‘custom software’. We ended up talking for a very long time about it, and I thought a blog post was in order.

Photo by Christian Wiediger on Unsplash

Infrastructure is the operating systems and hardware that applications live on. Think Windows, Linux, containers, and so much more. Sometimes hardware is included in this category (depending on who you talk to), and sometimes it is not. Infrastructure is necessary to run an application, even serverless runs (briefly) on a container. Operating systems are also all standardized, and not unique in nature. For instance, if I’m running SQL server 2012 R2, and so are you, we both have the same options for patches, configuration, etc. Operating systems are software that speak to hardware. 

Applications are software that speak to operating systems, databases, APIs and anything else you can think of. There are custom applications (what I’m almost always talking about, software developed for a specific business need or as a product to sell), COTS (configurable off the shelf, like sharepoint or confluence, administered by a person or team, installed locally on a server) and regular old software that you install or access via a web browser that you use as-is (no administration required/simpler). More newly there is SaaS, software as a service, which is basically a great big COTS product, hosted by someone else (no need for you to patch or otherwise take care of it, you pick your settings and use it). 

Infrastructure usually needs to be patched, updated/upgraded, and hardened (secure configuration choices). Patches and upgrades arrive in a prepackaged format, but sometimes these updates can break the applications living on that infrastructure. Testing and sometimes downtime is required. This is why so many people say ‘patching is hard’, it is difficult to plan for testing, downtime and to ensure everything will go smoothly. 

Software, on the other hand, includes many different components that will be provided prepackaged (such as a new version of a library or a framework) but when you update them sometimes other libraries or framework parts break and/or the custom code that your team wrote can break as well. Meaning you may need to re-code or rewrite things, or update a whole bunch of things at the same time. I’ve heard developers refer to this as “dependency hell”. 

If you have just released something brand new, it’s super easy to keep it up to date. Tiny changes present less risk (which is why people love devops over waterfall), making it easier to maintain. But because it’s sparkling and new… Usually management says “hey, please build this new feature, and update that library later”. This is how technical debt accrues. It’s not operational staff or software developers saying ”forget that, I don’t care about this“, it’s almost always conflicting priorities. 

I hope this helps clarify the difference.

Discoveries as a Result of the Log4j Debacle

Posted on by 2 Comments
Me, pre-log4j
Tanya making a silly face.
Happier times, before I knew anything about log4j.

Over the past 2 weeks many people working in IT have been dealing with the fallout of the vulnerabilities and exploits being carried out against servers and applications using the popular Log4J java library. Information security people have been responding 24/7 to the incident, operations folks have been patching servers at record speeds, and software developers have upgrading, removing libraries and crossing their fingers. WAFs are being deployed, CDN (Content Delivery Network) rules updated, and we are definitely not out of the woods yet.

​Those of you who know me realize I’m going to skip right over anything to do with servers and head right onto the software angle. Forgive me; I know servers are equally important. But they are not my speciality…

Although I already posted in my newsletter, on this blog and my youtube channel , I have more to say. I want to talk about some of the things that I and other incident responders ‘discovered’ as part of investigations for log4j. Things I’ve seen for years, that need to change.

After speaking privately to a few CISOs, AppSec pros and incident responders, there is a LOT going on with this vulnerability, but it’s being compounded by systemic problems in our industry. If you want to share a story with me about this topic, please reach out to me.

Shout-outs to every person working to protect the internet, your customers, your organizations and individuals against this vulnerability.

You are amazing. Thank you for your service.

Let’s get into some systemic problems.

Inventory: Not just for Netflix Anymore

I realize that I am constantly telling people that having a complete inventory of all of your IT assets (including Web apps and APIs) is the #1 most important AppSec activity you can do, but people still don’t seem to be listening… Or maybe it’s on their “to do” list? Marked as “for later”? I find it defeating at times that having current and accurate inventory is still a challenge for even major players, such as Netflix and other large companies/teams who I admire. If they find it hard, how can smaller companies with fewer resources get it done? When responding to this incident this problem has never been more obvious.

Look at past me! No idea what was about to hit her, happily celebrating her new glasses.

​Imagine past me, searching repos, not finding log4j and then foolishly thinking she could go home. WRONG! It turns out that even though one of my clients had done a large inventory activity earlier in the year, we had missed a few things (none containing log4j, luckily). When I spoke to other folks I heard of people finding custom code in all SORTS of fun places it was not supposed to be. Such as:

  • Public Repos that should have been private
  • Every type of cloud-based version control or code repo you can think of; GitLab, GitHub, BitBucket, Azure DevOps, etc. And of course, most of them were not approved/on the official list…
  • On-prem, saved to a file server – some with backups and some without
  • In the same repos everyone else is using, but locked down so that only one dev or one team could see it (meaning no AppSec tool coverage)
  • SVN, ClearCase, SourceSafe, subversion and other repos I thought no one was using anymore… That are incompatible with the AppSec tools I (and many others) had at hand.

Having it take over a week just to get access to all the various places the code is kept, meant those incident responders couldn’t give accurate answers to management and customers alike. It also meant that some of them were vulnerable, but they had no way of knowing.

Many have brought up the concept of SBOM (software bill of materials, the list of all dependencies a piece of software has) at this time. Yes, having a complete SBOM for every app would be wonderful, but I would have settled for a complete list of apps and where their code was stored. Then I can figure out the SBOM stuff myself… But I digress.

Inventory is valuable for more than just incident response. You can’t be sure your tools have complete coverage if you don’t know you’re assets. Imagine if you painted *almost* all of a fence. That one part you missed would become damaged and age faster than the rest of fence, because it’s missing the protection of the paint. Imagine year after year, you refresh the paint, except that one spot you don’t know about. Perhaps it gets water damage or starts to rot? It’s the same with applications; they don’t always age well.

We need a real solution for inventory of web assets. Manually tracking this stuff in MS Excel is not working folks. This is a systemic problem in our industry.

Lack of Support and Governance for Open-Source Libraries

This may or may not be the biggest issue, but it is certainly the most-talked about throughout this situation. The question posed is most-often is “Why are so many huge businesses and large products depending on a library supported by only three volunteer programmers?” and I would argue the answer is “because it works and it’s free”. This is how open-source stuff works. Why not use free stuff? I did it all the time when I was a dev and I’m not going to trash other devs for doing it now…. I will let others harp on this issue, hoping they will find a good solution, and I will continue on to other topics for the rest of this article.

Lack of Tooling Coverage

The second problem incident responders walked into was their tools not being able to scan all the things. Let’s say you’re amazing and you have a complete and current inventory (I’m not jealous, YOU’RE JEALOUS), that doesn’t mean your tools can see everything. Maybe there’s a firewall in the way? Maybe the service account for your tool isn’t granted access or has access but the incorrect set of rights? There are dozens are reasons your tool might not have complete coverage. I heard from too many teams that they “couldn’t see” various parts of the network, or their scanning tools weren’t authorized for various repos, etc. It hurts just to think about; it’s so frustrating.

Luckily for me I’m in AppSec and I used to be a dev, meaning finding workarounds is second nature for me. I grabbed code from all over the place, zipping it up and downloading it, throwing it into Azure DevOps and scanning it with my tools. I also unzipped code locally and searched simply for “log4j”. I know it’s a snapshot in time, I know it’s not perfect or a good long-term plan. But for this situation, it was good enough for me. ** This doesn’t work with servers or non-custom software though, sorry folks. **

But this points to another industry issue: why were our tools not set up to see everything already? How can we tell if our tool has complete coverage? We (theoretically) should be able to reach all assets with every security tool, but this is not the case at most enterprises, I assure you.

Undeployed Code

This might sound odd, but the more places I looked, the more I found code that was undeployed, “not in use” (whyyyyyyy is it in prod then?), the project was paused, “Oh, that’s been archived” (except it’s not marked that way), etc. I asked around and it turns out this is common, it’s not just that one client… It’s basically everyone. Code all over the place, with no labels or other useful data about where else it may live.

Then I went onto Twitter, and it turns out there isn’t a common mechanism to keep track of this. WHAT!??!?! Our industry doesn’t have a standardized place to keep track of what code is where, if it’s paused, just an example, is it deployed, etc. I feel that this is another industry-level problem we need to solve; not a product we need to buy, but part of the system development life cycle that ensures this information is tracked. Perhaps a new phase or something?

Lack of Incident Response/Investigation Training

Many people I spoke to who are part of the investigations did not have training in incident response or investigation. This includes operations folks and software developers, having no idea what we need or want from them during such a crucial moment. When I first started responding to incidents, I was also untrained. I’ve honestly not had near as much training as I would like, with most of what I have learned being from on the job experience and job shadowing. That said, I created a FREE mini course on incident response that you can sign up for here. It can at least teach you what security wants and needs from you.

The most important part of an incident is appointing someone to be in charge (the incident manager). I saw too many places where no one person was IN CHARGE of what was happening. Multiple people giving quotes to the media, to customers, or other teams. Different status reports that don’t make sense going to management. If you take one thing away from this article it should be that you really need to speak with one voice when the crap hits the fan….

No Shields

For those attempting to protect very old applications (for instance, any apps using log4j 1.X versions), you should consider getting a shield for your application. And by “shield” I mean put it behind a CDN (Content Delivery network) like CloudFlare, behind a WAF (Web Application Firewall) or a RASP (Run-Time Application Security Protection).

Is putting a shield in front of your application as good as writing secure code? No. But it’s way better than nothing, and that’s what I saw a lot of while responding and talking to colleagues about log4j. NOTHING to protect very old applications… Which leads to the next issue I will mention.

Ancient Dependencies

Several teams I advised had what I would call “Ancient Dependencies”; dependencies so old that the application would requiring re-architecting in order to upgrade them. I don’t have a solution for this, but it is part of why Log4J is going to take a very, very long time to square away.

Technical debt is security debt.

– Me

Solutions Needed

I usually try not to share problems without solutions, but these issues are bigger than me or the handful of clients I serve. These problems are systemic. I invite you to comment with solutions or ideas about how we could try to solve these problems.

I want to talk about Log4j

Posted on by 1 Comment
What my face looked like when I figured out how scary this log4j thing is.

Lots of people are talking about how Log4J affects servers, but if you subscribe to my newsletter or read my blog, you probably want to know about your apps. Let’s talk about what the problem is, how to figure out if you have it, then what to do about it.

Problem: this java logging dependency has a vulnerability in it that allows an attacker to take over your web server and run commands from it. They can run this attack before a login screen (unauthenticated). This is the “most scary possible” from a security viewpoint.

What my face looked like when I figured out how scary this log4j thing is.
My face when I understood how scary log4j is.

Do you have this problem? You can search for it in a bunch of ways, but I suggest just going to your code repo, searching for “*log4J*”. If you find nothing, ALSO search using any sort of dependency tool. This could be dependencyGraph in GitHub, Snyk, OWASP Dependency-Check, White source, etc. These are also often called “Software Composition Analysis” or SCA for short.

Versions 2.x (every single one except 2.17) are vulnerable. Note: 2.15 and 2.16 are also vulnerable.

Versions 1.x are only vulnerable if you call the JMSAppender functionality. You can look in your code for “JMSAppender” to see if you are calling it. If you are, you are vulnerable. If not, you’re good.

If you’re going to rule it out, make absolutely sure. If you don’t have it, go back to your week and chill. If you do, let’s get into that.

NOTE: Email your security team (appsec team) and let them know you don’t have it. They will be SO HAPPY.

Now onto if you have it.

Okay, so you have Log4J and you think your week is ruined, but maybe it’s not. For each instance you find make a list, to document, and to give to security later. Verify if the code has actually been deployed somewhere or not. As I did some IR this weekend, most instances I found were undeployed.

If the code has not been deployed anywhere, mark it as “do not deploy” and move on. Anything that HAS been deployed: where? WHERE is it deployed/where does it live? Behind a WAF or CDN? If so, add the rules to block this attack. CloudFlare & CloudFront both do, turn it on!

If you have your own RASP or WAF, and there are no rules available yet from your vendor (ask them if you don’t see it, tell them you want one if not). So if not available from the vendor, make your own “virtual patch”. Work with InfoSec to write regex that blocks the attack or fish some off of the internet (you are not the only one with this problem).

To be clear, if you make a virtual patch, this is a temporary measure, and you need to 1) monitor it to make sure it’s working, plus 2) upgrade the versions of Log4J as soon as you can. Don’t forget please. 😀

Worst case scenario: You have log4J, nothing to help you block it, and it’s a vulnerable version.

It’s go time.

Option 1: “accept the risk” and do nothing to block it. You will still monitor the situation, but that is all. You will instead spend your efforts on releasing the upgraded version of your software as soon as humanly possible. For some organizations, this is the only option. Don’t feel bad, use that energy of the update instead. Ensure you test thoroughly; you don’t want to release patches like our industry saw during Meltdown/Spector that broke the patched systems worse than the vulnerability would have.

Option 2: Shut off the vulnerable systems. Immediately. If your business can have a few systems down until you figure out how to do something better, this *might* present less risk. There are currently many systems all over the internet being turned off, in the short term. There is no shame in doing this if it’s your best option. I’d rather have egg on my face than an exploited server.

Option 3: Go through your code and remove this dependency from your project, then comment out the code that calls it. When you are ready to apply the upgrade/patch, you will add it and turn it back on. Stop logging, just for now. This is the only situation where I would ever recommend removing logging. Test it thoroughly before deploying and make sure you don’t have any sort of “backup logging” that could interfere or spoil your efforts.

No matter what you decide: Tell the InfoSec team. Tell your management. Do not make these decisions solo.

I made a video about it as well. This topic is important to me, and it should be to you too.

​

Log4J Affecting Civilians

This vulnerability will affect many systems, including the ones you use at home. Here are some tips to protect your personal devices and home network (tell your friends).

  • Apply updates. Especially this week and next week. If you’re computer, phone or any piece of software wants an update, say yes. Updates in general are great, they often contain security fixes, as well as new features.
  • At some point this week make time to call your internet provider and ask if your router or modem has any updates for log4j. A lot of *devices* are going to be vulnerable, and most of us forget about our modems.
  • If you have “smart” devices at home, check for updates on them too!

More tips!

While we’re at it, here are a few tips for securing your digital self (again, tell your friends!):

  • Turn on Multi-Factor Authentication (MFA) or two-factor authentication (2FA), on all your online accounts that are important to you. Your banking accounts, government services, shopping accounts that have a credit card saved, etc.
  • Get a password manager. Then change and save your passwords into it, one by one, as you visit all your favourite sites. Let it auto-generate unique passwords for you.
  • Think twice before you click on a link in an email if you were not expecting to receive that email. Verify who it’s from, that it’s a legitimate website, and that the link starts with a domain that you recognize. If you’re not sure, copy the link into google.com (not the address bar, go to the search engine website) then add “phishing” to your search and see what it says.
  • Never give personal information, such as pictures of your ID, your social insurance number, your home address, date of birth, etc. to anyone on the internet.

My Career Story

Posted on by Leave a comment
Me, smiling

I started coding at 17 years old, and it was love at first sight.

I got great marks in all of my classes in high school, but loved computer science because in every class, I could “make something out of nothing.” Computer science runs deep in my family as almost all of my aunts and uncles are computer scientists, and my cousins are engineers, scientists and programmers. When I announced that I wanted to go to college for computer science my family responded with “what else would you take?” It wasn’t until years after working in tech that I realized that this is not an experience that most young women share.

I landed my first job in tech at age 18, and haven’t stopped since, despite several career setbacks, harassment and toxic work environments. I realize this might not seem very encouraging, but I have to tell you; things in tech have really improved. I’ve had the fortune of work experience in a variety of different situations both in computer science and in my other passion, music. Both careers taught me the value of collaborating with others, confronting differences, and taking constructive criticism well. It’s also given me the benefit of becoming more resilient when it comes to unpleasant situations or less-than-constructive comments made in the workplace.

For many years, I was a programmer by day and a musician at night. My successful music career allowed me to play in countless venues and bars around town, and it taught me many lessons that have since turned out to be very helpful in tech, such as how to handle hecklers, how to capture the attention of a drunk and belligerent crowd, and what the best way to throw someone off a stage is. As you can imagine, there were challenges to being a young 20-something woman in a hardcore punk band.

Later in my career I met an ethical hacker who was also in a band and we became friends. He spent the next 1.5 years convincing me to join him as his apprentice and learn how to hack. I became fascinated with the security of software, I wanted to know everything. I joined my local OWASP chapter and almost immediately became a chapter leader, which helped me greatly since I had the chance to invite experts on topics that I was interested in to come speak for us. I also met my next 3 professional mentors though OWASP, who taught me even more. OWASP is an incredibly supportive and amazing community, I strongly recommend that everyone joins their local chapter.

OWASP Montreal, I drove there with my mom to speak at lunch time. I missed a day of work for it.

At this point in my career I felt like I had a thirst for knowledge that could not be quenched. Although I managed to switch over from software development to a full time security job, I was frustrated that there was no budget for me to go on the types of advanced training that I was interested in. Then one of my professional mentors convinced me to speak at a conference, and they let me in FOR FREE.

For the next 2 years, I spoke at meetups and local events, taught myself as much as I could, and worked in application security helping developers make more secure apps. I loved it, but I kept striving for more. I wanted to do more modern types of application security, and I realized that the organizations I worked for were not very modern, and resistant to change. I found that my drive and ambition was difficult for certain managers, and it became a point of friction for me in the workplace.

Then I broke through from meetups into speaking at conferences. I honestly couldn’t believe it when I received the email saying that I had been accepted to speak at AppSec EU, the international OWASP conference. I discovered that all of my musical stage performance skills transferred over and with all of my practice at meetups that I had become good at public speaking. After AppSec EU, I had invitations to speak all over the world. As conferences started sending me plane tickets, I took time off work and went off to learn for free. I realized that a career shift was necessary. I knew that I had something to offer to the right employer, but I wasn’t quite sure what that would be… Then Microsoft reached out to me.

A Microsoft representative said that he had heard about me, and wanted to interview me for a “Developer Advocate” position. I had no idea at that point that “developer relations” was a job, and when he described what the job would be I said “I already do that, for free.” It took him about 20 minutes to convince me that he was not kidding, this was a real job, and he was actually from Microsoft. Before I knew it, I was traveling the planet, learning about cloud security, working with absolutely brilliant people and so much more. All the while I was *getting paid* to do it! Talk about a dream!

During my many years traveling and talking to the community, I learned a lot about my industry, both good and bad. I learned that software developers had a lot of aches and pains in regards to security that I had also felt when I was a developer, and especially during my work in incident response and AppSec. My goal in being a developer and cloud advocate was to help push the industry forward, and to help people create more secure software, everywhere. During this time I founded the #CyberMentoringMonday online initiative and the WoSEC (Women of Security) organization, released countless articles, videos and podcasts, and spoke regularly at security events. Although I definitely felt I was helping many people in my industry, I felt like I could do even more. I also felt the constant travel was extremely exciting, but also exhausting and perhaps not the most efficient way to help the most people. I wanted to figure out how to make a bigger difference, and ’scale’ myself in a more effective manner.

With that in mind, I started to devise a plan; focus my efforts in a more concise way in order to deliver more impact. Do fewer things, but do those things in a very big way. I decided to choose two big goals; to write a book and start my own company. And I decided I would just go for it, even if it was scary.

I realized at this point that I was going to have to leave Microsoft to pursue my new career goals. I decided to start my own online training academy, We Hack Purple. We have a podcast, community and courses, it’s a dream come true!

I am also in the process of writing my first book! It’s an intro to AppSec, “Alice and Bob Learn Application Security”, and I’m excited to share it with the community at large when it’s ready. Even though I am at the very beginning of both of these adventures, you better believe I plan to knock them out of the park! ** Alice and Bob Learn AppSec is now available worldwide!

If I can offer advice to you it is this: if you want it, go get it. Don’t let anyone tell you that you can’t reach greatness; you can, you just need to be prepared to work like you’ve never worked before. The Information Security industry needs all the help it can get, and we definitely need you. Yes you, the person reading this right now. Please join us, and help us make the world a better and more secure place.

I have a mailing list, please subscribe, it’s free!

#CyberMentoringMonday

Posted on by Leave a comment
WoSEC Ottawa

Some people have been asking me online how to be a good mentor. Here are some thoughts for all of you. 😀

Some mentees don’t listen, and are not willing to put in the work. Some of them will astound you and excel beyond your wildest dreams. The key is finding a good match for you, and for them.

It’s your job as a mentor to try to help your mentee any way you can. That can be through advice, loaning them a book, sharing resources, introducing them to people that can help them, referring them for a job (if appropriate) or other opportunities.

WoSEC Ottawa— Women of Security

Example: I wrote an essay to explain to a conference why one of my mentees deserved a diversity grant. She has worked SO HARD to teach herself and change careers. She won the grant because of her hard work, AND my essay. It took me 30 minutes, and she benefited.

Example 2: I brainstormed talk ideas with a mentee, then she built an amazing proof of concept. I asked a conference that I was keynoting to book her, even though she’d never spoken before. She was AMAZING! Out of this world! I knew she would be good, but she was 10 times better than I would have dared to hope for.

Example 3: When I’m invited to speak somewhere but cannot make it, I ask if they would like me to recommend someone else. I have a list of people who are not well-known, but who are amazing. I always recommend one of them to take my place. I advocate for them.

Example 4: I asked a friend to let one of my mentees into his very expensive training for free, and he said yes. I let her stay in my hotel room with me so she could afford the trip. It cost me one favour and sharing my room to give her a huge leg up for her career.

I use the power and privileges of my current role to help others, and you can too. You may not even realize how much power you have until you start helping someone.

Sometimes it’s recommending or loaning someone the right book. Sometimes it’s about letting them have a place in your training, workshop, talk, or conference for free. Sometimes it’s helping them when they are stuck at work on a technical problem and you give them the answer. Maybe you will introduce them to the person who will hire them some day. It’s about helping however you can.

The key with mentoring is that they can trust you, and that you have their best interests at heart. It’s not about being perfect or knowing everything. It’s about your motivations.

Good luck folks!

#CyberMentoringMonday