You may not be aware but Canada’s Public Safety department put out a call to Canadian Citizens (sorry brilliant people who are not Canadian), asking for ideas, suggestions and thoughts on what they should prioritize next for the Canadian Government for InfoSec. I WAS SO EXCITED WHEN I SAW THIS AND WROTE THEM IMMEDIATELY. Obviously I made suggestions about AppSec. You have until August 19, 2022 to send your suggestions. The suggestions that I sent are below.
I would like to see the Canadian Public Service and Government of Canada focus on ensuring we are creating secure software for the public to use. I want to see formal application security programs (sometimes called a secure system development life cycle or S-SDLC) at every department. I have extensive training materials on this topic that I would be happy to provide for free to help.
I would also like to see a government-wide training for all software developers on secure coding, and AppSec training for every person tasked with ensuring the software of their department is secure. When I was in the government (13.5 years), I was never allowed to have security training, because it was too expensive ($7,000 USD for a SANS class was completely out of reach). I was told the government wouldn’t arrange giant classes (say 100 people, splitting the cost of one instructor), because that would be ‘unfair competition with private industry’. You need to fix that, having mostly untrained assets is not a winning strategy. There needs to be a government-wide training initiative to modernize your workforce. (Again, I have free online training that can be accessed here: https://community.wehackpurple.com – join the community (free), then take any courses you want (also free))
Create security policies that apply to all departments, then socialize them (do workshops, create videos, make sure everyone knows – don’t just post them to the TBS website and hope someone notices on their own). A secure coding guideline. An AppSec program/secure SDLC. Incident response, etc. Each department shouldn’t have to start from scratch each time. Then we could have a standardization of what level of security assurance that we expect from each department. I provide some of these policies in the AppSec foundations level 2 course, which is free in the link above.
Throw away all the old policies and procedures that are just not working. 90-day password rotation? Gone. SA&A process that takes several weeks to complete but doesn’t actually offer much in the way of actionable advice? Gone. Re-evaluate current process, get rid of the bad ones. We need agile processes, that let people get their work done. I felt like many of the processes that I had to do in the government were in place because of a lack of trust in the staff’s competency. Instead of not trusting the staff, train them, then trust them. If they continue to screw up after training, discipline them and eventually get rid of the bad apples. Most of your staff is GOOD. Some of them are truly amazing. Treat them with trust and many of them will astound you. Remove onerous administration that is there because you don’t trust them, then let them get their jobs done.
If you have any questions I would love to talk. Thank you for putting out an open call, I’m super-impressed!
The purpose of any post mortem is to look into the past in order to find ways to prevent similar issues from happening again, and also to improve upon our responses to issues found in the future. It is not to blame others, point fingers, or punish. A proper post mortem states facts, including what went well and what did not, and issues ideas for improvements going forward.
Short rehash: log4j is a popular java library used for application logging. On November 26, 2021, a vulnerability was discovered in it that allowed any attacker to paste a short string of characters into the address bar, and if vulnerable, the attacker would gain remote code execution (RCE) access to the web server. No authentication to the system was required, making this the simplest attack of all time to gain RCE (the highest possible level of privilege) on a victim’s system.
Points of interest and timeline:
This vulnerability was recorded in the CVE database on Friday, November 26, 2021, but was not weaponized until December 9, 2021.
This vulnerability is often referred to as #Log4Shell.
Log4j can only be found in software using java and/or jar files,
Non-java languages and frameworks were not affected: log4Net, log4Js, etc.
Both custom software (made in house) and COTS (configurable off the shelf) software was affected, including popular platforms that are household names.
Many pieces of software call other pieces of software, including log4j, and thus were vulnerable, despite not seeming to be a problem at first glance .
Several servers with middleware (due to the library running inside them) were vulnerable to log4j.
The CVE is named CVE-2021-44228
Log4j versions Log4j 2.0 – 2.14.x were vulnerable
A patch, Log4j 2.15 was released on December 16th, and almost immediately a vulnerability (CVE 2021-45046) was found in it that allowed denial of service attacks.
Another patch, Log4j 2.16, which was also deemed vulnerable
December 17th, a 3rd vulnerability in log4j was released, CVE-2021-45105
On December 28th Checkmarx released another vulnerability (CVE-2021-44832) within log4j, but it required that the user already have control over the configuration, meaning the system had already been breached, meaning it was not nearly as serious as previously reported vulnerabilities.
Version 2.17.2 is widely considered the safest version of this library.
Log4J 1.x was no longer supported as of 2015, and while all 1.x versions have several vulnerabilities, all were immune to this one exploit.
As soon as the alert came out, our industry acted. Incident responders immediately started contacting vendors, monitoring networks, researching, and contacting peers for more ideas. Application security teams worked with software developers to find out if any of their custom code had the affected libraries. CISOs started issuing statements to the media. And the entire industry waited for a patch to be released.
What was the root cause of this situation?
A very large percentage of all the applications in the world contain at least some open-source components, which generally have little-to-no budget for security activities, including security testing and code review. On top of this, even for-profit organizations that create software often have anywhere from acceptable to abysmal security assurance processes for the products and components they release. The part of our industry that is responsible for the security of software, often known as application security, is failing.
Little-to-no financial support for open-source software means there is usually no budget for security.
Due to not enough qualified people in the field of application security, it is extraordinarily expensive to engage a skilled expert to do this work.
No regulation or laws controlling or addressing security in IT systems in most countries means this industry runs without governmental influence or regulation.
Although there are some groups (such as NIST and OWASP), trying to create helpful frameworks for software creators to work within, there is no mandate for any person or organization to do so.
The security of software is not taught in most colleges, universities, or boot camps, meaning we are graduating new software engineers who do not know how to create secure applications, test applications for security, or recognize and correct many of the security issues they may encounter.
Education for software security is extremely expensive in the Westernized world, pricing it out of reach for most software developers and even organizations.
Due to companies not sharing information, it is impossible to state specifics in this category. That said, after speaking to a few sources who wish to remain anonymous, the following is likely true:
Damages are estimated in the hundreds of millions, for the industry world-wide.
Hundreds of thousands of hours of logged overtime, most likely resulting in or contributing to incident responder employee burnout.
Many organizations only applied this one patch and went back to business as usual. That said, some used this situation as an opportunity to create projects to simplify the patching process and/or the software release process, to ensure faster reaction times in the future.
Many companies that previously did not think supply chain security was important have updated their views, and hopefully also their toolset and processes.
When unable to create an accurate cost estimate, ‘guesstimates’ are often accepted.
Time to Detection?
Most companies (according to anonymous sources and online discussion) spent 2-3 straight weeks working on this issue, dropping all other priorities for the InfoSec teams and most other priorities for those applying patches and scanning.
Detection in 3rd party applications and SaaS was extremely difficult, as many organizations issued statements that they were unaffected, only to find out later they had been incorrect/uninformed.
Generally, most incident response teams responded the day-of the announcement.
AppSec teams checked their SCA tools and code repositories for the offending library and asked for patches/upgrades where necessary.
CDNs, WAFs and RASPs were updated with blocking rules.
Those managing servers searched dependencies and patched, feverishly.
Those managing operating systems, middleware and SaaS wrote vendors to ask for status reports.
Incident responders managed all activities, often leading the search efforts.
Lessons Learned? Opportunities for Improvement?
What follows are the author’s ideals for lessons learned. Each organization is different, but below is a list of potential lessons learned by any organization.
Patching processes for operating systems, middleware, configurable off the shelf software (COTS), and custom software must be improved. This is the main threat to organizations from this type of vulnerability, slow updates/upgrades/patches that leave organizations open to compromise for extended periods of time.
Incomplete inventory of software assets is a large threat to any business, we cannot protect what we do not know we have. This includes a software bill of materials (SBOM). Software asset inventory must be prioritized.
Organizations that learned later, rather than earlier, about this vulnerability were at a distinct disadvantage. Subscribing to various threat intelligence and bug alert feeds is mandatory for any large enterprise.
Many Incident Response teams and processes did not have caveats for software vulnerabilities. Updating incident response processes and team training to include this type of threat is mandatory for medium to large organizations.
Most service level agreements (SLAs) did not cover such a situation and updating these with current vendors would be a ‘nice to have’ for future, similar, situations. Adding this to vendor questions in the future would be an excellent negotiation point.
Many custom software departments were unprepared to find which applications did and did not contain this library. Besides creating SBOMS and inventory, deploying a software composition analysis tool to monitor custom applications and their dependencies would have simplified this situation for any dev department.
Many organizations with extensive technical debt found themselves in a situation where it would require a re-architecting of their application(s) in order to upgrade off of the offending library. Addressing deep technical debt is paramount in building the ability to respond to dependency-related vulnerability of this magnitude.
There are hundreds of thousands of open-source libraries all over the internet, littered with both known and unknown vulnerabilities. This problem is not new, but this specific situation has brought this issue into the public eye in a way that previous vulnerabilities have not. Our industry and/or governments must do something to ensure the safety and security of these open-source components that software creators around the world use every day. The current system is not safe nor reliable.
What went well?
Incident response teams worked quickly and diligently to triage, respond to, and eradicate this issue.
Operational and software teams responsible for patching and upgrading systems performed heroically, in many organizations.
Multiple vendors went above and beyond in assisting their customers and the public responded quickly and completely to this issue.
What could have gone better?
Messaging was confused at times, as few knew the extent of this issue at first.
The media released many articles that emphasized fear, uncertainty, and doubt (FUD), rather than helpful facts, creating panic when it was not necessary.
Companies that produce customer software, but who did not have application security resources, were left at a distinct disadvantage, unaware of what to do for the first few days (before articles with explicit instructions were available).
Many vendors issued statements that were just not true. “Our product could have stopped this” and “you would have known before everyone else if you had just bought us”, etc. Although there are some products that may have been able to block such an attack without additional configurations, they were few and far between compared to the number of vendors claiming this to be true of their own product(s).
Improve infrastructure, middleware, and COTS patching and testing processes.
Improve custom software release processes.
Request Software Bill of Materials (SBOMs) for all software, including purchased products and those which are home-grown.
Create a software asset inventory and create processes to ensure it continues to be up to date. This should include SBOM information.
Subscribe to threat feeds and bug alerts for all products you own, as well as programming languages and frameworks used internally.
Train your incident response team and/or AppSec team to respond to software-related security incidents.
For companies that build custom software: Install a software composition analysis tool and connect it to 100% of your code repos. Take the feedback from this tool seriously and update your dependencies accordingly.
Negotiate SBOMs and Service Level Agreements (SLAs) on patching for all new COTS and middleware products your organization purchases. Attempt to negotiate these after the fact for contracts you already have in place.
Do your best to keep your dependencies reasonably up to date and address technical debt in a reasonable way. If your entire application needs to be re-architected just to update a single dependency to current, this means your technical debt is currently unacceptable. Make time for maintenance now, rather than waiting for it to “make time for you”, later.
Create a process for evaluating and approving of all dependencies used in custom software, not just open-source ones. A software composition analysis tool can help with both an implementation and documentation point of view.
Could we have known about this sooner?
The very frustrating question the incident responders have been asked over and over again since this happened, is could we have known about this sooner? And the answer, unfortunately, is probably not. Not with the way we, as an industry, treat open-source software.
Could we have responded better?
This is a question only your organization can answer for itself. That said, reviewing the ‘Lessons Learned’ section and implementing one or more of the ‘Action Items’ in this article could certainly help.
How can we stop this from happening again?
Our industry needs to change the way we manage open-source libraries and other 3d party components. This is not something the author can answer, as a single person. This is something the industry must push for to implement real and lasting change. One person is not enough.
It is likely that much of our industry will remain unchanged from this major security incident. That said, it is the author’s hope that some organizations and individuals changed for the better, prioritizing fast and effective patching and upgrading processes, and the repayment of technical debt, setting themselves apart from others as leaders in this field.
Hello! I will be all over the place in San Francisco from Saturday June 4th to Thursday June 9, and I’d love to meet any of you that are going to be there. My schedule is pretty hectic, but I’m sharing it in hopes some of you can join me at one or more of the events. Thank you to BRIGHT SECURITYfor sponsoring & supporting every bit of this trip.
Saturday June 4th, the Bright team will be at B-Sides all day! They have a booth and if I arrive in time I will be there in the late afternoon (4:00-5:30). Sometimes flights are late, so this one is non-certain.
Sunday June 5th I will be hanging out with the Bright teamall day long! 8 am to 5 pm. Please come join me!
Sunday June 5, 6:00 – 8:00 pm, We Hack Purple is having a meetup at Share Bubble Tea, 135 4th St, San Francisco, CA 94103, United States. We will likely be milling around outside and inside. Bubble Tea is a tasty Asian dessert, and it’s about $6, so this meetup shouldn’t cost you much at all to join. You don’t even need to buy anything if you don’t want. Everyone is welcome to come hang out! RVSP within the WHP community to let us know you’ll be there.
Monday June 6, 8:30 am, my talk at RSA! Check the conference schedule for the room!
Monday June 6, 9:40 am, Birds of a Feather event at RSA: Transforming Security Champions – Check the schedule for room #
I will also be attending several meetups this day, but you need to get your invite to attend (I cannot get you an invite, sorry!): Microsoft Party, Forte Group, IANs Faculty Party, and RSA Speakers Event
Tuesday Jun 7, 12:40 pm, RSA Panel, Spreading Application Security Ownership Across the Entire Organization – Check RSA conference schedule for room number
Tuesday June 7, 1:30 pm to 2:00 Book Signing at RSA Library, bring a copy of Alice and Bob and I shall sign it for you! South Hall, Mezzanine Level Lobby.
Thursday June 10, 12:30 pm – Last book signing with free copies of Alice and Bob Learn with Cloud Defense! Please come see us in the Moscone center at their booth!
Lastly, if you want to talk to an expert from Bright, you can book them directly, here. Ask them all of your questions about dynamic security testing, security unit testing, and more! Seriously, they would love to meet you!
Yes, you read that right! My friends at Bright bought my company, We Hack Purple! Bright makes a DAST (dynamic application security testing tool), and I have been on their advisory board for some time, so we know each other well and have been working together for years. They also just released a brand new tool for the Lucky Framework (crystal programming language), which creates security-focused unit tests, automagically! Trust me, it’s very cool, and there’s more on the way!
As part of this deal, starting immediately, all of the courses from the We Hack Purple Academy will be available in the We Hack Purple Community, for FREE. Yes, you heard that right. Secure coding for everyone!
So what comes next? I plan to work with Bright for the next couple years, creating more content, running the We Hack Purple Community, speaking at conferences and helping to improve the Bright products until they are absolutely spectacular. I will also start on writing my next book, Alice and Bob Learn Secure Coding.
I haven’t written in my personal blog in a while, and I have good reasons (I moved to a new city, the new place will be a farm, I restarted my international travel, something secret that I can’t announce yet, and also did I mention I was a bit busy?). But I still can’t get over log4j (see previous article 1, article 2, and the parody song). The sheer volume of work involved (one company estimated 100 weeks of work, completed over the course of 8 days of time) in the response was spectacular, and the damage caused is still unknown at this point. We will likely never know the true extend of the cost of this vulnerability. And this bugs me.
I met up last month with a bunch of CISOs and incident responders, to discuss the havoc that was this zero-day threat. What follows are stories, tales, facts and fictions, as well as some of my own observations. I know it’s not the perfect story telling experience you are used to here, bear with me, please.
Short rehash: log4j is a popular java library used for application logging. A vulnerability was discovered in it that allowed any user to paste a short string of characters into the address bar, and if vulnerable, the user would have remote code execution (RCE). No authentication to the system was required, making this the simplest attack of all time to gain the highest possible level of privilege on the victim’s system. In summary: very, very scary.
Most companies had no reason to believe they had been breached, yet they pulled together their entire security team and various other parts of their org to fight against this threat, together. I saw and heard about a lot of teamwork. Many people I spoke to told me they had their security budgets increased my multitudes, being able to hire several extra people and buy new tools. I was told “Never let a good disaster go to waste”, interesting….
I read several articles from various vendors claiming that they could have prevented log4j from happening in the first place, and for some of them it was true, though for many it was just marketing falsehoods. I find it disappointing that any org would publish an outright lie about the ability of their product, but unfortunately this is still common practice for some companies in our industry.
I happened to be on the front line at the time, doing a 3-month full time stint (while still running We Hack Purple). I had *just* deployed an SCA tool that confirmed for me that we were okay. Then I found another repo. And another. And another. In the end they were still safe, but finding out there had been 5 repos full of code, that I was unaware of as their AppSec Lead, made me more than a little uncomfortable, even if it was only my 4th week on the job.
I spoke to more than one individual who told me they didn’t have log4j vulnerabilities because the version they were using was SO OLD they had been spared, and still others who said none of their apps did any logging at all, and thus were also spared. I don’t know about you, but I wouldn’t be bragging about that to anyone…
For the first time ever, I saw customers not only ask if vendors were vulnerable, but they asked “Which version of the patch did you apply?”, “What day did you patch?” and other very specific questions that I had never had to field before.
I heard several vendors have their customers demand “Why didn’t you warn us about this? Why can’t your xyz tool prevent this?” when in fact their tool has nothing to do with libraries, and therefore it’s not at all in the scope of the tool. This tells me that customers were quite frightened. I mean, I certainly was….
Several organizations had their incident response process TESTED for the first time. Many of us realized there were improvements to make, especially when it comes to giving updates on the status of the event. Many people learned to improve their patching process. Or at least I hope they did.
Those that had WAF, RASP, or CNDs were able to throw up some fancy REGEX and block most requests. Not a perfect or elegant solution, but it saved quite a few company’s bacon and reduced the risk greatly.
I’ve harped on many clients and students before that if you can’t do quick updates to your apps, that it is a vulnerability in itself. Log4j proved this, as never before. I’m not generally an “I told you so” type of person. But I do want to tell every org “Please prioritize your ability to patch and upgrade frameworks quickly, this is ALWAYS important and valuable as a security activity. It is a worthy investment of your time.”
Again, I apologize for this blog post being a bit disjointed. I wasn’t sure how to string so many different thoughts and facts into the same article. I hope this was helpful.
Three years ago I decided that I would share most of my talk content with my community (everything that I am not currently applying to conferences with). At the time, I only shared one, because…. I ran out of time. Now it’s time to share the second talk, “Security is Everybody’s Job!” By “share” I mean give my express permission for anyone, anywhere, to present content that I have written, with no need to pay anything or ask for my consent. You can even charge money to give the talk! Please, just teach people about security.
I’ve had a few people ask me why I would do this, and there are a few reasons. * To spread the word about how to secure software; it’s important to me to try to make the internet and other technologies safe to use. * To help new speakers (especially from underrepresented groups). If they have something they can present, with instructions they can follow, hopefully it will help make them more confident and skilled at presenting. * To share knowledge with my community in general: sharing is caring, yo. * The more people who present my talk the more people who may decide to follow me. SO MUCH WIN!
You can give this talk at any IT meetup, especially DevOps, InfoSec or any software development meetup.
Please go forth and teach AppSec! And if you have feedback I want to hear it!
Almost all of the people who respond to my #CyberMentoringMonday tweets each week say that they want to “get into InfoSec” or “become a Penetration Tester”; they rarely choose any other jobs or are more specific than that. I believe the reason for this is that they are not aware of all the different areas within the field of Information Security (InfoSec for short, and “Cyber” for those outside of our industry). I can sympathize; I was in the same position when I joined. I knew three Penetration Testers and lots of Risk Analysts and I had no clue that there were several other areas that may interest me or even existed. I knew I didn’t want to be a Risk Analyst, so I thought the only other option was PenTester. Now I know that is not true at all. This blog post will detail several other areas within the field of Information Security in hopes that newcomers to our field can find their niche more easily. It will not be exhaustive, but I’ll do my best
The above image shows 8 different potential areas within the field of Information Security according to the author, Henry Jiang; Governance, Risk, Career Development, User Education, Standards, Threat Intelligence, Security Architecture and Security Operations.
Since I come from the software development side of IT, and have done almost exclusively coding, my view is going to be extremely biased. With that in mind, the first area you may want to consider is Application Security (AppSec); any and all work towards ensuring that software is secure. This is the field that I work in, so it will have the most detail. There are all sorts of jobs within this field, but the most well-known is the web app pentester (sometimes called an ethical hacker); a person who does security testing on software. Such a person is often a consultant, but can also work in large companies. They test one system, intensively, perform a lot of manual testing, and then move on.
Jobs in Application Security:
Application Security Engineer — you do a mix of all the things listed under AppSec and you are generally a full-time employee. This includes making customer tools, launching a security champion program, writing guidelines, and anything else that will help ensure the security of your organization’s apps. I personally consider this the sweet spot, as I get to do changing and interesting work, and see the security posture improve over time. It is, however, usually a more senior role.
Threat Modeller, working with developers, business representatives and the security team (that’s you in this scenario) to find and document potential threats to your software, then create plans to test for and fix the issues.
Vulnerability Assessment: running lots of scans, all the time, of everything. You can scan the network too. Ideally, you will do more than this, to assess the security of the systems in your care, but it depends on where you work. This position is often an employee position and you tend to have prolonged relationships with the systems and teams you assess.
Vulnerability Management: Keeping Track of the vulnerabilities that all the tools and people find, reporting to management about it, and planning from a higher level. For instance; attempting to wipe out an entire bug class, implementing new tools because you see a deficiency, resource planning, etc. This is an employee position usually, and often a manager role or team lead.
Secure Code Reviewer: reading lots of code, using SAST (static application security testing) tools and SCA (Software Composition Analysis — are our 3rd party components secure?), finding vulnerabilities in written code and helping developers fix it.
DevSecOps Engineer: an AppSec engineer working in a DevOps environment. Same goal, different tactics. Adding security checks to pipelines, figuring out how to secure containers and anything else your DevOps engineers are up to.
Developer Education: this is usually a consultant role, but sometimes for large companies, someone can do this full time. The person teachers the developers to write secure code, the architects to design secure apps, threat modelling, and any other topic they can think of that will help ensure their mandate (secure apps). This person is likely also to training the security champions.
Governance: writing policies, guidelines, standards, etc, to ensure your apps are secure. This job is usually someone that does all the governance stuff for your org and the person is working with the AppSec team to get the details right, OR this person is likely a consultant because this is not an activity that needs to be re-done constantly.
Incident Response: this area includes jobs as an incident manager (you boss everyone around and make sure the incident goes as smoothly as possible), and investigations (Forensics/DFIR). Investigating incidents related to insecure software is a topic I personally find thrilling; detective work is exciting! But with the stress it causes, it’s not for everyone.
Security Testing: often called Penetration Testing, sometimes called Red Teaming, sometimes not officially recognized as a job because management isn’t “ready” to admit they need this yet. This person tests the software (and sometimes networks) to ensure they are secure. This includes manual testing, using lots of tools, and trying to break things without causing a huge mess.
Design Review: This is called a “Security Architect” but AppSec folks are often asked to review designs for potential security flaws. If asked, say yes! It’s super fun and always educational. Bonus; it’s a good way to build trust between security and the developers.
In AppSec you will also be asked to do a range of other things because that’s how life is. Potential asks; install this giant AppSec tool and figure out how it works, create a proof of concept for an exploit to show everyone that it is/is not a problem, create a proof of value with a new AppSec tool we are considering acquiring, get all the developers to log their apps like ‘so’ in order for the SIEM can read the results, research how to do something securely when you have no idea how to do that thing at all, etc. As I said, it’s superfun!
Security Architect (apps, cloud, network): Security architects ensure that designs are secure. This can mean reviewing a deployment, network or application design, adding recommendations, or even creating the design themselves from scratch. This tends to be a more senior role.
SOC Analyst/Threat Hunter: SOC analysts interpret output from the monitoring tools to try to tell if something bad is happening, while threat hunters go looking for trouble. This is mostly network-based, and I’m not good at networks, otherwise, I would have been all over this when I moved into security. The idea of threat hunting (using data and patterns to spot problems), is very appealing to my metric-adoring brain. Note: SOC Analyst is a junior or intermediate position and threat hunter is not a junior position, but if you want to get into InfoSec they are basically always hiring for SOC Analysts, at almost every company.
Risk Analyst: Evaluate systems to identify and measure risk to the business, then offer recommendations on how to mitigate or when to accept the risks. This tends to be coupled closely with Compliance, and Auditing, which I won’t describe here because I am shamefully under-educated in this area.
Security Policy Writer: Writing policies about security, such as how long network passwords need to be, that all public-facing web apps must be available via HTTPS, and that only TLS 1.2 and higher are acceptable on your network. Deciding, writing, socializing and enforcing these policies are all part of this role.
Malware Analyst/Reverse Engineer: Someone needs to look at malware and figure out how it works, and sometimes people need to write exploits (for legitimate reasons, such as to prove that something is indeed vulnerable, or… You need to ask them). If you enjoy puzzles and really low-level programming (such as ARM, assembler, etc), this job might be for you. But be careful; playing with malware at home is dangerous.
Chief Information Security Officer (CISO or CSO): ‘The boss” of security. This person (hopefully) has a seat at the executive table, directs all security aspects for a company, and is the person held responsible, for better or for worse. If you enjoy running programs, managing things from a high level, and making a big difference, this might be a role for you.
Blue Team/Defender/Security Engineer (enterprise security/implements security tools): The people that keep us safe! These people install tools, run the tools, monitor, patch, and freak out when people download and install things to their desktops without asking. They perform security operations, making sure all the things happen. While those in the SOC (Security operations centre), monitor everything that’s happening and respond when there are problems.
There are many, many, many jobs within the field of Information Security, please feel free to list some of the ones that I missed in the comments below. I hope this information helps more of you join our industry because we need all the help we can get!
Recently someone asked me what the difference was between Applications and Infrastructure. He asked why a Linux operating system wasn’t “software” and I said it was but it’s a perfect copy… I tend to speak about ‘custom software’. We ended up talking for a very long time about it, and I thought a blog post was in order.
Infrastructure is the operating systems and hardware that applications live on. Think Windows, Linux, containers, and so much more. Sometimes hardware is included in this category (depending on who you talk to), and sometimes it is not. Infrastructure is necessary to run an application, even serverless runs (briefly) on a container. Operating systems are also all standardized, and not unique in nature. For instance, if I’m running SQL server 2012 R2, and so are you, we both have the same options for patches, configuration, etc. Operating systems are software that speak to hardware.
Applications are software that speak to operating systems, databases, APIs and anything else you can think of. There are custom applications (what I’m almost always talking about, software developed for a specific business need or as a product to sell), COTS (configurable off the shelf, like sharepoint or confluence, administered by a person or team, installed locally on a server) and regular old software that you install or access via a web browser that you use as-is (no administration required/simpler). More newly there is SaaS, software as a service, which is basically a great big COTS product, hosted by someone else (no need for you to patch or otherwise take care of it, you pick your settings and use it).
Infrastructure usually needs to be patched, updated/upgraded, and hardened (secure configuration choices). Patches and upgrades arrive in a prepackaged format, but sometimes these updates can break the applications living on that infrastructure. Testing and sometimes downtime is required. This is why so many people say ‘patching is hard’, it is difficult to plan for testing, downtime and to ensure everything will go smoothly.
Software, on the other hand, includes many different components that will be provided prepackaged (such as a new version of a library or a framework) but when you update them sometimes other libraries or framework parts break and/or the custom code that your team wrote can break as well. Meaning you may need to re-code or rewrite things, or update a whole bunch of things at the same time. I’ve heard developers refer to this as “dependency hell”.
If you have just released something brand new, it’s super easy to keep it up to date. Tiny changes present less risk (which is why people love devops over waterfall), making it easier to maintain. But because it’s sparkling and new… Usually management says “hey, please build this new feature, and update that library later”. This is how technical debt accrues. It’s not operational staff or software developers saying ”forget that, I don’t care about this“, it’s almost always conflicting priorities.
Over the past 2 weeks many people working in IT have been dealing with the fallout of the vulnerabilities and exploits being carried out against servers and applications using the popular Log4J java library. Information security people have been responding 24/7 to the incident, operations folks have been patching servers at record speeds, and software developers have upgrading, removing libraries and crossing their fingers. WAFs are being deployed, CDN (Content Delivery Network) rules updated, and we are definitely not out of the woods yet.
Those of you who know me realize I’m going to skip right over anything to do with servers and head right onto the software angle. Forgive me; I know servers are equally important. But they are not my speciality…
Although I already posted in my newsletter, on this blog and my youtube channel , I have more to say. I want to talk about some of the things that I and other incident responders ‘discovered’ as part of investigations for log4j. Things I’ve seen for years, that need to change.
After speaking privately to a few CISOs, AppSec pros and incident responders, there is a LOT going on with this vulnerability, but it’s being compounded by systemic problems in our industry. If you want to share a story with me about this topic, please reach out to me.
Let’s get into some systemic problems.
Inventory: Not just for Netflix Anymore
I realize that I am constantly telling people that having a complete inventory of all of your IT assets (including Web apps and APIs) is the #1 most important AppSec activity you can do, but people still don’t seem to be listening… Or maybe it’s on their “to do” list? Marked as “for later”? I find it defeating at times that having current and accurate inventory is still a challenge for even major players, such as Netflix and other large companies/teams who I admire. If they find it hard, how can smaller companies with fewer resources get it done? When responding to this incident this problem has never been more obvious.
Imagine past me, searching repos, not finding log4j and then foolishly thinking she could go home. WRONG! It turns out that even though one of my clients had done a large inventory activity earlier in the year, we had missed a few things (none containing log4j, luckily). When I spoke to other folks I heard of people finding custom code in all SORTS of fun places it was not supposed to be. Such as:
Public Repos that should have been private
Every type of cloud-based version control or code repo you can think of; GitLab, GitHub, BitBucket, Azure DevOps, etc. And of course, most of them were not approved/on the official list…
On-prem, saved to a file server – some with backups and some without
In the same repos everyone else is using, but locked down so that only one dev or one team could see it (meaning no AppSec tool coverage)
SVN, ClearCase, SourceSafe, subversion and other repos I thought no one was using anymore… That are incompatible with the AppSec tools I (and many others) had at hand.
Having it take over a week just to get access to all the various places the code is kept, meant those incident responders couldn’t give accurate answers to management and customers alike. It also meant that some of them were vulnerable, but they had no way of knowing.
Many have brought up the concept of SBOM (software bill of materials, the list of all dependencies a piece of software has) at this time. Yes, having a complete SBOM for every app would be wonderful, but I would have settled for a complete list of apps and where their code was stored. Then I can figure out the SBOM stuff myself… But I digress.
Inventory is valuable for more than just incident response. You can’t be sure your tools have complete coverage if you don’t know you’re assets. Imagine if you painted *almost* all of a fence. That one part you missed would become damaged and age faster than the rest of fence, because it’s missing the protection of the paint. Imagine year after year, you refresh the paint, except that one spot you don’t know about. Perhaps it gets water damage or starts to rot? It’s the same with applications; they don’t always age well.
We need a real solution for inventory of web assets. Manually tracking this stuff in MS Excel is not working folks. This is a systemic problem in our industry.
Lack of Support and Governance for Open-Source Libraries
This may or may not be the biggest issue, but it is certainly the most-talked about throughout this situation. The question posed is most-often is “Why are so many huge businesses and large products depending on a library supported by only three volunteer programmers?” and I would argue the answer is “because it works and it’s free”. This is how open-source stuff works. Why not use free stuff? I did it all the time when I was a dev and I’m not going to trash other devs for doing it now…. I will let others harp on this issue, hoping they will find a good solution, and I will continue on to other topics for the rest of this article.
Lack of Tooling Coverage
The second problem incident responders walked into was their tools not being able to scan all the things. Let’s say you’re amazing and you have a complete and current inventory (I’m not jealous, YOU’RE JEALOUS), that doesn’t mean your tools can see everything. Maybe there’s a firewall in the way? Maybe the service account for your tool isn’t granted access or has access but the incorrect set of rights? There are dozens are reasons your tool might not have complete coverage. I heard from too many teams that they “couldn’t see” various parts of the network, or their scanning tools weren’t authorized for various repos, etc. It hurts just to think about; it’s so frustrating.
Luckily for me I’m in AppSec and I used to be a dev, meaning finding workarounds is second nature for me. I grabbed code from all over the place, zipping it up and downloading it, throwing it into Azure DevOps and scanning it with my tools. I also unzipped code locally and searched simply for “log4j”. I know it’s a snapshot in time, I know it’s not perfect or a good long-term plan. But for this situation, it was good enough for me. ** This doesn’t work with servers or non-custom software though, sorry folks. **
But this points to another industry issue: why were our tools not set up to see everything already? How can we tell if our tool has complete coverage? We (theoretically) should be able to reach all assets with every security tool, but this is not the case at most enterprises, I assure you.
This might sound odd, but the more places I looked, the more I found code that was undeployed, “not in use” (whyyyyyyy is it in prod then?), the project was paused, “Oh, that’s been archived” (except it’s not marked that way), etc. I asked around and it turns out this is common, it’s not just that one client… It’s basically everyone. Code all over the place, with no labels or other useful data about where else it may live.
Then I went onto Twitter, and it turns out there isn’t a common mechanism to keep track of this. WHAT!??!?! Our industry doesn’t have a standardized place to keep track of what code is where, if it’s paused, just an example, is it deployed, etc. I feel that this is another industry-level problem we need to solve; not a product we need to buy, but part of the system development life cycle that ensures this information is tracked. Perhaps a new phase or something?
Lack of Incident Response/Investigation Training
Many people I spoke to who are part of the investigations did not have training in incident response or investigation. This includes operations folks and software developers, having no idea what we need or want from them during such a crucial moment. When I first started responding to incidents, I was also untrained. I’ve honestly not had near as much training as I would like, with most of what I have learned being from on the job experience and job shadowing. That said, I created a FREE mini course on incident response that you can sign up for here. It can at least teach you what security wants and needs from you.
The most important part of an incident is appointing someone to be in charge (the incident manager). I saw too many places where no one person was IN CHARGE of what was happening. Multiple people giving quotes to the media, to customers, or other teams. Different status reports that don’t make sense going to management. If you take one thing away from this article it should be that you really need to speak with one voice when the crap hits the fan….
For those attempting to protect very old applications (for instance, any apps using log4j 1.X versions), you should consider getting a shield for your application. And by “shield” I mean put it behind a CDN (Content Delivery network) like CloudFlare, behind a WAF (Web Application Firewall) or a RASP (Run-Time Application Security Protection).
Is putting a shield in front of your application as good as writing secure code? No. But it’s way better than nothing, and that’s what I saw a lot of while responding and talking to colleagues about log4j. NOTHING to protect very old applications… Which leads to the next issue I will mention.
Several teams I advised had what I would call “Ancient Dependencies”; dependencies so old that the application would requiring re-architecting in order to upgrade them. I don’t have a solution for this, but it is part of why Log4J is going to take a very, very long time to square away.
Technical debt is security debt.
I usually try not to share problems without solutions, but these issues are bigger than me or the handful of clients I serve. These problems are systemic. I invite you to comment with solutions or ideas about how we could try to solve these problems.
Franziska Bühler and I installed several security headers during the OWASP DevSlop Show in Episode 2, 2.1 and 2.2. Unfortunately we found out that .Net Core apps don’t have a web.config, so the next time we published it wiped out the beautiful headers we had added. Although that is not good news, it was another chance to learn, and it gave me great excuse to finally write my Security Headers blog post that I have been promising. Here we go!
If you want in-depth details about what we did on the show and what each security header means, you should read Franziska’s blog post. She explains every step, and if you are trying to add security headers for the first time to your web.config (ASP.Net, not .Net CORE), you should definitely read it.
The new code for ASP.Net in your web.config looks like this: