A Month of Reckoning for SaaS software creators and consumers

An illustration of transitive and deeply connected software supply chains

The U.S. was caught off guard by foreign interference in the 2016 election. Given the powerful role of social media in political contests, understanding the Russian efforts was crucial in preventing or blunting similar, or more sophisticated, attacks in the 2020 congressional races. Tracking back to 2016, it was far more difficult to trace Russia’s experimentation on Facebook and Twitter social networks, who essentially weaponized the social network platform to become engines of deception and propaganda.

Fast forward to end of 2020 and switching context to software supply chain domain, this `new` meltdown began on Dec. 13 when Reuters reported that nation-state hackers potentially linked to Russia had gained access to email systems at the U.S. Commerce and Treasury departments, and that the attackers infiltrated by way of SolarWinds Orion software updates.

This is not in any vein similar to recalls affecting products such as automobiles, food and toys that tend to affect a narrower supply chain.

The outcome is evidenced by shares of SolarWinds rallying downward ~ 23%. On the contrary, a BlackRock iShares fund of cybersecurity stocks surged nearly 10% last week and rose another 3.5% this week entering Thursday. FireEye rose this week to a 5-year high, Microsoft topped a 90-day peak and Palo Alto Networks jumped to an all-time record.

What’s the alternative?” said Venkatesh Shankar, marketing professor at Texas A&M University.
But “the magnitude of this breach is not just within the software industry,” he said, noting SolarWinds’ customers span countless industries.

Kartik Kalaignanam, a University of South Carolina marketing professor, said traders are expecting organizations will bolster their defenses even if it means purchasing services from companies that were hacked.

Although one could argue each one of them has some sort of flaw in their system, there’s a feeling there’s going to be more spending happening, and the market will be pushed up overall,” Kalaignanam said.

~ SOURCE : Paresh Dave, Reuters

As a vendor, this is clearly not the time to exploit the misfortune of SolarWinds as sooner than later it could be you dealing with these circumstances. Try not to ambulance chase or victim shame by portraying that your solution is a miracle cure to all such problems.

This can happen to any one of our software services that we’ve authored or supply chains that we’ve subscribed into. Besides combing through our logs and incident response data we would need to elevate this discussion and understand how software services are assessed for security during a procurement phase. Unfortunately, CIOs still rely on security questionnaires (little more than excel spreadsheets) to assess the security posture of their vendors prior to signing MSAs.. The consumer of a SaaS solution or services provided by an on-premise agent have little or no understanding of the transitive supply chain of the services that they procure.

In this retrospective exercise, lets attempt to understand the following

  1. Ricochet effect from triage room to board room
  2. Buying a matryoshka doll : The emerging and entangled software supply chain across producers and consumers.
  3. Connecting backdoors : The mutation of an “new” attack pattern — one backdoor opens several others (creating a nexus effect across threat actors)

Ricochet effect from triage room to board room

Every executive (VPs, Directors, CxOs) of a SaaS based companies (both producing and consuming side) are introspecting their own security posture in light of the SolarWinds incident. The impact is far reaching as

  1. SolarWinds shares dropped ~ 33% (as of 11/27/20) over the past few weeks
  2. Their existing customers are in disarray (should we switch off the monitoring agents or not? — as advised by CISA)
  3. Their investors and board members — Silver Lake and Thoma Bravo sold approximately $158MM and $128MM respectively (source : Washington post)
  4. They would have to deal with emerging investigations and multiple independent probes triggered by bipartisan senators
  5. Churn of existing customers and partners directly impacting top line
  6. An everlasting and irreparable damage to brand and reputation

Buying a matryoshka doll : The emerging and entangled software supply chain

“When you buy software, you’re buying a matryoshka doll of various vendors’ products nested inside and connected to the product [that] you think you’re buying,” says Joel Fulton, who was the former CISO of Splunk. “Your relationship is between you and your supplier’s unseen tertiary pyramid.” Combing through all of those pyramids is practically impossible, so CIOs will likely have to rely on random checks.

This unseen tertiary pyramid (as Joel states) is continuously evolving on a daily basis. Engineering teams procure new services, install new software agents and add new open source libraries/frameworks to their software stacks. Merely exporting and assessing their asset inventory at a point and time cannot assist in effectively quantifying exposure and risk.

Today’s software stack is a web of overlapping dependencies (depending on OSS supply chain and consuming SaaS/API services). Why? Because incentives are aligned with speed and release velocity over everything else.

Borrowing from Steve Yegge’s excellent post where he drew a clear distinction between “products” and “platforms” on the basis of Matryoshka principle.

At its simplest, a product is an application that is as good as it will ever be. A platform is an application that allows other things to be built with it that even its creators may be surprised with what users do with it. The easy way is to design a system using the Matryoshka (or Russian Doll) principle so that each layer is complete and perfectly suited to what the layer does so that other layers may be built on-top or around it.

The Matryoshka principles led to the creation of many successful platform plays like Stripe, Segment, PayPal, AppDynamics, DataDog, NewRelic, SalesForce, Facebook, Google, Slack, etc that benefited the greater good of our software ecosystem

In an attempt to abuse this principle, the Russians and other nation-state actors have now shifted their attention from social-media deception to supply-chain infiltration.

Connecting backdoors : The mutation of an “new” attack pattern — one backdoor opens several others

While analyzing the recent SolarWinds supply-chain attack security researchers have found a second backdoor, suggesting involvement of another hacker group, unrelated to the suspected government-backed threat actor that compromised SolarWinds.

Tracked as Supernova, the backdoor is a memory resident web-shell injected into SolarWinds Orion code that would allow threat actors to execute arbitrary code on systems running the compromised version of Orion. Supernova web shell was used to download, compile and execute a malicious PowerShell script (dubbed CosmicGale by some researchers).

How can we keep up with this mutation?

What does your security team need to do?

  • The attacker has officially infiltrated your SDLC pipeline. Every SaaS vendor creating / distributing agents, services, APIs are ADVISED to conduct NG-SAST (Next Generation Static Analysis) with support to detect logic abuse and backdoor detection across every commit/patch/build. Summary reports SHOULD be shared preemptively with their upstream transitive supply-chain on a regular and random basis.
  • Gut out legacy SAST (Static Analysis) tools that were deployed over a decade ago to meet your compliance needs as compliance is not equal security. We are past the point of “#devsecops slogans, vanilla CVE based software composition checks, snake oil marketing tactics and compliance based checkbox security”. Besides bolstering your defense-in-depth, use a next generation SAST (static analysis) solution that is capable of going beyond pattern matching or taint analysis (to known CVEs) to detect complex backdoor patterns, business logic abuse, etc. The attacker has evolved and mutated, so should your practices.
  • Emerging attack techniques are designed to subvert your legacy static and dynamic analysis engines as indicated here. So examine every signal from NG-SAST toolchain (despite of it being categorized as moderate or low severity). A moderate vulnerability in patch-I can escalate to become severe in the next patch.

What expectations would you need to set amongst your vendor supply chain?

  • Set a mandate across all of your vendors to submit recent reports of continuous security code analysis. Do not trust or accept ineffective spreadsheet based security assessments. If CIOs don’t demand stricter assessments from their supply chain, software vendors will continue to prioritize features with little regard for security.

We at ShiftLeft have been studying and provisioning backdoor/insider detection policies using code property graph since mid 2019. Speak to us and we can help assess and recommend more efficient processes and procedures.


A Month of Reckoning for SaaS software creators and consumers was originally published in ShiftLeft Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

*** This is a Security Bloggers Network syndicated blog from ShiftLeft Blog – Medium authored by Chetan Conikee. Read the original post at: https://blog.shiftleft.io/a-month-of-reckoning-for-saas-software-creators-and-consumers-da791a4189e9?source=rss—-86a4f941c7da—4