Christian Schneider | @cschneider4711

Security DevOpsWie Sie in agilen Projekten trotzdem sicher bleiben

`whoami`

• Software Developer, Whitehat Hacker & Trainer

• als Freelancer im Projektgeschäft seit 1997

• Fokus auf JavaEE Architekturen und Web Security

• Speaker auf Konferenzen (JAX, WebTechCon, OWASP AppSecEU, DevOps Conference, …) + HackPra

• @cschneider4711

www.mail@ Christian-Schneider.net

Security – Status Quo

Static Application Security Testing (SAST)

Dynamic Application Security Testing (DAST)

Fun Part

The Ratio Problem

Dev OpsSec

100 10 1 : :

The Frequency Problem

mehr Rollouts als Pentests

Security DevOps ?

Integrate Security Tools into Build-Jobs / CI-Chain

on every commit or nightly

Reaching the different levelsof "Security DevOps" integration…

Security DevOps Maturity Model (SDOMM)

implicit masterfour axes each with four belts as small steps

Four different axesDynamic Depth

Intensity

Static DepthConsolidation

4

3

2

1

1

2

3

4

4 3 2 1 1 2 3 4

Four different axesDynamic Depth

Intensity

Static DepthConsolidation

4

3

2

1

1

2

3

4

4 3 2 1 1 2 3 4

Axis of "Dynamic Depth"

How deep are dynamic scans executed within a Security DevOps CI chain?

i.e. "where" are dynamic security tests applied?

Axis "Dynamic Depth": Level 1Scanning of public attack surface (pre-auth):

• Spidering of UI layer

• No requirement to authenticate scanner with target

• Easy integration of scanner(s) in nightly build as post-step

• "Throw tool at it (in CI-chain) and see what it generates…"

Tool example: ZAP"OWASP ZAP" features:

• passive scanning (Proxy / Spider)

• active scanning (Proxy / Spider)

• manual Payload Delivery (Intercepting Proxy)

• Spider (classic & AJAX)

• Fuzzing

• Plugins

• …

Request/Response History (Proxy)

Active Scanning…

Findings

ZAP in SecDevOps?"OWASP ZAP" features relevant for Security DevOps integration:

• …

• Headless Operation Mode / Daemon

• REST-API (with several language bindings as pre-built clients)

• Scriptable

• CLI

ZAP + Jenkins = SecDevOps?"OWASP ZAP" (spider & scanner) + Jenkins plugin "ZAProxy"

• Allows us to "Spider & Scan" as step in build job via Jenkins plugin

• Point plugin config to URL of integration system to test

• Plugin saves HTML-report in project’s job for inspection

• Best as separate Jenkins job to run during nightly build (duration)

ZAProxy Jenkins Plugin: ZAP startup

ZAProxy Jenkins Plugin: ZAP scan

UI- and Service-Tests in CI

Web Browser

Build-Scripts

UI-Tests (Selenium, etc.)

Service-Tests(REST, SOAP)

Application Server

CI Jobs

… enhanced with OWASP ZAP

Web Browser

Build-Scripts

UI-Tests (Selenium, etc.)

Service-Tests(REST, SOAP)

Application Server

OWASP ZAP

CI Jobs

passive & active scanning

place report (html,xml) into workspace

HTML report of ZAP scan

Tool example: Arachni"Arachni Scanner" features relevant for Security DevOps integration:

• Passive scanning (Proxy / Spider)

• Active scanning (Proxy / Spider)

• Uses internally a headless browser-cluster (for apps with lots of JS)

• Automation?

• CLI + RPC API

• Web-UI (helpful when permanently running as server)

Arachni + Jenkins = SecDevOps?"Arachni Scanner" + Jenkins CLI step in build

• Start in build job as CLI step and point to URL of system under test

• Generate HTML report and place into workspace for inspection

• Better execute within nightly build job (due to duration)

HTML report of Arachni scan (Summary) Bildquelle: Arachni Scanner

HTML report of Arachni scan (Details) Bildquelle: Arachni Scanner

Tool example: BDD-SecurityBDD-based framework for functional and technical security tests:

• Technical security tests (i.e. check against XSS, SQL-Injection, XXE, etc.)

• mainly uses ZAP as scanning engine

• Functional security tests (i.e. check UserA doesn’t see data from UserB)

• Tightly integrates with Selenium based app navigation workflows

• Uses JBehave for G/W/T stories & reporting

• Can run within CI (Jenkins, etc.) due to JBehave or as JUnit tests

BDD-Security Story: Scan for XSS

HTML report of BDD-Security

Other dynamic scanners exist also…Some have different focus

• Nikto

• w3af

• IronWASP

• Nessus

• Gauntlt (BDD-based Security tool orchestration)

• Mittn (BDD-based)

• Minion (integration framework)

• OWTF (integration framework)

• good commercial ones

• etc.

Axis "Dynamic Depth": Level 2Scanning of authenticated parts (= "post-auth") via UI layer

• Properly maintaining sessions

• Logout-detection & automatic re-login

• Different users / roles

• Spider & scan post-auth

Handling of hardening measures of application under test

• CSRF-Tokens, CAPTCHAs, etc.

Guide ZAP into Post-AuthMany ways to give authentication infos to ZAP

• Use ZAP manually to configure "Context": RegExps for Logged-In/Out Indicators, Users etc. + save as "ZAP Session-File" (could be in SCM)

• use that "Session-File" from SCM as starting point of scan (loaded via Jenkins plugin)

• Also Scripts in JavaScript or Zest can be registered in ZAP context to programmatically give authentication to ZAP

• Alternatively set these auth values via ZAP’s REST-API during each build before scan starts (from Jenkins/Maven/…)

"Forced User Mode" to stick to certain users during scans

Login config example within ZAP

ZAProxy Jenkins Plugin: ZAP session use

Guide Arachni into Post-AuthGive authentication infos to Arachni (Auth, Logged-In Indicators, Users)

• Use Arachni "autologin" plugin to specify via command line

• Login URL, formfield names, credentials, logged-in indicator, excludes

• Alternatively write custom ruby script for "login_script" plugin

• Individual custom login logic possible

• Logged-In indicators (RegExp) to know when to re-login

Login config example within Arachni

./arachni --plugin=autologin: url=http://10.33.48.80/marathon/secured/profile.page, parameters='j_username=john&j_password=john', check='Logout' --scope-exclude-pattern=logout.page http://10.33.48.80/marathon

Or individual ruby script if more custom login logic required…

Guide BDD-Security into Post-AuthUse Selenium to navigate through the login process

• Based on excellent integration of BDD-Security with Selenium

• Separate app navigation code (Selenium) from Security testing code

• Use (along with test user credentials) within BDD stories

• Perform further spidering & active scanning (through ZAP) post-auth

BDD-Security based stories integrate with custom code to deliver test CAPTCHA value to scanner

• while also checking that mismatches are handled with denial)

What about CSRF-Tokens?ZAP can handle CSRF-Tokens

• Can be individualised via Config

BDD-Security uses (among others) ZAP for scanning

Arachni can also handle CSRF-Tokens during spidering & scanning

Apply targeted delta scanning:

• Developer teams maintain "scope delta files" in SCM

• like "what parts of the application have changed during this sprint" (= URL patterns)

• DAST Scanner uses it to scan only this during onCommit or nightly CI run

• Feed this delta information as argument to scope definitions Arachni: --scope-include-pattern / --scope-exclude-pattern ZAP: Include/Excludes in Context (via Session-File or REST-API) BDD-Security: maintain as "table file" of RegExps as scope

If scanning takes too long?

Train spiders about the structure of application to reduce scan redundancy

• ZAP: use RegExp "Exclude from scanner" rules to prevent redundant scans in data-driven applications (i.e. scan only one instance of it)

• Arachni: --scope-redundant-path-pattern / --scope-auto-redundant

Don’t forget to scan complete application at least on nightly or weekly run

• Only apply full scan, if simple smoke-tests of system under test are green

If scanning takes too long?

Axis "Dynamic Depth": Level 3Separate scanning of different application layers / backends

• Scan internal WebServices (e.g. SOAP / REST) = directly scan backends

• Detect and scan parameter positions within XML, JSON, …

• Scan from "within" the different application’s layers

• IAST with distributed agents & instrumentation aims into that direction

• At least one simple step in that direction:

• Use the proxy also between your backend service calls

Backend scans with ZAPHow to achieve this with ZAP?

• ZAP operates as proxy server: place it between backend calls

• ZAP can inject payloads in observed XML tags/attributes & JSON fields

• Capture service call traffic in integration test during CI while either

A. executing service tests that directly access the service endpoint, or

B. frontend UI tests execute service backend calls indirectly

• Automatically scan as new requests are seen: "ATTACK Mode"

Also keep an eye on an alpha-level SOAP-Scanner (by WSDL) ZAP addon

Backend scans with ArachniHow to achieve this with Arachni?

• Arachni can also operate as proxy server: place it between backend calls

• Use passive proxy plugin to "train" Arachni of the XML / JSON requests

• New addition in v1.1 to extract XML / JSON input vectors from that

• Use that collected input vector data to feed the active scan for the observed requests

Axis "Dynamic Depth": Level 4Targeted scanning of individual forms / wizards (UI) and service layers

• More individualised workflow coverage (not just simple spidering)

• Business-logic compliant usage patterns & inputs

• "fill shopping cart followed by checkout process"

• "access backend WebServices in special order to test workflow"

• etc.

• Custom coded security tests tailored to the application

ZAP with special workflows (1/2)Re-use existing UI tests (Selenium, …) and/or WebService tests

• Proxy Selenium-generated traffic through ZAP (in integration test phase) in ATTACK-Mode

More individual: Re-use & enhance existing "UI test code" at the proper workflow steps with points of where to attack by calling ZAP’s REST-API (simple to use REST-API clients are pre-shipped with ZAP)

• Type of scans at these positions can be individualised (policy config, etc.)

• See https://github.com/continuumsecurity/zap-webdriver for a working example with ZAP

• Start ZAP as Daemon from Jenkins with Zapper Jenkins plugin

ZAP with special workflows (2/2)Alternatively "train" ZAP about the workflow by recording Zest scripts

• Keep an eye on "Sequence Scanning" alpha-level ZAP extension

• Still alpha-level (as of May 2015), but interesting approach

Use Selenium to further drive BDD-Security initiated checks:

• Selenium-based test code navigates application workflows

• This code is integrated with BDD (via Java interfaces), so that:

• BDD-Security stories can use that code to navigate and generate traffic

• This generated traffic will be scanned by ZAP via BDD

BDD with special workflows

BDD-Security: Selenium integration

If no Selenium testcases exist?Simply give developer teams access to ZAP to pre-seed the scanner (at least):

• Developer teams use browser to navigate app workflows while proxying

• Thereby seed the ZAP session(s) with navigation nodes/workflows

• Save the ZAP session(s) and check-in into SCM (Git, SVN, …)

• Point the Jenkins ZAP plugin to the saved ZAP session(s) as starting point

• Devs can add to this list of URLs for ZAP with each new UI

BTW: ZAP is also available as Docker image…

Security-UnitTests for APIsSometimes it’s useful to have programmed unit-tests directly calling a business API

• Test directly at the API-level (i.e. Java code level)

• Utilise fuzzing values from popular sources like "fuzzdb" and/or generate on-the-fly with "radamsa"

• Check the API method’s response (like regular unit test, but seeking for signs of successful attack tests)

• Leverage the benefit of direct API-level testing (i.e. testing "from within")

• Place "probes" into the application to watch out for successful attack tests

• Mock objects (Mockito, etc.) or weave aspects (AOP) collecting data (i.e. method calls and arguments, etc.)

• Even logs (of error messages) are good candidates to check

Axis of "Static Depth"

How deep is static code analysis performed within a Security DevOps CI chain?

i.e. "where" are static security tests applied?

Axis "Static Depth": Level 1Assurance that no third-party code with known vulnerabilities is used

• Check application’s server-side dependencies

• For Java applications: JAR files, etc.

• Check application’s client-side dependencies

• JavaScript frameworks, etc.

Useful even for projects not currently under development

• New vulnerabilities could become public: Build regularly to check

Tool: OWASP Dependency CheckScans all dependencies (even transitive ones) against CVE list

• Available as Maven plugin and Ant task

• CLI version also available

• Jenkins plugin for nice reporting (and build breaking thresholds)

• Not false-positive free

• regularly needs some time to triage findings

Reporting in Jenkins

Findings (CVE-Details)

For JavaScript libs: retire.jsScans application’s JavaScript files against list of known vulnerable ones

• Available also as Maven Plugin…

• … and as CLI (point it to your folder containing .js files)

Axis "Static Depth": Level 2Scan important parts of source code for vulnerability patterns

• At least important parts of applications’ codebases are scanned

• like in-house reused code & custom developed frameworks

• for big multi-project Maven projects: at least frontend and backend projects

• or code of changed modules within a sprint

Java: FindSecurityBugsPlugin for FindBugs with enhanced checks for security issues in Java code

• Runs within FindBugs, so that it can execute in Maven and/or Sonar

• Generates XML-file with potential findings

• Jenkins plugin for representing results (also possible to display in Sonar)

• Integrates with Jenkins "Static Code Analysis" as part of FindBugs

• Delta reporting of new (and fixed) findings & trends

FindBugs in Jenkins: Trends, etc.

in Sonar: Vulnerability & Code-Pointer

JavaScript: ScanJSScans JavaScript code for vulnerabilities

• Most importantly DOM-based XSS (Cross-Site Scripting) issues

• Flags also "Usage of sensitive API" issues

• Executable as CLI from within Jenkins build steps

• Optional Web-UI (running as local server)

ScanJS Findings as HTML: DOM-XSS

Ruby on Rails apps: BrakemanScans RoR code for vulnerabilities

• CLI based

• Nicely integrates with Jenkins

• Rendering scan results and trend analysis

• Allows to compare scan results (delta detection)

• "What has been fixed and what was newly introduced?"

Other static scanners exist also…

Some have different focus

• PrimeJS (like ScanJS for JavaScript regarding DOM-XSS etc.)

• PMD (for Java code regarding API safety issues)

• good commercial ones

• etc.

Axis "Static Depth": Level 3Scan the complete applications’ source code for vulnerability patterns

• Just like "Level 2", but now for the complete codebase (no excludes)

• i.e. all projects of a multi-project Maven project

• best via parent or corporate pom.xml

For FindSecurityBugs this means:

• pre-compile even all JSPs (if you have any) to scan them

Axis "Static Depth": Level 4Scan source code of (important) third-party dependencies used in application

• If the component is open-source:

• include it in regular scanning activities

• If the component is closed-source:

• consider using code scanners that operate on binaries

• FindBugs (hence its FindSecurityBugs plugin) can also scan binaries, since it operates on Bytecode

Axis of "Intensity"

How intense are the majority of the executed attacks within a Security DevOps CI chain?

i.e. "what" is being checked for?

Axis "Intensity": Level 1Dynamic checks: Only passive scanning

• Simply proxy testcase generated traffic through passive operating scanners

• Quick-win for using existing testcase code (i.e. Service- or UI-tests)

• Can execute within existing tests that execute on every commit

• No requirement for a separate nightly CI job (i.e. doesn’t slow down build)

Static checks: Just scan the code along with other code metrics

• Use the default rules of the scanners and integrate them into CI

Axis "Intensity": Level 2Use lightweight active scanning options for dynamic checks

• ZAP: active scanning of observed URLs (proxied or spidered)

• perform an "active scan": For Jenkins ZAProxy Plugin provide a ZAP policy with desired active scan intensity (or via ZAP’s REST-API during build for other setups)

• Arachni: enable "active scan" during spidering

• BDD-Security stories can call ZAP to actively attack the application

Better to execute in separate (nightly) build job due it’s to duration

Think of resetting databases (+ more environment) prior or after test execution

Axis "Intensity": Level 3Use heavyweight scanning options on important parts of application

Dynamic checks: create "riskier" custom attack profiles

ZAP:

• set "Threshold" to "Low" / "Strength" to "High" (or selectively "Insane")

• enable more scanning rules via Policy Files (could be maintained & used via SCM)

• use ZAP’s "Advanced SQLInjection Scanner" extension (or directly call sqlmap)

Arachni: Enable the more aggressive "audit flags" like --audit-forms, …

Static checks: FindSecurityBugs: set "Threshold" to "Low" / "Effort" to "Max"

Axis "Intensity": Level 4Use customised rule sets for dynamic checks

• ZAP (which is very extensible via custom scripts, recored and customised Zest or plugins):

• Custom coded scan scripts as active & passive scan rules for company-individual checks

• Security regression tests (in Zest) against previous found vulns (also logic flaws)

• Custom input vector scripts: e.g. define certain injection points in custom encoded params

• Proxy scripts (for example for generic request/response modifications)

Scripting possibilities in ZAP

Axis "Intensity": Level 4Use customised rule sets for static checks

• FindSecurityBugs: Custom coded rule files (in Java) to check for wrong usages of server-side inhouse frameworks (or alike)

• ScanJS: Custom rule files (in JSON) to check for wrong usages of client-side inhouse frameworks (or alike)

Axis of "Consolidation"

How complete is the process of handling findings within a Security DevOps CI chain?

i.e. "how" are the results used?

Axis "Consolidation": Level 1Generate human-readable (HTML) reports from tools and link them in Jenkins

• All relevant mentioned static and dynamic scanners generate HTML reports

• Collect and publish them in Jenkins build: via Jenkins "HTML Publisher Plugin"

Use simple criteria to "break the build" on heavy findings (ok, at least "unstable")

• Dependency-Check, BDD-Security (with the JBehave-stories), FindSecurityBugs (via Sonar when rated as blocker), Arachni (via Gauntlt execution with BDD-like stories), etc. all have capabilities to automatically flag the build

• For others: at least do a simple log parse from Jenkins "Log Parser Plugin" to flag the build as instable and/or broken

OWASP Dependency Check: Thresholds to break build

Reports & SessionsSimple reporting with ZAP in CI:

• Export HTML report & link from build (via Jenkins Plugin "ZAProxy" or "HTML Publisher")

• Save the ZAP session with build for later manual inspection (by security team)

For Arachni scans in CI:

• Generate & link readable HTML report from that (via arachni-report tool in build)

• Optionally also save AFR file with build

BDD-Security directly generates reports in JBehave formats (HTML, XML, …) or via JUnit from within build job

Jenkins HTML Publisher Plugin: Configuration of HTML reports to link

Jenkins HTML Publisher Plugin: Result in build

FindSecurityBugs: IDE support & Collaboration

FindBugs has excellent Jenkins integration

Additionally consider placing the FindBugs generated XML report file within Jenkins

• Developers can load the XML report file into FindBugs IDE plugins (Eclipse, IntelliJ, NetBeans, …)

• Developers & Security Analysts can collaborate on bugs via the IDE plugins

• See https://code.google.com/p/findbugs/wiki/FindBugsCloudTutorial for more details

• Consider installing it as your private on-premise "FindBugs"-Server to keep results only on your servers…

FindSecurityBugs Eclipse Perspective with Collaboration View + Remediation

Axis "Consolidation": Level 2Custom logic to make build unstable and/or broken depending on

• Type of vulnerability (CWE or WASC or …)

• Confirmation level (firm)

• Severity rating (high risk)

Provide useful remediation info to developers

Respect suppression mechanisms to rule out false positives

Flagging builds from reportsHow (from within a CI job)?

• Most scanners also emit XML reports that can be parsed

• mostly a simple XPath count should be fine

• Alternatively fetch the results by accessing the scanner’s APIs

Providing useful infos to DevsLookup findings by CWE-ID (provided by most scanners) in vulndb on GitHub

• https://github.com/vulndb/data (currently 65+ vulns described)

• Useful if the messages from scanners are too generic or don’t help much

• Maps vulnerability to useful remediation messages

• Possibility to extend that repository locally with company-individual texts

Suppression of false positivesBDD-Security

• Use false positive tables in story files

FindSecurityBugs

• Use exclusion lists in config (XML filter files)

• Use annotation @SuppressWarnings on false positive code lines

Dependency-Check

• XML file of suppressions (checked-in along with the project)

retire.js

• "ignore file" parameters on CLI (cloud be loaded in CI from SCM)

Axis "Consolidation": Level 3Consolidation goals:

• Consolidate & de-duplicate findings from different scanner reports (with better false positive handling)

• Push consolidated findings into established bug-tracker (known to devs)

• Delta analysis & trends over consolidated data sets

ThreadFix as result consolidatorUse a local ThreadFix server, which imports native scanner outputs

• does the heavy lifting of consolidation & de-duplication

• pushes findings toward bug-tracker and IDE (via plugins)

• process can be customised using it’s own REST-API

• ThreadFix integrates with ZAP to import scans from ZAP

Report generated by ThreadFixBildquelle: ThreadFix

Trend Chart generated by ThreadFixBildquelle: ThreadFix

Trends about vulnerable dependencies

OWASP Dependency-Track provides trend insight into findings about use of vulnerable components (from OWASP Dependency-Check):

• across multiple applications and versions

• across time

Trend Chart generated by OWASP Dependency Track Bildquelle: OWASP

Axis "Consolidation": Level 4Measure the concrete code coverage of your security testing activities

• Find untested "white spots"

• Derive where static checks and code reviews should focus more to compensate

How?

• Use "OWASP Code Pulse", which instruments your Java app via agent

• collects coverage data during dynamic security testing scans

• generates reports ("code treemaps") of coverage

Code Treemap of dynamic scan coverageBildquelle: OWASP Code Pulse

Without CI build chain?Now that we’ve seen several ways to introduce security

checks into the build chain (four axes with four levels each)

…

What if we don’t have a full-blown CI?

Or simply cannot extend it that way (i.e. build job would run too long)?

Use recurring scans instead!As an alternative to a build-chain based scan recurring scans can be used:

• Don’t block the build, but get nightly or weekly results delivered

• No need to integrate into full build chain

• Still leverage the other benefits

• Dev-maintained delta files defining scopes

• Consolidation levels of results

• Basically all four levels for each axis apply the same

• = Quick-win for DAST + SAST scans independent of build-chain

Recurring scans: ArachniRemote Arachni (non-blocking in build chain):

• Instead of a scan step in Jenkins build, the RPC-API of aremote Arachni server can be used to run/schedule a scan

• Collect findings separately after scan or link to WebUI of Arachni Server to present them

Recurring scans: ZAPRemote OWASP ZAP (non-blocking in build chain):

• Instead of a scan step in Jenkins build, the REST-API of a remote ZAP can be used to run/schedule a scan (cron, etc.)

• Collect findings separately after scan

• Or use Seccubus to have an integrated framework for scan scheduling with ZAP

• also has comparison of different scans to report deltas

• ZAP authenticated scans scheduled via Seccubus: -o "--auth-form http://example.com/login --auth-user myuser --auth-pass mypass --auth-verify-url http://example.com/profile -X /logout" --hosts @HOSTS

Thank you very much!

LinksOWASP ZAP https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_ProjectZAP Selenium Demo https://github.com/continuumsecurity/zap-webdriverZAP Jenkins Plugin https://wiki.jenkins-ci.org/display/JENKINS/ZAProxy+PluginBDD-Security http://www.continuumsecurity.net/bdd-intro.htmlArachni http://www.arachni-scanner.comOWASP Dependency Check https://www.owasp.org/index.php/OWASP_Dependency_CheckOWASP Dependency Track https://www.owasp.org/index.php/OWASP_Dependency_Track_ProjectFindSecurityBugs http://h3xstream.github.io/find-sec-bugs/FindSecurityBugs-Cloud https://code.google.com/p/findbugs/wiki/FindBugsCloudTutorialretire.js http://bekk.github.io/retire.js/ScanJS https://github.com/mozilla/scanjsJenkins Log Parser Plugin https://wiki.jenkins-ci.org/display/JENKINS/Log+Parser+PluginThreadFix http://www.threadfix.orgOWASP Code Pulse https://www.owasp.org/index.php/OWASP_Code_Pulse_Project

Interested in more web security stuff? Visit my Blog: www.Christian-Schneider.net @cschneider4711Bildquelle: dreamstime.com