esapi4java-core-2.2.1.0-release-notes.txt

Release notes for ESAPI 2.2.1.0
    Release date: 2020-July-12
    Project leaders:
        -Kevin W. Wall <kevin.w.wall@gmail.com>
        -Matt Seil <matt.seil@owasp.org>

Previous release: ESAPI 2.2.0.0, 2019-June-24


Executive Summary: Important Things to Note for this Release
------------------------------------------------------------

This is a minor release. It's main purpose was to update dependencies to eliminate potential vulnerabilities arising from dependencies with known CVEs. See the section "Changes requiring special attention" below for additional details.

Also special props to Bill Sempf for stepping up and volunteering to prepare the initial cut of these release notes. Had he not done so, this release either would not have release notes or it would have been delayed another 6 months while I procrastinated further with various distractions. (Squirrel!)

=================================================================================================================

Basic ESAPI facts
-----------------

ESAPI 2.2.0.0 release:
    194 Java source files
    4150 JUnit tests in 118 Java source files

ESAPI 2.2.1.0 release:
    211 Java source files
    4309 JUnit tests in 134 Java source files

39 GitHub Issues closed in this release

Issue #         GitHub Issue Title
----------------------------------------------------------------------------------------------

143 - Enhance encodeForOS to auto-detect the underling OS
173 - DOMConfigurator is being used inappropriately in the ESAPIWebApplicationFirewallFilter
226 - Javadoc Inaccuracy in getRandomInteger() and getRandomReal()
232 - SecurityWrapperResponse.createCookieHeader modification request  (closed; marked 'wontfix')
235 - exception is java.lang.NoClassDefFoundError: org.owasp.esapi.codecs.Codec
245 - KeyDerivationFunction::computeDerivedKey - possible security level mismatch
256 - Whitespace in JavaEncryptor
263 - I am getting validation exception while validating a parameter coming from http request
268 - SecurityWrapperResponse setStatus should not always set SC_OK
269 - org.owasp.esapi.reference.DefaultValidator reports ValidationException with IE 9
271 - Add Constructor to DefaultSecurityConfiguration to accept a properties file (1.4)
276 - Patch for /branches/2.1/src/main/java/org/owasp/esapi/reference/DefaultExecutor.java
310 - Make HTMLValidationRule to look for antisamy-esapi.xml in classpaths enhancement
382 - Build Fails on path with space
465 - Update both ESAPI.properties files to show comment for ESAPI logger support for SLF4J
488 - Missed a legal input case in DefaultSecurityConfiguration.java
494 - Encoder's encodeForCSS doesn't handle RGB Triplets
495 - Maven Install Requires GPG Key
499 - ValidatorTest.isValidDirectoryPath() has tests that fail under Windows if ESAPI tests run from different drive where Windows installed
500 - Suppress noise from ESAPI searching for properties and stop ignoring important IOExceptions
503 - Bug on on referrer header when value contains `&section` like `www.asdf.com?a=1&section=2`
509 - HTMLValidationRule.getValid(String,String) does not follow documented specifications
511 - Add missing documentation to Validator.addRule() and Validator.getRule()
512 - Update Apache Commons Bean Utils to 1.9.4
515 - NullPointerException can result from line 163 of SecurityWrapperRequest.java:
521 - JUnit test ValidatorTest.testIsValidSafeHTML() now failing
522 - javadoc corrections for Encoder.canonicalize()
523 - Links in README to users list and dev list are reversed
527 - Configuration flag for disabling Logger User and App Information
530 - Apply default logging content to SLF4J messages
532 - Update JUL and Log4J code structure and workflow to match SLF4J
536 - SecurityWrapperResponse setHeader error message is unclear
538 - Addressing log4j 1.x CVE-2019-17571
542 - Write up ESAPI release notes for planned 2.2.1.0 release
552 - Rewrite implementation of some ESAPI classes to remove Java 8 dependencies
554 - CryptoHelper.arrayCompare() fails with NPE under Java 7 when one of the arguments is null
555 - JUnit test org.owasp.esapi.reference.AccessControllerTest.testIsAuthorizedForData fails when run against Java 7 on Linux
556 - Major overhaul to ESAPI Encoder Javadoc based on ESAPI Encoder Usability Study
558 - ValidatorTest.testIsValidDirectoryPath() JUnit test fails under MacOS

-----------------------------------------------------------------------------

        Changes Requiring Special Attention

-----------------------------------------------------------------------------
The new default ESAPI logger is JUL (java.util.logging packages) and we have deprecated the use of Log4J 1.x because we now support SLF4J and Log4J 1.x is way past its end-of-life. We did not want to make SLF4J the default logger (at least not yet) as we did not want to have the default ESAPI use require additional dependencies. However, SLF4J is likely to be the future choice, at least once we start on ESAPI 3.0. A special shout-out to Jeremiah Stacey for making this possible by re-factoring much of the ESAPI logger code. Note, the straw that broke the proverbial camel's back was the announcement of CVE-2019-17571 (rated Critical), for which there is no fix available and likely will never be.

Related to that CVE and how it affects ESAPI, be sure to read
   https://github.com/ESAPI/esapi-java-legacy/blob/develop/documentation/ESAPI-security-bulletin2.pdf
which describes CVE-2019-17571, a deserialization vulnerability in Log4J 1.2.17. ESAPI is not affected by this (even if you chose to use Log4J 1 as you default ESAPI logger). This security bulletin describes why this CVE is not exploitable as used by ESAPI.

Notable dependency updates (excludes those only used with JUnit tests):
    antiSamy            1.5.8   ->  1.5.10
    batik-css           1.11    ->  1.13
    commons-beansutil   1.9.3   ->  1.9.4
    slf4j-api           1.7.26  ->  1.7.30

Finally, while ESAPI still supports JDK 7 (even though that too is way past end-of-life), the next ESAPI release will move to JDK 8 as the minimal baseline. (We already use Java 8 for development but still to Java 7 source and runtime compatibility.)

-----------------------------------------------------------------------------

        Known Issues / Problems

-----------------------------------------------------------------------------
If you use Java 7 (the minimal Java baseline supported by ESAPI) and try to run 'mvn test' there is one test that fails. This test passes with Java 8. The failing test is:

        [ERROR] Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.203 s
        <<< FAILURE! - in org.owasp.esapi.crypto.SecurityProviderLoaderTest
        [ERROR] org.owasp.esapi.crypto.SecurityProviderLoaderTest.testWithBouncyCastle
        Time elapsed: 0.116 s <<< FAILURE!
        java.lang.AssertionError: Encryption w/ Bouncy Castle failed with
        EncryptionException for preferred cipher transformation; exception was:
        org.owasp.esapi.errors.EncryptionException: Encryption failure (unavailable
        cipher requested)
        at
        org.owasp.esapi.crypto.SecurityProviderLoaderTest.testWithBouncyCastle(Security
        ProviderLoaderTest.java:133)

I will spare you all the details and tell you that this has to do with Java 7 not being able to correctly parse the signed Bouncy Castle JCE provider jar. More details are available at:
    https://www.bouncycastle.org/latest_releases.html
and
    https://github.com/bcgit/bc-java/issues/477
I am sure that there are ways of making Bouncy Castle work with Java 7, but since ESAPI does not rely on Bouncy Castle (it can use any compliant JCE provider), this should not be a problem. (It works fine with the default SunJCE provider.) If it is important to get the BC provider working with the ESAPI Encryptor and Java 7, then open a GitHub issue and we will take a deeper look at it and see if we can suggest something.



Another problem is if you run 'mvn test' from the 'cmd' prompt (and possibly PowerShell as well), you will get intermittent failures (generally between 10-25% of the time) at arbitrary spots. If you run it again without any changes it will work fine without any failures. We have discovered that it doesn't seem to fail if you run the tests from an IDE like Eclipse or if you redirect both stdout and stderr to a file; e.g.,

    C:\code\esapi-java-legacy> mvn test >testoutput.txt 2>&1

We do not know the reason for these failures, but only that we have observed them on Windows 10. If you see this error, please do NOT report it as a GitHub issue unless you know a fix for it.


    *** IMPORTANT WORKAROUND for 2.2.1.0 ESAPI Logging ***

Lastly, if you try to use the new ESAPI 2.2.1.0 logging, you will notice that you need to change ESAPI.Logger and also possibly provide some other logging properties as well. This is because the logger packages were reorganized to improve maintainability, but we failed to mention it. To use ESAPI logging in ESAPI 2.2.1.0 (and later), you MUST set the ESAPI.Logger property to one of:

    org.owasp.esapi.logging.java.JavaLogFactory     - To use the new default, java.util.logging (JUL)
    org.owasp.esapi.logging.log4j.Log4JLogFactory   - To use the end-of-life Log4J 1.x logger
    org.owasp.esapi.logging.slf4j.Slf4JLogFactory   - To use the new (to release 2.2.0.0) SLF4J logger

In addition, if you wish to use JUL for logging, you *must* supply an "esapi-java-logging.properties" file in your classpath. Unfortunately, we failed to drop add that to the ESAPI configuration jar under the GitHub 'Releases', so this file has been added explicitly to the 2.2.1.0 release 'Assets' for this release (for details, see https://github.com/ESAPI/esapi-java-legacy/releases/esapi-2.2.1.0). Even worse, there was a logic error in the static initializer of JavaLogFactory (now fixed in the 2.2.1.1 patch release) that causes a NullPointerException to be thrown so that the message about the missing "esapi-java-logging.properties" file was never seen.

If you are using JavaLogFactory or Slf4JLogFactory, you will also want to ensure that you have the following ESAPI logging properties set to get the logs to appear what you are used to with Log4J 1.x logging:
    # Set the application name if these logs are combined with other applications
    Logger.ApplicationName=ExampleApplication
    # If you use an HTML log viewer that does not properly HTML escape log data, you can set LogEncodingRequired to true
    Logger.LogEncodingRequired=false
    # Determines whether ESAPI should log the application name. This might be clutter in some single-server/single-app environments.
    Logger.LogApplicationName=true
    # Determines whether ESAPI should log the server IP and port. This might be clutter in some single-server environments.
    Logger.LogServerIP=true
    # LogFileName, the name of the logging file. Provide a full directory path (e.g., C:\\ESAPI\\ESAPI_logging_file) if you
    # want to place it in a specific directory.
    Logger.LogFileName=ESAPI_logging_file
    # MaxLogFileSize, the max size (in bytes) of a single log file before it cuts over to a new one (default is 10,000,000)
    Logger.MaxLogFileSize=10000000
    # Determines whether ESAPI should log the user info.
    Logger.UserInfo=true
    # Determines whether ESAPI should log the session id and client IP.
    Logger.ClientInfo=true

See GitHub issue #560 for additional details.
-----------------------------------------------------------------------------

        Other changes in this release, some of which not tracked via GitHub issues

-----------------------------------------------------------------------------

Documentation updates for locating Jar files
Unneeded code removed from ExtensiveEncoderURI test
Inline reader added to ExtensiveEncoder
Additional time for windows to always sleep more than given seconds in CryptoTokenTest
Change required by tweak to CipherText.toString() method
Removed call to deprecated CryptoHelper.computeDerivedKey() method
New JUnit tests for org.owasp.esapi.crypto.KeyDerivationFunction class
Miscellaneous documentation and tests
JavaLogger moved to new package
Log4J 1.x no longer ESAPI's default logger

-----------------------------------------------------------------------------

Developer Activity Report (Changes between release 2.2.0.0 and 2.2.1.0, i.e., between 2019-06-25 and 2020-07-11)
Generated manually (this time) -- all errors are the fault of kwwall and his inability to do simple arithmetic.

Developer       Total       Total Number        # Merged
(GitHub ID)     commits   of Files Changed        PRs
========================================================
davewichers      2              1               0
HJW8472         11              8               1
jeremiahjstacey 78             70               6
kwwall          67             64               8
Michael-Ziluck  3               2               2
sempf           1               1               1
wiitek          6               4               2
xeno6696        3               5               1
========================================================
                                        Total: 21


-----------------------------------------------------------------------------


21 Closed PRs merged since 2.2.0.0 release (those rejected not listed)
======================================================================
PR#    GitHub ID                Description
----------------------------------------------------------------------
504 -- kwwall           -- New scripts to suppress noise for 'mvn test'
505 -- kwwall           -- Close issue #256. White-space clean up.
506 -- kwwall           -- Closes Issue 245
508 -- Michael-Ziluck   -- Resolves #226 - Corrected docs for the bounded, numeric, random methods
510 -- Michael-Ziluck   -- Resolve #509 - Properly throw exception when HTML fails
513 -- kwwall           -- Close issue #512 by updating to 1.9.4 of Commons Beans Util.
514 -- xeno6696         -- Fixed issues #503 by writing a new addReferer method, also temporarily…
516 -- jeremiahjstacey  -- Issue 515
518 -- jeremiahjstacey  -- Issue #511 Copying Docs from DefaultValidator
519 -- jeremiahjstacey  -- Issue 494 CSSCodec RGB Triplets
520 -- jeremiahjstacey  -- OS Name DefaultExecutorTests #143
533 -- jeremiahjstacey  -- #532 JUL and Log4J match SLF4J class structure and Workflow
535 -- kwwall           -- Issue 521
537 -- jeremiahjstacey  -- Issue 536
539 -- wiiitek          -- upgrade for convergence
540 -- wiiitek          -- Issue 382: Build Fails on path with space
541 -- HJW8472          -- Fixed issue #310
543 -- sempf            -- Release notes for 2.2.1.0
551 -- kwwall           -- Misc cleanup
553 -- kwwall           -- Fix for GitHub Issue 552
557 -- kwwall           -- Final prep for 2.2.1.0 release


CHANGELOG:      Create your own. May I suggest:

        git log --since=2019-06-25 --reverse --pretty=medium

    which will show all the commits since just after the last (2.2.0.0) release.

-----------------------------------------------------------------------------

Direct and Transitive Runtime and Test Dependencies:

        $ mvn dependency:tree
        [INFO] Scanning for projects...
        [INFO]
        [INFO] -----------------------< org.owasp.esapi:esapi >------------------------
        [INFO] Building ESAPI 2.2.1.0
        [INFO] --------------------------------[ jar ]---------------------------------
        [INFO]
        [INFO] --- maven-dependency-plugin:3.1.2:tree (default-cli) @ esapi ---
        [INFO] org.owasp.esapi:esapi:jar:2.2.1.0
        [INFO] +- javax.servlet:javax.servlet-api:jar:3.0.1:provided
        [INFO] +- javax.servlet.jsp:javax.servlet.jsp-api:jar:2.3.3:provided
        [INFO] +- com.io7m.xom:xom:jar:1.2.10:compile
        [INFO] +- commons-beanutils:commons-beanutils:jar:1.9.4:compile
        [INFO] |  +- commons-logging:commons-logging:jar:1.2:compile
        [INFO] |  \- commons-collections:commons-collections:jar:3.2.2:compile
        [INFO] +- commons-configuration:commons-configuration:jar:1.10:compile
        [INFO] +- commons-lang:commons-lang:jar:2.6:compile
        [INFO] +- commons-fileupload:commons-fileupload:jar:1.3.3:compile
        [INFO] +- log4j:log4j:jar:1.2.17:compile
        [INFO] +- org.apache.commons:commons-collections4:jar:4.2:compile
        [INFO] +- org.apache-extras.beanshell:bsh:jar:2.0b6:compile
        [INFO] +- org.owasp.antisamy:antisamy:jar:1.5.10:compile
        [INFO] |  +- net.sourceforge.nekohtml:nekohtml:jar:1.9.22:compile
        [INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.12:compile
        [INFO] |  |  \- org.apache.httpcomponents:httpcore:jar:4.4.13:compile
        [INFO] |  \- commons-codec:commons-codec:jar:1.14:compile
        [INFO] +- org.slf4j:slf4j-api:jar:1.7.30:compile
        [INFO] +- commons-io:commons-io:jar:2.6:compile
        [INFO] +- org.apache.xmlgraphics:batik-css:jar:1.13:compile
        [INFO] |  +- org.apache.xmlgraphics:batik-shared-resources:jar:1.13:compile
        [INFO] |  +- org.apache.xmlgraphics:batik-util:jar:1.13:compile
        [INFO] |  |  +- org.apache.xmlgraphics:batik-constants:jar:1.13:compile
        [INFO] |  |  \- org.apache.xmlgraphics:batik-i18n:jar:1.13:compile
        [INFO] |  +- org.apache.xmlgraphics:xmlgraphics-commons:jar:2.4:compile
        [INFO] |  \- xml-apis:xml-apis-ext:jar:1.3.04:compile
        [INFO] +- xalan:xalan:jar:2.7.2:compile
        [INFO] |  \- xalan:serializer:jar:2.7.2:compile
        [INFO] +- xerces:xercesImpl:jar:2.12.0:compile
        [INFO] +- xml-apis:xml-apis:jar:1.4.01:compile
        [INFO] +- com.github.spotbugs:spotbugs-annotations:jar:4.0.4:compile (optional)
        [INFO] |  \- com.google.code.findbugs:jsr305:jar:3.0.2:compile (optional)
        [INFO] +- net.jcip:jcip-annotations:jar:1.0:compile (optional)
        [INFO] +- junit:junit:jar:4.13:test
        [INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
        [INFO] +- org.bouncycastle:bcprov-jdk15on:jar:1.65.01:test
        [INFO] +- org.powermock:powermock-api-mockito2:jar:2.0.7:test
        [INFO] |  \- org.powermock:powermock-api-support:jar:2.0.7:test
        [INFO] |     \- org.powermock:powermock-core:jar:2.0.7:test
        [INFO] +- org.javassist:javassist:jar:3.25.0-GA:test
        [INFO] +- org.mockito:mockito-core:jar:2.28.2:test
        [INFO] |  +- net.bytebuddy:byte-buddy:jar:1.9.10:test
        [INFO] |  +- net.bytebuddy:byte-buddy-agent:jar:1.9.10:test
        [INFO] |  \- org.objenesis:objenesis:jar:2.6:test
        [INFO] +- org.powermock:powermock-module-junit4:jar:2.0.7:test
        [INFO] |  \- org.powermock:powermock-module-junit4-common:jar:2.0.7:test
        [INFO] +- org.powermock:powermock-reflect:jar:2.0.7:test
        [INFO] +- org.openjdk.jmh:jmh-core:jar:1.23:test
        [INFO] |  +- net.sf.jopt-simple:jopt-simple:jar:4.6:test
        [INFO] |  \- org.apache.commons:commons-math3:jar:3.2:test
        [INFO] \- org.openjdk.jmh:jmh-generator-annprocess:jar:1.23:test
        [INFO] ------------------------------------------------------------------------
        [INFO] BUILD SUCCESS
        [INFO] ------------------------------------------------------------------------

-----------------------------------------------------------------------------

Acknowledgments:

    Release notes written by Bill Sempf (bill.sempf@owasp.org), but please direct any communication to the project leaders.

Special shout-outs to:
    Jeremiah Stacey (jeremiahjstacey) -- All around ESAPI support and JUnit test case developer extraordinaire and for refactoring ESAPI loggers.
    Dave Wichers (davewichers) - for several extremely useful pom.xml improvements.
    Bill Sempf (sempf) -- for these release notes. Awesome job, Bill. I owe you a brew.
    Chamila Wijayarathna <cdwijayarathna@gmail.com> and Nalin A. G. Arachchilage <nalin.arachchilage@gmail.com> for their authorship and subsequent extensive discussion of their paper "Fighting Against XSS Attacks: A Usability Evaluation of OWASP ESAPI Output Encoding" (https://scholarspace.manoa.hawaii.edu/bitstream/10125/60167/0727.pdf). Their paper and their willingness to engage with me to discuss it was what led to the (hopefully) improved Javadoc for the ESAPI Encoder interface.
    And lastly a special thanks to first-time contributors Michael-Ziluck, wiiitek, and HJW8472.

Thanks you all for your time and effort to ESAPI and making it a better project. And if I've missed any, my apologies; let me know and I will correct it.

A special thanks to the ESAPI community from the ESAPI project co-leaders:
    Kevin W. Wall (kwwall) <== The irresponsible party for these release notes!
    Matt Seil (xeno6696)