Commit Graph

97 Commits

Author SHA1 Message Date
Alex Waygood 401d7a7f00
gh-102515: Remove unused imports in the `Lib/` directory (#102516) 2023-03-08 11:45:38 +00:00
Gregory P. Smith 0b13575e74
gh-99108: Refactor _sha256 & _sha512 into _sha2. (#101924)
This merges their code. They're backed by the same single HACL* static library, having them be a single module simplifies maintenance.

This should unbreak the wasm enscripten builds that currently fail due to linking in --whole-archive mode and the HACL* library appearing twice.

Long unnoticed error fixed: _sha512.SHA384Type was doubly assigned and was actually SHA512Type. Nobody depends on those internal names.

Also rename LIBHACL_ make vars to LIBHACL_SHA2_ in preperation for other future HACL things.
2023-02-15 22:08:20 -08:00
Jonathan Protzenko 1fcc0efdaa
gh-99108: Replace SHA2-224 & 256 with verified code from HACL* (#99109)
replacing hashlib primitives (for the non-OpenSSL case) with verified implementations from HACL*. This is the first PR in the series, and focuses specifically on SHA2-256 and SHA2-224.

This PR imports Hacl_Streaming_SHA2 into the Python tree. This is the HACL* implementation of SHA2, which combines a core implementation of SHA2 along with a layer of buffer management that allows updating the digest with any number of bytes. This supersedes the previous implementation in the tree.

@franziskuskiefer was kind enough to benchmark the changes: in addition to being verified (thus providing significant safety and security improvements), this implementation also provides a sizeable performance boost!

```
---------------------------------------------------------------
Benchmark                     Time             CPU   Iterations
---------------------------------------------------------------
Sha2_256_Streaming            3163 ns      3160 ns       219353     // this PR
LibTomCrypt_Sha2_256          5057 ns      5056 ns       136234     // library used by Python currently
``` 

The changes in this PR are as follows:
- import the subset of HACL* that covers SHA2-256/224 into `Modules/_hacl`
- rewire sha256module.c to use the HACL* implementation

Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
2023-02-06 18:11:01 -08:00
Victor Stinner 71d5299b73
gh-94199: Remove hashlib.pbkdf2_hmac() Python implementation (GH-94200)
Remove the pure Python implementation of hashlib.pbkdf2_hmac(),
deprecated in Python 3.10. Python 3.10 and newer requires OpenSSL
1.1.1 or newer (PEP 644), this OpenSSL version provides a C
implementation of pbkdf2_hmac() which is faster.
2022-06-28 11:51:13 +02:00
Victor Stinner 5185956527
gh-69443: Add test.support.Py_DEBUG constant (#93226) 2022-05-26 00:12:54 +02:00
Christian Heimes 2b16a08bc7
bpo-40280: Detect missing threading on WASM platforms (GH-32352)
Co-authored-by: Brett Cannon <brett@python.org>
2022-04-07 09:22:47 +02:00
Christian Heimes 48e2010d92
bpo-47101: list only activated algorithms in hashlib.algorithms_available (GH-32076) 2022-03-23 13:30:05 -07:00
Christian Heimes e03db6d5be
bpo-45150: Fix testing under FIPS mode (GH-32046) 2022-03-22 16:40:43 +01:00
Christian Heimes 4f97d64c83
bpo-45150: Add hashlib.file_digest() for efficient file hashing (GH-31930) 2022-03-22 02:37:00 -07:00
Victor Stinner 6d0d7d2b8c
bpo-46913: test_hashlib skips _sha3 tests on UBSan (GH-31673)
If Python is built with UBSan, test_hashlib skips tests on the _sha3
extension which currently has undefined behaviors.

This change allows to run test_hashlib to check for new UBSan regression,
but the known _sha3 undefined behavior must be fixed.
2022-03-04 00:41:34 +01:00
Christian Heimes 443b308fee
bpo-40479: Fix hashlib's usedforsecurity for OpenSSL 3.0.0 (GH-30455) 2022-01-13 09:46:38 +01:00
stratakis a46c220edc
bpo-44048: Fix two hashlib test cases under FIPS mode (GH-26470)
test_disallow_instantiation and test_readonly_types try to test all the available
digests, however under FIPS mode, while the algorithms are available, trying to use
them will fail with a ValueError.
2021-06-04 17:47:59 +01:00
Erlend Egeberg Aasland fbff5387c3
bpo-43988: Use check disallow instantiation helper (GH-26392) 2021-05-27 08:43:52 +02:00
Christian Heimes 91554e4c5c
bpo-43908: Mark ssl, hash, and hmac types as immutable (GH-25792)
Signed-off-by: Christian Heimes <christian@python.org>
2021-05-02 09:47:45 +02:00
Christian Heimes ddbef71a2c
bpo-43916: Rewrite new hashlib tests, fix typo (GH-25791)
* bpo-43916: Rewrite new hashlib tests, fix typo
* Flag test as cpython only
2021-05-01 22:42:36 +02:00
Victor Stinner 665c7746fc
bpo-43916: _md5.md5 uses Py_TPFLAGS_DISALLOW_INSTANTIATION (GH-25753)
The following types use Py_TPFLAGS_DISALLOW_INSTANTIATION flag:

* _md5.md5
* _sha1.sha1
* _sha256.sha224
* _sha256.sha256
* _sha512.sha384
* _sha512.sha512
2021-04-30 18:40:30 +02:00
Victor Stinner 0cad068ec1
bpo-43916: Remove _disabled_new() function (GH-25745)
posix and _hashlib use the new Py_TPFLAGS_DISALLOW_INSTANTIATION
flag on their heap types, rather than using a custom tp_new function
(_disabled_new).
2021-04-30 14:06:49 +02:00
Christian Heimes 2875c603b2
bpo-43880: Show DeprecationWarnings for deprecated ssl module features (GH-25455)
* ssl.OP_NO_SSLv2
* ssl.OP_NO_SSLv3
* ssl.OP_NO_TLSv1
* ssl.OP_NO_TLSv1_1
* ssl.OP_NO_TLSv1_2
* ssl.OP_NO_TLSv1_3
* ssl.PROTOCOL_SSLv2
* ssl.PROTOCOL_SSLv3
* ssl.PROTOCOL_SSLv23 (alias for PROTOCOL_TLS)
* ssl.PROTOCOL_TLS
* ssl.PROTOCOL_TLSv1
* ssl.PROTOCOL_TLSv1_1
* ssl.PROTOCOL_TLSv1_2
* ssl.TLSVersion.SSLv3
* ssl.TLSVersion.TLSv1
* ssl.TLSVersion.TLSv1_1
* ssl.wrap_socket()
* ssl.RAND_pseudo_bytes()
* ssl.RAND_egd() (already removed since it's not supported by OpenSSL 1.1.1)
* ssl.SSLContext() without a protocol argument
* ssl.match_hostname()
* hashlib.pbkdf2_hmac() (pure Python implementation, fast OpenSSL
  function will stay)

Signed-off-by: Christian Heimes <christian@python.org>
2021-04-19 07:27:10 +02:00
stratakis 685719871a
bpo-37630: Do not skip the sha3 tests in case of missing builtin sha3 module (GH-20986)
Since we can use now the sha3 and shake algorithms from OpenSSL, we shouldn't
skip the tests in case the builtin algorithm was not compiled or removed.
2021-04-17 14:27:07 -07:00
Inada Naoki 3caea9adda
bpo-43651: PEP 597: Fix EncodingWarning in some tests (GH-25171)
* Fix test_float
* Fix _osx_support
* Fix test_fstring
* Fix test_gc
* Fix test_gzip
* Fix test_hashlib
* Fix unrelated whitespace issue

Co-authored-by: Ned Deily <nad@python.org>
2021-04-04 17:01:10 +09:00
Christian Heimes 975022b77b
bpo-40637: Don't test builtin PBKDF2 without builtin hashes (GH-20980)
Skip testing of pure Python PBKDF2 when one or more builtin hash module
is not available. Otherwise the import of hashlib prints noise on
stderr.

Signed-off-by: Christian Heimes <christian@python.org>
2020-11-17 15:40:35 +01:00
Hai Shi deb016224c
bpo-40275: Use new test.support helper submodules in tests (GH-21317) 2020-07-06 14:29:49 +02:00
stratakis 236a0f5cf0
bpo-40637: Do not emit warnings for disabled builtin hashes (GH-20937)
test_hashlib emits some warnings when it cannot find some hashes
as it assumes they failed to compile. Since we can disable hashes
through configure, we emit the warnings only in the case that we
did not intentionaly disable them.

Automerge-Triggered-By: @tiran
2020-06-17 10:26:08 -07:00
Hai Shi e80697d687
bpo-40275: Adding threading_helper submodule in test.support (GH-20263) 2020-05-28 00:10:27 +02:00
Christian Heimes be63019ed7
bpo-40637: Fix test_pbkdf2_hmac_py for missing sha1 (#20422) 2020-05-26 12:26:29 +02:00
Christian Heimes 909b5714e1
bpo-9216: hashlib usedforsecurity fixes (GH-20258)
func:`hashlib.new` passed ``usedforsecurity`` to OpenSSL EVP constructor
``_hashlib.new()``. test_hashlib and test_smtplib handle strict security
policy better.

Signed-off-by: Christian Heimes <christian@python.org>

Automerge-Triggered-By: @tiran
2020-05-22 11:04:33 -07:00
Christian Heimes d5b3f6b7f9
bpo-37630: Use SHA3 and SHAKE XOF from OpenSSL (GH-16049)
OpenSSL 1.1.1 comes with SHA3 and SHAKE builtin.

Signed-off-by: Christian Heimes <christian@python.org>

Automerge-Triggered-By: @tiran
2020-05-16 13:27:06 -07:00
Victor Stinner b1e11c31c5
bpo-40443: Remove unused imports in tests (GH-19804) 2020-04-30 02:21:30 +02:00
Victor Stinner e3dfb9b967
bpo-9216: Expose OpenSSL FIPS_mode() as _hashlib.get_fips_mode() (GH-19703)
test.pythoninfo logs OpenSSL FIPS_mode() and Linux
/proc/sys/crypto/fips_enabled in a new "fips" section.

Co-Authored-By: Petr Viktorin <encukou@gmail.com>
2020-04-29 09:04:22 -07:00
Hai Shi 66abe98a81
bpo-40275: Move requires_hashdigest() to test.support.hashlib_helper (GH-19716)
Add a new test.support.hashlib_helper submodule.
2020-04-29 03:11:29 +02:00
Christian Heimes 9055815809 bpo-38270: More fixes for strict crypto policy (GH-16418)
test_hmac and test_hashlib test built-in hashing implementations and
OpenSSL-based hashing implementations. Add more checks to skip OpenSSL
implementations when a strict crypto policy is active.

Use EVP_DigestInit_ex() instead of EVP_DigestInit() to initialize the
EVP context. The EVP_DigestInit() function clears alls flags and breaks
usedforsecurity flag again.

Signed-off-by: Christian Heimes <christian@python.org>



https://bugs.python.org/issue38270
2019-09-27 06:03:53 -07:00
Christian Heimes df69e75edc
bpo-38142: Updated _hashopenssl.c to be PEP 384 compliant (#16071)
* Updated _hashopenssl.c to be PEP 384 compliant
* Remove refleak test from test_hashlib. The updated type no longer accepts random arguments to __init__.
2019-09-25 23:03:30 +02:00
Christian Heimes 995b5d38e7 bpo-38153: Normalize hashlib algorithm names (GH-16083)
Signed-off-by: Christian Heimes <christian@python.org>
2019-09-13 14:31:19 +01:00
Christian Heimes 7cad53e6b0 bpo-9216: Add usedforsecurity to hashlib constructors (GH-16044)
The usedforsecurity keyword only argument added to the hash constructors is useful for FIPS builds and similar restrictive environment with non-technical requirements that legacy algorithms be forbidden by their implementations without being explicitly annotated as not being used for any security related purposes.  Linux distros with FIPS support benefit from this being standard rather than making up their own way(s) to do it.

Contributed and Signed-off-by: Christian Heimes christian@python.org
2019-09-12 19:30:00 -05:00
Christian Heimes 5a4f82f457 bpo-38132: Simplify _hashopenssl code (GH-16023)
Signed-off-by: Christian Heimes <christian@python.org>
2019-09-12 13:42:07 +01:00
Serhiy Storchaka 9b8c2e7676
bpo-34922: Fix integer overflow in the digest() and hexdigest() methods (GH-9751)
for the SHAKE algorithm in the hashlib module.
2018-10-11 07:41:00 +03:00
Serhiy Storchaka f1d36d8efa
bpo-33729: Fix issues with arguments parsing in hashlib. (GH-8346)
* help(hashlib) didn't work because of incorrect module name in blake2b and
  blake2s classes.
* Constructors blake2*(), sha3_*(), shake_*() and keccak_*() incorrectly
  accepted keyword argument "string" for binary data, but documented as
  accepting the "data" keyword argument. Now this parameter is positional-only.
* Keyword-only parameters in blake2b() and blake2s() were not documented as
  keyword-only.
* Default value for some parameters of blake2b() and blake2s() was None,
  which is not acceptable value.
* The length argument for shake_*.digest() was wrapped out to 32 bits.
* The argument for shake_128.digest() and shake_128.hexdigest() was not
  positional-only as intended.
* TypeError messages for incorrect arguments in all constructors sha3_*(),
  shake_*() and keccak_*() incorrectly referred to sha3_224.

Also made the following enhancements:

* More accurately specified input and result types for strings, bytes and
  bytes-like objects.
* Unified positional parameter names for update() and constructors.
* Improved formatting.
2018-07-31 09:50:16 +03:00
Serhiy Storchaka 7cb7bcff20
bpo-20260: Implement non-bitwise unsigned int converters for Argument Clinic. (GH-8434) 2018-07-26 13:22:16 +03:00
INADA Naoki f0bc645dfe
bpo-31787: Skip refleak check when _hashlib is not available (GH-5660) 2018-02-13 23:41:19 +09:00
Oren Milman d019bc8319 bpo-31787: Prevent refleaks when calling __init__() more than once (GH-3995) 2018-02-13 19:28:33 +09:00
Benjamin Peterson 95441809ef
closes bpo-32721: do not fail test_hashlib if _md5 isn't available (GH-5441) 2018-01-29 22:14:17 -08:00
Benjamin Peterson 77526f05fa replace dynamic import with 'exec' with importlib.import_module (#5433) 2018-01-29 18:03:01 -08:00
Jack O'Connor dcfb0e3c04 bpo-31933: fix blake2 multi-byte params on big endian platforms (#4250)
All Blake2 params have to be encoded in little-endian byte order. For
the two multi-byte integer params, leaf_length and node_offset, that
means that assigning a native-endian integer to them appears to work on
little-endian platforms, but gives the wrong result on big-endian. The
current libb2 API doesn't make that very clear, and @sneves is working
on new API functions in the GH issue above. In the meantime, we can work
around the problem by explicitly assigning little-endian values to the
parameter block.

See https://github.com/BLAKE2/libb2/issues/12.
2017-11-03 20:02:41 +01:00
Victor Stinner 8dcf22f442 bpo-31234: Join threads in test_hashlib (#3573)
* bpo-31234: Join threads in test_hashlib

Use thread.join() to wait until the parallel hash tasks complete
rather than using events. Calling thread.join() prevent "dangling
thread" warnings.

* test_hashlib: minor PEP 8 coding style fixes
2017-09-14 08:43:22 -07:00
Antoine Pitrou a6a4dc816d bpo-31370: Remove support for threads-less builds (#3385)
* Remove Setup.config
* Always define WITH_THREAD for compatibility.
2017-09-07 18:56:24 +02:00
Christian Heimes e370409cb1 Add test cases for internal SHA3 helpers 2016-09-23 11:32:30 +02:00
Christian Heimes 81188246fa Issue 26798: fetch OSError and HTTPException like other tests that use open_urlresource. 2016-09-08 10:53:40 +02:00
Christian Heimes 6fe2a75b64 Issue #16113: Add SHA-3 and SHAKE support to hashlib module. 2016-09-07 11:58:24 +02:00
Christian Heimes 59a0464af4 Download hashlib test vectors from pythontest.net 2016-09-07 01:21:14 +02:00
Christian Heimes 121b9487d1 Issue #26798: Add BLAKE2 (blake2b and blake2s) to hashlib. 2016-09-06 22:03:25 +02:00