test(dynseal): run the security scenarios, and fix the two they were hiding #12
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/dynseal-tests-run"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Stage 5 of reviving dynseal. Independent of spel#901 — these need no database.
18 passed, 1 xfailed, from zero.
The scenarios were never collected, and three were failing
dynseal/test_security_scenarios.pycollected zero tests: the class isSecurityScenarioTester, notTest*, so pytest walked past all eight scenarios.Running them found three failing — and two are real defects in the modules under
test, not in the test.
The demo manifest was signed with the one value the demo verifier rejects
minimal_boot_verifier._verify_signature_minimalreturnsany(b != 0 for b in signature[:8])— "not all zeros"._read_yubikey_slothanded itb'\x00' * 64.The two halves of the same demo contradicted each other, so
verify_from_yubikeycould never return
Trueandtest_yubikey_bootcould never pass.A device could only be compliant if it was jailbroken
enterprise_adapter._check_device_compliancebuilt a requirements dict and tookall(requirements.values())over it — including'jailbreak_detected': False.Falseis the good state for a problem flag, soall()read "no jailbreak" asa failed requirement and returned
NON_COMPLIANTfor every healthy device. Only ajailbroken device could have passed.
Both were invisible for exactly as long as nothing collected the file.
The third failure is left failing, on purpose
test_privacy_preservationis a strict xfail with its reasoning on the marker.It expects an untrusted source's content to be cryptographically verified but not
amplified — the module's thesis — and both halves are currently inverted.
It is not obviously a bug: the fixture signs with the literal string
test_sig,so verification failing may well be correct, and whether unverified content
should be shown is a design question about
attention_sovereigntyrather thansomething with an obvious repair. I would rather hand you the question than guess
at the answer.
strict=Truemeans it fails if the behaviour changes in either direction, soit cannot rot into a silent pass. Verified by mutation — making the scenario pass
turns the run red.
Shape
A wrapper class rather than renaming
SecurityScenarioTester: the scenariomethods return their result dicts, and pytest 8 makes a returning test an
error, so a bare rename would collect eight tests and fail all eight. The file
also still works as a script, which prints a trust-level matrix no pytest run
will.
pytest.iniis this repo's first pytest configuration — it had none at allwhile declaring
pytest>=7.4as a dependency.testpathsis the green set, so abare
pytestlocally runs exactly what CI runs.The CI comment was half wrong
Measured, not recalled:
test_security_scenarios.pycollects zerotest_schema_sync.pycannot importschema_validatorfrom the repo roottest_api.pyholds real tests"I had repeated the first of those to you as an overstatement earlier, on a
subagent's reading. Testing it directly showed the comment was right and I was
wrong; this corrects only the parts that are actually false.
Testing
pytestfrom the repo root: 18 passed, 1 xfailed.docs/test_schema_sync.py: 11 passed — a suite CI has never run.docs/test_api.pymeasured with nothing listening: 1 failed, 2 passed, 29errors — hence its exclusion, with that reason written beside it.
Impact Statement
Breaking changes: none to any interface. Two behaviour fixes inside dynseal's
demo paths, both making the code do what its own tests and comments already said
it did.
Risk.
enterprise_adapter's compliance change is the one to look at: it flips_check_device_compliancefrom always-NON_COMPLIANT to compliant-when-healthy.Nothing in production calls it — dynseal still has no caller — so the blast radius
today is the test suite. That is also why it is worth fixing now rather than after
something depends on the wrong answer.
CI cost: one
pip installand a sub-second pytest run.dynseal/test_security_scenarios.py collected ZERO tests — the class is SecurityScenarioTester, not Test*, so pytest walked past all eight scenarios. Running them found three failing, and two of those are real defects in the modules under test rather than in the test. THE DEMO MANIFEST WAS SIGNED WITH THE ONE VALUE THE DEMO VERIFIER REJECTS. minimal_boot_verifier._verify_signature_minimal returns `any(b != 0 for b in signature[:8])` — "not all zeros". _read_yubikey_slot handed it `b'\x00' * 64`. The two halves of the same demo contradicted each other, so verify_from_yubikey could never return True and test_yubikey_boot could never pass. The demo signature is now non-zero. A DEVICE COULD ONLY BE COMPLIANT IF IT WAS JAILBROKEN. enterprise_adapter._check_device_compliance built a requirements dict and took all(requirements.values()) over it — including 'jailbreak_detected': False. False is the GOOD state for a problem flag, so all() read "no jailbreak" as a failed requirement and returned NON_COMPLIANT for every healthy device. Only a jailbroken one could have passed. Problem flags are now evaluated on their own polarity. Both were invisible for as long as nothing collected the file. The third failure is left failing, as a strict xfail with its reasoning on the marker. test_privacy_preservation expects an untrusted source's content to be cryptographically verified but not amplified — the module's thesis — and both halves are currently inverted. It is not obviously a bug: the fixture signs with the literal string 'test_sig', so verification failing may well be correct, and whether unverified content should be shown is a design question about attention_sovereignty. strict=True means it fails if the behaviour changes in EITHER direction, so it cannot rot into a silent pass. Verified by mutation: making the scenario pass turns the run red. A wrapper class rather than renaming SecurityScenarioTester, because the scenario methods RETURN their result dicts and pytest 8 makes a returning test an error. The file also still works as a script, which prints a trust-level matrix no pytest run will. pytest.ini is the repo's first pytest configuration — it had none at all while declaring pytest>=7.4 as a dependency. testpaths is the green set, so a bare `pytest` locally runs exactly what CI runs. docs/test_api.py and docs/test_prism.py are excluded there, each with its reason beside it. AND THE CI COMMENT, which was half wrong. Measured, not recalled: · duplicate basename aborts collection — RIGHT. · test_security_scenarios collects zero — RIGHT. · test_schema_sync "cannot import schema_validator from the repo root" — WRONG. It collects and passes 11 tests from the repo root. · "only test_api.py holds real tests" — WRONG, same reason. · "23 of its 32 error without a live API" — 29, not 23. 18 passed, 1 xfailed, from zero.