Port DX7 import/export tools to OPSL 0.3 device JSON #1

Merged

peterswimm merged 6 commits from feat/opsl-0.3-device into master

2026-10-04 22:51:09 +00:00

peterswimm commented

2026-10-04 22:48:15 +00:00

Owner

Scope

Tools/dx7-import/ (importer, exporter, reclassify, __main__) imported
a Python package, opsl (from opsl import parse, serialize, from opsl.model import OPSLPreset), that has never existed anywhere this repo
can reach — and its own files cross-imported each other as from dx7.reader import ... even though no dx7 package exists either (this
directory is literally named dx7-import, not a valid Python
identifier). Both meant the tool could not run at all.

This PR:

Adds opsl_device.py, a small, stdlib-only module that builds/reads/
writes OPSL 0.3 "device" documents directly, replacing the dependency
on the nonexistent opsl package. It mirrors opsl-data's
scripts/opsl01_to_device.py (the converter that built the published
dx7-corpus-v1 corpus from this tool's old OPSL 0.1 output) so
chavo's own output matches that corpus's shape: device_id: "yamaha/dx7", role: "patch", kind: "hardware", protocol: "sysex_bulk", and the same 145-entry DX7 params table.
Switches importer.py / exporter.py / reclassify.py /
__main__.py to it, and to plain same-directory imports instead of
the never-working dx7.* path. Globs *.device.json instead of
*.opsl.
Vendors the OPSL 0.3 device schema (schema/device.schema.json,
pinned to opsl-public tag v0.3.1, byte-identical — see
schema/VERSION.md) so tests can validate real output against it.
Fixes two DX7 packed-voice codec bugs found while proving a real SysEx
file round-trips byte-identically (both documented in detail in
opsl_device.py's module docstring):
1. Operator detune was packed into its own 6-byte block instead of
  being merged into the rate-scale byte (bits 3-6), which also pushed
  every later global-param offset 6 bytes too far.
2. Keyboard level-scaling L/R depth was masked to 6 bits (max 63)
  instead of the real 7-bit (0-99) range — harmless for opsl-data's
  corpus (no preset there happened to use >63) but wrong for at least
  one real factory bank used as a test fixture here (two voices use
  96 and 99).
Fixes the matching bug in reader.py's single-voice (163-byte) SysEx
parsing path, and a latent downstream consequence in __main__.py
(the classifier's algorithm-extraction byte offset was hardcoded to
the old, wrong global-params offset).
Adds a GPL-3.0 LICENSE (chavo had none; Dexed's own source files are
already GPL-3.0-or-later per their headers).
Untracks Tools/dx7-import/__pycache__ (accidentally committed in the
initial commit) and adds a .gitignore.

Did not touch: the C++/JUCE plugin source (Source/) — nothing in
this port required it.

Testing

30 unittest cases, runnable with:

python3 -m unittest discover -s Tools/dx7-import -p "test_*.py"

All 30 pass locally. Summary by file:

tests/test_opsl_device.py — packed-voice codec round-trips
byte-identically for every voice in a real DX7 factory SysEx bank
(tests/fixtures/brass.syx, a standard brass/horn bulk dump); the
145-entry params table's shape; and (skipped if jsonschema isn't
installed, otherwise exercised for real) JSON Schema validation of
built documents against the vendored schema.
tests/test_round_trip.py — the core acceptance claim: .syx ->
device documents -> .syx reproduces the original 4104-byte file
byte-for-byte (test_full_bank_without_dedup_is_byte_identical_to_source_file).
tests/test_cli.py — drives the actual __main__.py CLI dispatcher
(import then export) against a temp directory.
tests/test_corpus_sample.py — compares a chavo-built document
against a real document from the published dx7-corpus-v1 corpus,
asserting where they match (document shape, params table) and where
they're expected to differ (param values — different source patches
sharing a factory name; and the one param-domain divergence from the
depth-mask fix above).

Per the "prove the test can fail" requirement, several tests include an
explicit negative control before trusting the positive case: corrupting
one byte of a real voice and confirming the round-trip comparison now
(correctly) fails; corrupting a built document and confirming schema
validation now (correctly) rejects it.

Also manually smoke-tested reclassify.py (not covered by a dedicated
test file) against a real imported+categorized preset tree: KNN
dry-run and --move both work, and moved files now get their
category:<slug> tag rewritten to match their new folder (previously
this tool left the old, now-wrong tag in place after a move — a small
latent inconsistency bug fixed as part of this port).

Impact Statement

Existing features: DX7 SysEx import/export/reclassify go from
"cannot run" (missing opsl package) to working end-to-end. No
existing working behavior changes, since none of it previously ran.
Public APIs / contracts: importer.voice_to_preset(packed, name, source) -> OPSLPreset becomes importer.voice_to_device(packed, name, source) -> dict (an OPSL 0.3 document). exporter/reclassify
function signatures now take/return plain dicts instead of
OPSLPreset objects. None of this was callable before (the import
that defined OPSLPreset didn't exist), so there is no live caller to
break.
Schemas: output moves from ad hoc OPSL 0.1 text
(preset { schema: "0.1" ... }) to OPSL 0.3 JSON device documents
validated against the vendored schema. opsl_device.py's params table
has a wider domain (0-127, not 0-63) for
kbd-lev-scl-{l,r}-depth than opsl-data's published corpus uses for
the same two keys — a deliberate, documented fix (see above); anything
reading those two domains across both sources should not assume they
match.
Data / artifacts: output file extension changes from *.opsl to
*.device.json. Nothing in the repo currently consumes the old
*.opsl output (no such files exist in chavo; the only consumer was
opsl-data's separate converter, which is a different repo and not
changed here).
Integrations: none — this tool has no other integration points in
this repo (Source/ C++ plugin code, untouched, does not call it).
Mappings: the op{N}.* / global DX7 param naming convention is
unchanged, so any external document keyed by those names still lines
up with opsl_device.dx7_params_table()'s keys.
Backward compatibility: not applicable — the tool was non-functional
before this change (the referenced opsl package does not exist), so
there is no working prior behavior to remain compatible with.

Docs

Tools/dx7-import/README.md (new): how to run the tool and its tests,
and the format change.
Tools/dx7-import/opsl_device.py: extensive module docstring covering
the document shape, the two codec bug fixes, and why fresh imports
carry lineage/x-dx7import instead of opsl-data's x-opsl01
carry-over block.
Tools/dx7-import/schema/VERSION.md (new): schema provenance/pinning.

Open decisions for review

Depth-mask domain divergence from opsl-data. opsl_device.py
uses a 0-127 domain for kbd-lev-scl-{l,r}-depth; the published
dx7-corpus-v1 corpus uses 0-63 for the same two keys (opsl-data's
scripts/opsl01_to_device.py is not changed in this PR — that's a
separate repo). Flagging in case opsl-data's maintainers want the
same fix, or want to re-derive/re-publish the corpus with it.
Fresh .syx imports now carry a lineage + x-dx7import extension
block (voice sha256, 1-indexed algorithm, and the exact raw 10 name
bytes losslessly latin-1 encoded) instead of an x-opsl01 block,
since there's no OPSL 0.1 document in this pipeline to carry over.
Documented in opsl_device.py's module docstring; flagging here in
case a different shape is preferred for interop with other OPSL 0.3
producers.

## Scope `Tools/dx7-import/` (importer, exporter, reclassify, `__main__`) imported a Python package, `opsl` (`from opsl import parse, serialize`, `from opsl.model import OPSLPreset`), that has never existed anywhere this repo can reach — and its own files cross-imported each other as `from dx7.reader import ...` even though no `dx7` package exists either (this directory is literally named `dx7-import`, not a valid Python identifier). Both meant the tool could not run at all. This PR: - Adds `opsl_device.py`, a small, stdlib-only module that builds/reads/ writes OPSL 0.3 "device" documents directly, replacing the dependency on the nonexistent `opsl` package. It mirrors opsl-data's `scripts/opsl01_to_device.py` (the converter that built the published `dx7-corpus-v1` corpus from this tool's old OPSL 0.1 output) so chavo's own output matches that corpus's shape: `device_id: "yamaha/dx7"`, `role: "patch"`, `kind: "hardware"`, `protocol: "sysex_bulk"`, and the same 145-entry DX7 params table. - Switches `importer.py` / `exporter.py` / `reclassify.py` / `__main__.py` to it, and to plain same-directory imports instead of the never-working `dx7.*` path. Globs `*.device.json` instead of `*.opsl`. - Vendors the OPSL 0.3 device schema (`schema/device.schema.json`, pinned to `opsl-public` tag `v0.3.1`, byte-identical — see `schema/VERSION.md`) so tests can validate real output against it. - Fixes two DX7 packed-voice codec bugs found while proving a real SysEx file round-trips byte-identically (both documented in detail in `opsl_device.py`'s module docstring): 1. Operator detune was packed into its own 6-byte block instead of being merged into the rate-scale byte (bits 3-6), which also pushed every later global-param offset 6 bytes too far. 2. Keyboard level-scaling L/R depth was masked to 6 bits (max 63) instead of the real 7-bit (0-99) range — harmless for opsl-data's corpus (no preset there happened to use >63) but wrong for at least one real factory bank used as a test fixture here (two voices use 96 and 99). - Fixes the matching bug in `reader.py`'s single-voice (163-byte) SysEx parsing path, and a latent downstream consequence in `__main__.py` (the classifier's algorithm-extraction byte offset was hardcoded to the old, wrong global-params offset). - Adds a GPL-3.0 `LICENSE` (chavo had none; Dexed's own source files are already GPL-3.0-or-later per their headers). - Untracks `Tools/dx7-import/__pycache__` (accidentally committed in the initial commit) and adds a `.gitignore`. **Did not touch:** the C++/JUCE plugin source (`Source/`) — nothing in this port required it. ## Testing 30 `unittest` cases, runnable with: ``` python3 -m unittest discover -s Tools/dx7-import -p "test_*.py" ``` All 30 pass locally. Summary by file: - `tests/test_opsl_device.py` — packed-voice codec round-trips byte-identically for every voice in a real DX7 factory SysEx bank (`tests/fixtures/brass.syx`, a standard brass/horn bulk dump); the 145-entry params table's shape; and (skipped if `jsonschema` isn't installed, otherwise exercised for real) JSON Schema validation of built documents against the vendored schema. - `tests/test_round_trip.py` — the core acceptance claim: `.syx` -> device documents -> `.syx` reproduces the original 4104-byte file byte-for-byte (`test_full_bank_without_dedup_is_byte_identical_to_source_file`). - `tests/test_cli.py` — drives the actual `__main__.py` CLI dispatcher (`import` then `export`) against a temp directory. - `tests/test_corpus_sample.py` — compares a chavo-built document against a real document from the published `dx7-corpus-v1` corpus, asserting where they match (document shape, params table) and where they're expected to differ (param values — different source patches sharing a factory name; and the one param-domain divergence from the depth-mask fix above). Per the "prove the test can fail" requirement, several tests include an explicit negative control before trusting the positive case: corrupting one byte of a real voice and confirming the round-trip comparison now (correctly) fails; corrupting a built document and confirming schema validation now (correctly) rejects it. Also manually smoke-tested `reclassify.py` (not covered by a dedicated test file) against a real imported+categorized preset tree: KNN dry-run and `--move` both work, and moved files now get their `category:<slug>` tag rewritten to match their new folder (previously this tool left the old, now-wrong tag in place after a move — a small latent inconsistency bug fixed as part of this port). ## Impact Statement - **Existing features:** DX7 SysEx import/export/reclassify go from "cannot run" (missing `opsl` package) to working end-to-end. No existing *working* behavior changes, since none of it previously ran. - **Public APIs / contracts:** `importer.voice_to_preset(packed, name, source) -> OPSLPreset` becomes `importer.voice_to_device(packed, name, source) -> dict` (an OPSL 0.3 document). `exporter`/`reclassify` function signatures now take/return plain dicts instead of `OPSLPreset` objects. None of this was callable before (the import that defined `OPSLPreset` didn't exist), so there is no live caller to break. - **Schemas:** output moves from ad hoc OPSL 0.1 text (`preset { schema: "0.1" ... }`) to OPSL 0.3 JSON device documents validated against the vendored schema. `opsl_device.py`'s params table has a wider domain (0-127, not 0-63) for `kbd-lev-scl-{l,r}-depth` than opsl-data's published corpus uses for the same two keys — a deliberate, documented fix (see above); anything reading those two domains across both sources should not assume they match. - **Data / artifacts:** output file extension changes from `*.opsl` to `*.device.json`. Nothing in the repo currently consumes the old `*.opsl` output (no such files exist in chavo; the only consumer was opsl-data's separate converter, which is a different repo and not changed here). - **Integrations:** none — this tool has no other integration points in this repo (Source/ C++ plugin code, untouched, does not call it). - **Mappings:** the `op{N}.*` / global DX7 param naming convention is unchanged, so any external document keyed by those names still lines up with `opsl_device.dx7_params_table()`'s keys. - **Backward compatibility:** not applicable — the tool was non-functional before this change (the referenced `opsl` package does not exist), so there is no working prior behavior to remain compatible with. ## Docs - `Tools/dx7-import/README.md` (new): how to run the tool and its tests, and the format change. - `Tools/dx7-import/opsl_device.py`: extensive module docstring covering the document shape, the two codec bug fixes, and why fresh imports carry `lineage`/`x-dx7import` instead of opsl-data's `x-opsl01` carry-over block. - `Tools/dx7-import/schema/VERSION.md` (new): schema provenance/pinning. ## Open decisions for review 1. **Depth-mask domain divergence from opsl-data.** `opsl_device.py` uses a 0-127 domain for `kbd-lev-scl-{l,r}-depth`; the published `dx7-corpus-v1` corpus uses 0-63 for the same two keys (opsl-data's `scripts/opsl01_to_device.py` is not changed in this PR — that's a separate repo). Flagging in case opsl-data's maintainers want the same fix, or want to re-derive/re-publish the corpus with it. 2. Fresh `.syx` imports now carry a `lineage` + `x-dx7import` extension block (voice sha256, 1-indexed algorithm, and the exact raw 10 name bytes losslessly latin-1 encoded) instead of an `x-opsl01` block, since there's no OPSL 0.1 document in this pipeline to carry over. Documented in `opsl_device.py`'s module docstring; flagging here in case a different shape is preferred for interop with other OPSL 0.3 producers.

peterswimm added 6 commits

2026-10-04 22:48:15 +00:00

Add GPL-3.0 LICENSE matching Dexed's license e7279ce973

Chavo is a Dexed fork; Dexed's source files are already GPL-3.0-or-later
(see license headers in Source/*.cpp), but the repo had no top-level
LICENSE file. Add the canonical GPLv3 text.

Stop tracking __pycache__ in Tools/dx7-import 419934a0ee

The initial commit accidentally committed compiled bytecode for this
tool. Add a .gitignore and untrack it; nothing in this repo should ship
.pyc files.

Vendor OPSL 0.3 device schema and add opsl_device.py 5ecc1b903f

Tools/dx7-import has imported a Python package, opsl (from opsl import
parse, serialize / from opsl.model import OPSLPreset), that has never
existed anywhere this repo can reach, so the DX7 import/export tools
could not run at all.

opsl_device.py replaces that dependency with a small, stdlib-only module
that builds/reads/writes OPSL 0.3 'device' documents directly, mirroring
opsl-data's scripts/opsl01_to_device.py (the converter that produced the
published dx7-corpus-v1 corpus from this tool's old 0.1 output) so
chavo's own output matches that corpus's shape.

Also fixes two DX7 packed-voice codec bugs found while proving a real
SysEx file round-trips byte-identically through the new module: operator
detune was packed as a separate 6-byte block instead of being merged
into the rate-scale byte, and the keyboard level-scaling L/R depth
fields were masked to 6 bits instead of the real 7-bit (0-99) range. See
the module's docstring for the full rationale, including the one
deliberate divergence this creates from opsl-data's published param
domains for those two depth fields.

schema/device.schema.json is vendored from opsl-public tag v0.3.1
(byte-identical, see schema/VERSION.md) so this tool's own tests can
validate real output against the real schema without a runtime
dependency on it.

Fix single-voice packed-byte layout; expose extract_name/clean_name 35b690169a

_repack_single (the 155-byte 'single voice' SysEx -> 128-byte packed
converter) had the same packing bug opsl_device.py's module docstring
describes for the main codec: it gave operator detune its own output
byte instead of merging it into the rate-scale byte, and masked
keyboard level-scaling L/R depth to 6 bits instead of 7. Both are fixed
here to match the corrected layout.

Also split _extract_name (private, required a full 128-byte packed
voice) into clean_name (works on just the 10 raw name bytes) and
extract_name (full packed voice, delegates to clean_name), and makes
both public. opsl_device.py needs the former; keeping only the
full-packed-voice version invites exactly the silent-empty-string bug
that a first draft of opsl_device.py actually had (caught by
tests/test_opsl_device.py).

Switch importer/exporter/reclassify/__main__ to OPSL 0.3 device docs 8fbd99fddb

Replaces the broken 'from opsl import ...' / 'from dx7.X import ...'
imports across all four entry points with opsl_device.py and plain
same-directory imports, and switches every file glob from *.opsl to
*.device.json:

- importer.py: voice_to_preset() -> voice_to_device(), now a thin
  wrapper around opsl_device.build_patch_device().
- exporter.py: reads OPSL 0.3 param_values instead of a preset's
  ParamValue list; export_presets_by_category() now groups by the
  'category:<slug>' state.meta.tags entry via opsl_device.category_of().
- reclassify.py: loads *.device.json param_values for its KNN feature
  vectors instead of parsing *.opsl text. Also fixes a pre-existing gap:
  moving a file between category folders now rewrites its category tag
  to match (opsl_device.with_category()), where it previously left the
  tag pointing at the old category.
- __main__.py: import/export subcommands operate on *.device.json.
  Also fixes a latent consequence of the reader.py packed-voice bug:
  the algorithm-extraction byte offset used for the classifier's
  fallback heuristic was hardcoded to the old (wrong) global-params
  offset; now delegates to opsl_device.unpack_voice().
- __init__.py: re-exports via plain imports; documents why this
  directory can never be imported as a package (its name, containing
  a hyphen, is not a valid Python identifier).

Add tests and README for the OPSL 0.3 dx7-import port 5f6254371d

30 unittest cases, runnable with:

    python3 -m unittest discover -s Tools/dx7-import -p "test_*.py"

- test_opsl_device.py: packed-voice codec unit tests against every voice
  in a real DX7 factory SysEx bank (tests/fixtures/brass.syx), the
  145-entry params table's shape, and (when jsonschema is installed)
  real JSON Schema validation of built documents.
- test_round_trip.py: the core acceptance claim -- a real .syx file
  converted to device documents and back to .syx reproduces the original
  file byte-for-byte.
- test_cli.py: drives the actual __main__.py import/export subcommands
  against a temp directory, proving the tool runs end to end (it could
  not run at all before this port).
- test_corpus_sample.py: compares a chavo-built document against a real
  document from the published dx7-corpus-v1 corpus (opsl-data), copied
  into tests/fixtures/corpus_brass_1.device.json, documenting where they
  match (structure, params table) and where they're expected to differ
  (param values, and the one param-domain divergence from opsl_device.py's
  depth-mask fix).

Several tests include a deliberate negative control (corrupt a byte or a
field, confirm the test now fails) before trusting the positive case,
per the task's "prove the test can fail" requirement.

tests/fixtures/brass.syx is a standard Yamaha DX7 factory-style SysEx
bulk dump (brass/horn patches); chosen because two of its voices use a
keyboard-level-scaling depth value above 63, which is exactly what
exposed the depth-mask bug opsl_device.py fixes.

README.md documents how to run the tool and the tests, and the document
format change (*.opsl -> *.device.json).

peterswimm merged commit 9654718acb into master

2026-10-04 22:51:09 +00:00

peterswimm deleted branch feat/opsl-0.3-device

2026-10-04 22:51:09 +00:00

peterswimm referenced this pull request from a commit

2026-10-04 22:51:11 +00:00

Merge pull request 'Port DX7 import/export tools to OPSL 0.3 device JSON' (#1) from feat/opsl-0.3-device into master