Status: protocol published; results pending
This page does not contain product scores, winners, rankings, or benchmark conclusions. No completed product runs have been published. The protocol and test-case files are public now so the method can be challenged before results are collected.
Current data status
| Item | Status on 24 July 2026 |
|---|---|
| Protocol version | 0.1.0 published |
| Test-case definitions | Published for review |
| Product/version roster | Not frozen |
| Completed product runs | None published |
| Raw result files | Not available |
| Screenshots and logs | Not available |
| Scores and rankings | Not calculated or published |
Download the protocol
These files define the method and cases only. They contain no product outcomes.
Ownership and conflict disclosure
PLC Programming IO promotes PLC Simulation Software and may benefit commercially when readers visit that product. The product therefore cannot receive private criteria, different weights, or undocumented retests. Its raw cases, failures, limitations, version, tier, and evidence must be published under the same rules as every other product.
Products and version freeze
A product roster is not yet frozen. Before testing starts, the roster must record each product's exact version or build, operating environment, licence tier, price check date, and the reason it was included. A product may be excluded only under a rule published before its outcome is known.
Browser and desktop tools are not assumed to have identical scope. Unsupported features are recorded as unsupported; they are not silently replaced with a different test that favours the product.
Core test cases
Protocol version 0.1.0 defines ten core cases:
- Launch or install and reach a runnable project.
- Build a stop-priority motor seal-in circuit.
- Verify on-delay timer boundary behaviour.
- Verify counter increment and reset behaviour.
- Verify a one-scan rising-edge event.
- Confirm deterministic top-to-bottom scan ordering.
- Observe and document invalid arithmetic handling.
- Save, close, reopen, and rerun a project.
- Export or share a project and document portability limits.
- Locate versioned support and instruction-limit documentation.
Exact procedures, pass conditions, and evidence fields are in the downloadable JSON and CSV files.
Scoring framework
Execution correctness
30%Boolean logic, timers, counters, edges, scan behaviour, and fault handling.
Reproducibility and diagnostics
20%Repeatable outcomes, observable state, error messages, traces, and debugging support.
Dialect and instruction coverage
20%Declared dialects and instructions confirmed against exact versions and tiers.
Project portability
10%Save/reopen integrity, export availability, and documented interchange limitations.
Access and time to first run
10%Prerequisites, installation or browser launch, account friction, and first working program.
Documentation and limitations
10%Versioned help, supported-feature clarity, licensing clarity, and visible limitations.
Weights are frozen before outcome data is collected. A score remains unpublished until raw case outcomes and evidence are available.
Run and publication rules
- Each execution case is run three times from a documented starting state; inconsistent outcomes remain visible.
- A failed first run is not discarded. Setup mistakes are recorded and classified separately from product defects.
- Vendors may identify factual errors in the environment or procedure, but cannot approve the conclusion before publication.
- Retests use a published reason and preserve the original result.
- Result rows include pass, fail, unsupported, blocked, and not tested; missing evidence is never treated as a pass.
- Public screenshots must avoid personal data, licence keys, and confidential project information.
What the benchmark will not prove
These tests cannot establish suitability for a real machine, validate functional safety, certify vendor compatibility beyond the tested version, or replace a hardware commissioning test. Results will describe the recorded environment and task—not every use case.
Challenge the protocol before testing
Send proposed edge cases, pass-rule objections, or conflict concerns to expert@plcprogramming.io. Corrections are handled under the corrections policy.