Alon, thanks for connecting me with Koby.
Meanwhile, i’ve made substantial progress but need to report other issues i’ve found. I’d be grateful if you could help…
BillRun Bug Reports
Environment: BillRun Cloud v5.25.0, self-hosted via microk8s on AWS EC2 (Ubuntu), PHP 7.4.33, MongoDB 7.0. billrun-app and billrun-worker run as separate Kubernetes pods.
Six issues below, each reproduced with concrete evidence (logs, source references, before/after data). Two (#1, #2) surfaced while testing prepaid realtime charging; the remainder (#3–#6) surfaced while building and testing a file-based (CSV/SFTP) input processor for backdated postpaid CDR testing.
1. Prepaid Plan creation wizard writes a malformed tax field, breaking realtime customer resolution (returnCode: -2)
Summary: Creating a plan via Prepaid → Plans → Create New Prepaid Plan (a wizard that only exposes a Title field) silently defaults the plan’s tax field to an object with blank placeholder strings, instead of the array shape used everywhere else. This appears to break realtime charging’s customer-resolution step for prepaid subscribers on that plan.
Evidence — structural diff between a working postpaid plan and a broken prepaid one:
| Field |
BASIC (postpaid, realtime charging works) |
PREPAID_BASIC (prepaid, -2 on every charge) |
tax |
[{"type":"vat","taxation":"global"}] (array) |
{"service_code":" ","product_code":" ","safe_harbor_override_pct":" "} (object, blank strings) |
connection_type |
"postpaid" |
"prepaid" |
| extra fields |
— |
"charging_type":"prepaid", "type":"customer" |
Steps to reproduce:
- Create a plan via the Prepaid Plan wizard (Title-only form).
- Create a prepaid subscriber on that plan, with valid buckets/balances.
- Send an allocation-based realtime Init request (
op:1) via a custom input processor.
- Observe
returnCode: -2 even though sid is correctly resolved in the response (ruling out a simple “subscriber not found”).
Fix applied: manually update the plan’s tax field to the array shape (db.plans.updateOne(..., {$set: {tax: [{"type":"vat","taxation":"global"}]}})). After this single-field fix, -2 no longer occurs.
Suggested fix: either have the Prepaid Plan wizard default tax to the same array shape as postpaid plans, or expose full tax configuration in that wizard instead of silently defaulting it.
2. Undocumented returnCode: -5 on prepaid allocation-based realtime charging (root cause not found)
Summary: After fixing #1, prepaid subscribers still fail allocation-based (Diameter Gy-style) realtime Init requests with returnCode: -5. The realtime API docs document 1 (success) through -4 (failed to price) — -5 is not documented anywhere we could find, including in Anthropic-searchable GitHub source.
Evidence:
- Comparing the
lines document for a successful postpaid allocation-based charge (returnCode: 1) against the failing prepaid one for the same processor type:
- Working line:
"aprice": 5, "tx": [], "lcount": 3, "usagev": 5242880
- Failing line:
"apr": 0 (note the different field name — apr vs aprice, confirmed as two genuinely distinct fields via grep across library/Billrun/), "tx": [4 populated hash values], "lcount": 4, "usagev": 0
- The populated
tx array (vs. empty on success) suggests failed internal retry/sub-transaction attempts.
- The
sessionId returned in the API response is byte-for-byte identical across multiple different failing requests against different subscribers — suggesting a static/fallback value on this particular error path rather than a freshly generated identifier.
- A separate, structurally different failure mode was also observed: sending a one-time-charge (postpaid-style,
request_type: 4) request against the same prepaid subscriber correctly computed a real price (granted_volume: 3, granted_cost: €0.10/€0.30 for voice/text) but still returned -5 at the final commit step — suggesting -5 may also serve as a generic “connection_type/request_type mismatch, cannot commit” guard, distinct from whatever’s happening in the allocation-based case above.
Request: documentation of what -5 means, or a pointer to the source file/class responsible for allocation-based prepaid charge commitment, so we can self-diagnose further.
3. Plan price field silently coerces to 0 when saved as an empty string, with no validation error
Summary: A plan’s price array element (e.g. {"price": "", "from": 0, "to": "UNLIMITED"}) with an empty-string price produces a genuine €0 charge on invoicing, with no error or warning surfaced anywhere in the UI or API response — only a PHP-log-level notice.
Evidence:
PHP Warning: A non-numeric value encountered in /billrun/library/Billrun/Plans/Charge/Upfront.php on line 64
This traces to 'value' => $price * $cycleData['fraction'] * $quantity where $price is the literal string "". PHP’s non-numeric-string coercion silently yields 0.
Impact: an invoice can be generated and appear entirely successful, with a plan’s recurring fee silently missing, with no signal to the operator that anything is wrong.
Suggested fix: validate price as numeric at save time (both API and UI), and reject the save (or require an explicit 0) rather than accepting an empty string that silently becomes 0 deep in the billing pipeline.
4. SFTP receiver key-based auth never works when the key is uploaded via the Input Processor wizard UI
Summary: Configuring a File-based Input Processor’s SFTP Receiver with a private key uploaded via the wizard’s Key file-chooser stores only a reference string in config — never the actual key content — and nothing in the codepath that actually uses it ever resolves that reference back into real key material, at least not across pods in a multi-pod deployment.
Evidence:
-
After uploading a key named billrun_sftp_key via the UI, the resulting connection config in config.file_types[].receiver.connections[] shows:
key_label: 'billrun_sftp_key',
key: 'CDR_Generic_1786375402' // a generated reference, not the key content
-
Billrun_Receiver_Ssh::receive() (library/Billrun/Receiver/Ssh.php) builds:
$auth = array('key' => $sharedDirectoryPath . $config['key']);
// where $sharedDirectoryPath = Billrun_Util::getBillRunSharedFolderPath('files/keys/input_processors/')
i.e. it expects a real file to exist at <shared_folder>/files/keys/input_processors/<reference>.
-
Running --receive --type <processor> against this config fails with:
PHP Notice: Undefined index: keytext in /billrun/library/Billrun/Ssh/Seclibgateway.php on line 312
Exception: NoKeyLoadedException: Unable to read key
This is because Seclibgateway::readRsaKey() correctly falls back to $auth['keytext'] when is_file($auth['key']) is false — but nothing anywhere in Billrun_Receiver_Ssh ever populates a keytext field from the connection config, so that fallback is unreachable dead code from this caller’s perspective.
-
Directly inspecting the billrun-worker pod’s filesystem confirmed no file exists at the expected shared-folder path — despite the UI reporting the upload as successful and the config correctly storing a reference to it.
Workaround applied: manually place the actual private key file at the exact expected path inside the worker pod (microk8s kubectl cp <key> billrun/<worker-pod>:/billrun/shared/container/files/keys/input_processors/<reference>). Connection then succeeds immediately.
Suggested fix: either ensure the app-pod’s upload handler writes the key to a location genuinely shared with (and readable by) the worker pod in multi-pod deployments, or document this shared-storage requirement explicitly, since the wizard gives no indication the upload is anything other than fully self-contained.
5. Queue-based calculator can leave entries permanently stuck (tracked upstream as BRCD-3852)
Summary: This is an already-acknowledged bug in BillRun’s own source (https://billrun.atlassian.net/browse/BRCD-3852), which we reproduced cleanly with a minimal single-line CSV import — offered here as additional field evidence/reproduction steps for that existing ticket.
Evidence — the acknowledgment in source:
// library/Billrun/Calculator.php, ~line 545
$foundLines = $queue->query(...)->cursor();
if ($foundLines->count() != $response['nModified']) {
// TODO remove when https://billrun.atlassian.net/browse/BRCD-3852 is fixed
Billrun_Factory::log('Found wrong number of lines: ' . $foundLines->count(), Zend_Log::DEBUG);
}
Our reproduction:
- Imported a single line via
--process.
- Ran
--calculate --type rate_Usage. Log showed "Found wrong number of lines: 2" and "Entity not found for row <our stamp>", while a different, older, unrelated queue entry in the same batch was processed successfully.
- The affected queue entry was left with a
hash and calc_time set (marking it as “claimed” by that run), but was never actually processed.
- Every subsequent
--calculate run of any type (customer, rate_Usage) then reported "Entities loaded: 0" for this entry specifically — it had become permanently invisible to the queue-selection query, evidently because the update-hash step and the subsequent read-back step raced against each other and left the entry in an inconsistent state.
- Manually clearing
hash/calc_time via direct DB update did not restore it to a processable state either — it remained excluded from all further selection queries.
Workaround: delete the affected lines/queue documents entirely and re-ingest a fresh copy of the source CDR. A second, never-touched import processed the full customer → rate → pricing → tax chain cleanly with zero manual intervention.
6. File-based CDR processing correctly derives urt from the CDR’s own timestamp, but the billrun cycle-assignment tag does not
Summary: When a CSV-ingested line is priced via the calcCpu inline pipeline (triggered automatically during --process), the resulting line’s urt/eurt fields correctly reflect the CDR’s own submitted historical timestamp — but the billrun field (which determines which invoice cycle the charge is aggregated into) appears to bind to processing time instead, producing an inconsistent result.
Evidence:
- CSV
timestamp column: 2026-07-08 12:00:00
- Resulting line:
urt: ISODate('2026-07-08T12:00:00.000Z'), eurt: ISODate('2026-07-08T12:00:00.000Z') — correct.
- Same processing run’s log:
"Create empty balance, from: 2026-07-01 to: 2026-08-01" — also correctly derived from urt (July’s balance period).
- But the same line’s final
billrun field: '202609' — the cycle key for August’s usage (closing in September), under BillRun’s own YYYYMM-as-close-date convention. The correct value for July’s usage is '202608'.
- Processing was run on 2026-08-10 —
'202609' is consistent with “the cycle currently open right now,” not with anything derived from the line’s own urt.
Impact: a line’s usage-window calculation (balance period) and its cycle-assignment tag can disagree with each other within the same processing run, meaning --aggregate --type customer --stamp <correct-cycle> will silently miss lines that were otherwise correctly processed and priced, unless the billrun field is corrected by hand first.
Suggested fix: derive billrun from urt using the same logic already used for the balance-period calculation in this same code path, rather than (apparently) from current processing time.
Related, possibly-connected minor notice observed in the same processing run, in case it’s relevant to whoever investigates this:
PHP Notice: Undefined variable: invoice_day in /billrun/library/Billrun/Billingcycle.php on line 196