CLN Application backend repeatedly crashes with invalid status code error after Umbrel reboot

[blockdyor]

Environment:

Device: Raspberry Pi Compute Module 5 Rev 1.0
umbrelOS: 1.4.2
Core Lightning: v25.05

Issue Description:

A few days back I mentioned my CLN Application had slowed to a crawl and was basically unusable because of #130. Today I rebooted Umbrel and got hit with two screens that keep switching back and forth while opening the Core Lightning App:

I can still connect to the node through Ride The Lightning just fine. Looks like the issue is specifically with the Core Lightning Application itself.

Logs:

core-lightning_lightningd_1  | Core-Lightning starting
core-lightning_lightningd_1  | 2025-11-01T17:20:43.235Z INFO    lightningd: v25.05
core-lightning_lightningd_1  | 2025-11-01T17:20:44.246Z UNUSUAL plugin-bookkeeper: topic 'utxo_deposit' is not a known notification topic
core-lightning_lightningd_1  | 2025-11-01T17:20:44.246Z UNUSUAL plugin-bookkeeper: topic 'utxo_spend' is not a known notification topic
core-lightning_lightningd_1  | 2025-11-01T17:20:48.779Z INFO    connectd: Static Tor service onion address: "[REDACTED].onion:9735,10.21.21.96:9735" bound from extern port 9735 
core-lightning_lightningd_1  | 2025-11-01T17:20:48.827Z INFO    plugin-bcli: bitcoin-cli initialized and connected to bitcoind.
core-lightning_lightningd_1  | lightningd --network="${LIGHTNINGD_NETWORK}" "$@"
core-lightning_lightningd_1  | 2025-11-01T17:20:51.618Z INFO    plugin-cln-grpc: Connecting to "lightning-rpc" and serving grpc on 10.21.21.96:2110
core-lightning_lightningd_1  | 2025-11-01T17:20:51.619Z INFO    plugin-clnrest: REST server running at https://10.21.21.96:2107
core-lightning_lightningd_1  | 2025-11-01T17:20:51.955Z INFO    plugin-sling: read startup options
core-lightning_lightningd_1  | 2025-11-01T17:20:52.003Z INFO    plugin-sling: Refreshed 0 liquidity beliefs in 0ms!
core-lightning_lightningd_1  | 2025-11-01T17:20:52.023Z INFO    plugin-chanbackup: Loaded 3 stored backups for peers
core-lightning_lightningd_1  | 2025-11-01T17:20:52.306Z INFO    plugin-sling: Read 0 askerene liquidity constraints in 63ms!
core-lightning_lightningd_1  | 2025-11-01T17:20:52.424Z INFO    lightningd: --------------------------------------------------
core-lightning_lightningd_1  | 2025-11-01T17:20:52.424Z INFO    lightningd: Server started with public key 038844375f952dcc5d35b33a0233a555fdeea65ebe88a4c8833eddb817fe371b7c, alias blockdyor CLN (color #000000) and lightningd v25.05
core-lightning_lightningd_1  | 2025-11-01T17:20:55.155Z INFO    plugin-sling: Refreshing alias map done in 3199ms!

core-lightning_app_proxy_1   | yarn run v1.22.19
core-lightning_app_proxy_1   | $ node ./bin/www
core-lightning_app_proxy_1   | [HPM] Proxy created: /  -> http://10.21.21.94:2103
core-lightning_app_proxy_1   | Waiting for 10.21.21.94:2103 to open...
core-lightning_app_proxy_1   | Core Lightning is now ready...
core-lightning_app_proxy_1   | Listening on port: 2103
core-lightning_app_proxy_1   | Proxy error: socket hang up
core-lightning_app_proxy_1   | [HPM] Error occurred while proxying request umbrel.local:2103/v1/shared/rate/USD to http://10.21.21.94:2103/ [ECONNRESET]
core-lightning_app_proxy_1   | Proxy error: connect ECONNREFUSED 10.21.21.94:2103
core-lightning_app_proxy_1   | [HPM] Error occurred while proxying request umbrel.local:2103/static/js/813.cdbe58bb.chunk.js to http://10.21.21.94:2103/ [ECONNREFUSED]

core-lightning_c-lightning-rest_1  | Reading config file
core-lightning_c-lightning-rest_1  | error: WARN: /root/.lightning/lightning-rpc is missing, using the bitcoin mainnet subdirectory at /root/.lightning/bitcoin instead.
core-lightning_c-lightning-rest_1  | warn: --- Starting the cl-rest server ---
core-lightning_c-lightning-rest_1  | warn: --- cl-rest api server is ready and listening on 0.0.0.0:2104 ---
core-lightning_c-lightning-rest_1  | warn: --- cl-rest doc server is ready and listening on 0.0.0.0:4001 ---
core-lightning_c-lightning-rest_1  | error: Lightning client connection error
core-lightning_c-lightning-rest_1  | node:events:491
core-lightning_c-lightning-rest_1  |       throw er; // Unhandled 'error' event
core-lightning_c-lightning-rest_1  |       ^
core-lightning_c-lightning-rest_1  | 
core-lightning_c-lightning-rest_1  | Error: connect ECONNREFUSED /root/.lightning/bitcoin/lightning-rpc
core-lightning_c-lightning-rest_1  |     at PipeConnectWrap.afterConnect [as oncomplete] (node:net:1278:16)
core-lightning_c-lightning-rest_1  | Emitted 'error' event on LightningClient instance at:
core-lightning_c-lightning-rest_1  |     at Socket.<anonymous> (/usr/src/app/lightning-client-js.js:80:23)
core-lightning_c-lightning-rest_1  |     at Socket.emit (node:events:513:28)
core-lightning_c-lightning-rest_1  |     at emitErrorNT (node:internal/streams/destroy:157:8)
core-lightning_c-lightning-rest_1  |     at emitErrorCloseNT (node:internal/streams/destroy:122:3)
core-lightning_c-lightning-rest_1  |     at processTicksAndRejections (node:internal/process/task_queues:83:21) {
core-lightning_c-lightning-rest_1  |   errno: -111,
core-lightning_c-lightning-rest_1  |   code: 'ECONNREFUSED',
core-lightning_c-lightning-rest_1  |   syscall: 'connect',
core-lightning_c-lightning-rest_1  |   address: '/root/.lightning/bitcoin/lightning-rpc'
core-lightning_c-lightning-rest_1  | }

core-lightning_app_1               | EXISTING_PUBKEY
core-lightning_app_1               | LIGHTNING_PUBKEY="038844375f952dcc5d35b33a0233a555fdeea65ebe88a4c8833eddb817fe371b7c"
core-lightning_app_1               | EXISTING_RUNE
core-lightning_app_1               | LIGHTNING_RUNE="[REDACTED]"
core-lightning_app_1               | Waiting for lightningd
core-lightning_app_1               | 2025/11/01 17:20:43 socat[11] E connect(5, AF=1 "/root/.lightning/bitcoin/lightning-rpc", 40): Connection refused
core-lightning_app_1               | 
core-lightning_app_1               | Waiting for lightningd
core-lightning_app_1               | {"jsonrpc":"2.0","id":1,"result":{"id":"038844375f952dcc5d35b33a0233a555fdeea65ebe88a4c8833eddb817fe371b7c","alias":"blockdyor CLN","color":"000000","num_peers":8,"num_pending_channels":0,"num_active_channels":8,"num_inactive_channels":0,"address":[{"type":"torv3","address":"[REDACTED].onion","port":9735}],"binding":[{"type":"ipv4","address":"10.21.21.96","port":9735}],"version":"v25.05","blockheight":921760,"network":"bitcoin","fees_collected_msat":1567215,"lightning-dir":"/root/.lightning/bitcoin","warning_lightningd_sync":"Still loading latest blocks from bitcoind.","our_features":{"init":"08a8880a8a59a1","node":"88a8880a8a59a1","channel":"","invoice":"02000002024100"}}}
core-lightning_app_1               | 038844375f952dcc5d35b33a0233a555fdeea65ebe88a4c8833eddb817fe371b7c
core-lightning_app_1               | Pubkey matches with existing pubkey.
core-lightning_app_1               | 
core-lightning_app_1               | > cln-application@0.0.7 start
core-lightning_app_1               | > npm run serve -w cln-application-backend
core-lightning_app_1               | 
core-lightning_app_1               | > cln-application-backend@0.0.7 serve
core-lightning_app_1               | > node ./dist/server.js
core-lightning_app_1               | 
core-lightning_app_1               | {"level":"warn","message":"\tServer running at http://10.21.21.94:2103","timestamp":"2025-11-01 17:20:55:2055"}
core-lightning_app_1               | {"level":"error","message":"\tRequest failed with status code 404","timestamp":"2025-11-01 17:21:00:210"}
core-lightning_app_1               | node:internal/errors:496
core-lightning_app_1               |     ErrorCaptureStackTrace(err);
core-lightning_app_1               |     ^
core-lightning_app_1               | 
core-lightning_app_1               | RangeError [ERR_HTTP_INVALID_STATUS_CODE]: Invalid status code: ERR_BAD_REQUEST
core-lightning_app_1               |     at new NodeError (node:internal/errors:405:5)
core-lightning_app_1               |     at ServerResponse.writeHead (node:_http_server:347:11)
core-lightning_app_1               |     at ServerResponse._implicitHeader (node:_http_server:338:8)
core-lightning_app_1               |     at write_ (node:_http_outgoing:915:9)
core-lightning_app_1               |     at ServerResponse.end (node:_http_outgoing:1026:5)
core-lightning_app_1               |     at res.end (/app/node_modules/express-winston/index.js:317:17)
core-lightning_app_1               |     at ServerResponse.send (/app/node_modules/express/lib/response.js:232:10)
core-lightning_app_1               |     at ServerResponse.json (/app/node_modules/express/lib/response.js:278:15)
core-lightning_app_1               |     at handleError (file:///app/apps/backend/dist/shared/error-handler.js:13:69)
core-lightning_app_1               |     at file:///app/apps/backend/dist/controllers/shared.js:74:24 {
core-lightning_app_1               |   code: 'ERR_HTTP_INVALID_STATUS_CODE'
core-lightning_app_1               | }
core-lightning_app_1               | 
core-lightning_app_1               | Node.js v18.20.3
core-lightning_app_1               | npm error Lifecycle script `serve` failed with error:
core-lightning_app_1               | npm error Error: command failed
core-lightning_app_1               | npm error   in workspace: cln-application-backend@0.0.7
core-lightning_app_1               | npm error   at location: /app/apps/backend

[paulscode]

I see a similar error (though the chunk number varies) on both the Umbrel and Start9 packages.

[blockdyor]

I'm curious if this could be a common issue? For now, I've closed all the channels and shut down the node just to be safe.

[ShahanaFarooqui]

Errors from core-lightning_c-lightning-rest_1 can be safely ignored for now, as c-lightning-rest is already in its sunset phase and AFAIK no application actively rely on it anymore.

The other error, Request failed with status code 404, originates from the /v1/shared/rate endpoint. This likely occurs when the server at https://green-bitcoin-mainnet.blockstream.com/prices/v0/venues/ is temporarily unavailable. This server provides the BTC-to-fiat conversion rates we use. However, the application should not fail entirely just because the fiat rate cannot be fetched. I’ve updated the code with PR #133 to return zero instead of raising an error in such cases, so fiat values will display as "0" rather than causing the application to fail.