Health release for pypowerwall?

[JohnJ9ml]

Is there a new release for the pypowerwall changes needed for the health endpoint? I didn't see one accompanying the dashboard release.

[jasonacox]

That's a good point and I can cut one, but also it wasn't a library change, just the proxy: jasonacox/pypowerwall:0.13.1t76

[JohnJ9ml]

I picked up the new build, and now I'm not getting the data back that drives the animation on the dashboard.

`jjames@crewchief:powerwall$ curl http://powerwall:8675/csv
0.00,0.00,0.00,0.00,41.44
jjames@crewchief:powerwall$ time !!
time curl http://powerwall:8675/csv
0.00,0.00,0.00,0.00,41.44

real 0m54.553s
user 0m0.011s
sys 0m0.016s
`

I'm still looking about for other changes that have taken place.

I did get moved to PW version 25.18.1-g, but that was a couple days ago.

[JohnJ9ml]

Perhaps knowing which calls were resulting in failure would be something good to add to the health data:

{ "cache_ttl_seconds": 30, "graceful_degradation": true, "fail_fast_mode": false, "health_check_enabled": true, "startup_time": "2025-07-10T09:54:26", "current_time": "2025-07-10T11:50:50.293953", "connection_health": { "consecutive_failures": 0, "total_failures": 14541, "total_successes": 16018, "is_degraded": false, "last_success_time": 1752173450.0075507, "last_success_age_seconds": 0.28641390800476074 }, "cached_data": { "cache_size": 1, "endpoints": { "/soe": { "age_seconds": 0.28641247749328613, "is_expired": false } } } }

A lot of failures, a lot of successes - it would probably be good to know what's succeeding, what's failing.

[JohnJ9ml]

Attached is a 20 second or so dump of the debug logs from the proxy while all of this is going on.
proxydebug.txt

[JohnJ9ml]

And then, later on it seemingly healed itself:

{ "cache_ttl_seconds": 30, "graceful_degradation": true, "fail_fast_mode": false, "health_check_enabled": true, "startup_time": "2025-07-10T09:54:26", "current_time": "2025-07-10T14:03:05.036225", "connection_health": { "consecutive_failures": 0, "total_failures": 25964, "total_successes": 46685, "is_degraded": false, "last_success_time": 1752181385.0065017, "last_success_age_seconds": 0.029733896255493164 }, "cached_data": { "cache_size": 3, "endpoints": { "/soe": { "age_seconds": 0.029995441436767578, "is_expired": false }, "/strings": { "age_seconds": 4.401860475540161, "is_expired": false }, "/aggregates": { "age_seconds": 1.3289411067962646, "is_expired": false } } } }

pyproxy.txt

[JohnJ9ml]

The blasted thing just heals itself. I even restarted my primary PW3 - first time ever - and treated it like a Windows device - what's a good reboot among friends? The problems continued on through the PW restart. Almost by surprise, the data suddenly started flowing again with no intervention at all from myself.

[jasonacox]

Here is a new one to try: jasonacox/pypowerwall:0.13.2t77-beta

I added endpoint statistics. I will need to load test it to ensure it doesn't create too much overhead, but should help with your troubleshooting efforts. Also, I incorporated some optimization from v0.13.2.

{
  "cache_ttl_seconds": 30,
  "graceful_degradation": true,
  "fail_fast_mode": false,
  "health_check_enabled": true,
  "startup_time": "2025-07-11T23:10:34",
  "current_time": "2025-07-11T23:13:01.210777",
  "proxy_stats": {
    "total_gets": 411,
    "total_posts": 0,
    "total_errors": 0,
    "total_timeouts": 0
  },
  "connection_health": {
    "consecutive_failures": 0,
    "total_failures": 0,
    "total_successes": 531,
    "is_degraded": false,
    "last_success_time": 1752300780.027656,
    "last_success_age_seconds": 1.1831517219543457
  },
  "cached_data": {
    "cache_size": 4,
    "endpoints": {
      "/aggregates": {
        "age_seconds": 1.2027292251586914,
        "is_expired": false
      },
      "/soe": {
        "age_seconds": 1.1908392906188965,
        "is_expired": false
      },
      "/strings": {
        "age_seconds": 1.1831464767456055,
        "is_expired": false
      },
      "/vitals": {
        "age_seconds": 119.38730812072754,
        "is_expired": true
      }
    }
  },
  "endpoint_statistics": {
    "/aggregates": {
      "total_calls": 53,
      "successful_calls": 53,
      "failed_calls": 0,
      "success_rate_percent": 100.0,
      "last_success_age_seconds": 1.2027406692504883
    },
    "/soe": {
      "total_calls": 30,
      "successful_calls": 30,
      "failed_calls": 0,
      "success_rate_percent": 100.0,
      "last_success_age_seconds": 1.1908516883850098
    },
    "/strings": {
      "total_calls": 30,
      "successful_calls": 30,
      "failed_calls": 0,
      "success_rate_percent": 100.0,
      "last_success_age_seconds": 1.1831586360931396
    },
    "/vitals": {
      "total_calls": 1,
      "successful_calls": 1,
      "failed_calls": 0,
      "success_rate_percent": 100.0,
      "last_success_age_seconds": 119.38731694221497
    }
  }
}

[JohnJ9ml]

Sorry for not picking this out earlier, but the health output should include the proxy version. It would be good to track historical behavior.

[jasonacox]

Are you thinking about something similar to what we do for /stats?

{
"pypowerwall": "0.13.2 Proxy t79",
...
}

[jasonacox]

If so, you can test: jasonacox/pypowerwall:0.13.2t79-beta

[JohnJ9ml]

yeah, that's what I was thinking about. Just a way to track it and store it and be able to look back and note behavioral changes.

[JohnJ9ml]

I realize the title of the thread is not what is being discussed here, but it does continue on in the spirit of the ensuing discussion:

I reach this state from time to time where I can get the level() of the batteries, but not the data that would be the result of the power() call, the /csv endpoint, etc. I'm back in that mode again this morning, and have this for the health output:

{"pypowerwall": "0.14.3 Proxy t83", "cache_ttl_seconds": 60, "graceful_degradation": true, "fail_fast_mode": false, "health_check_enabled": true, "startup_time": "2025-11-03T07:04:10", "current_time": "2025-11-03T07:39:23.419721", "proxy_stats": {"total_gets": 1081, "total_posts": 0, "total_errors": 0, "total_timeouts": 729}, "connection_health": {"consecutive_failures": 0, "total_failures": 729, "total_successes": 161, "is_degraded": false, "last_success_time": 1762184359.53583, "last_success_age_seconds": 3.8839049339294434}, "cached_data": {"cache_size": 0, "endpoints": {}}}

Lacking any endpoint information in the health call, what I'm wondering at this point is this: what tedapi endpoint does the data for the power() call come from? I do find it perplexing that some of the conversation with the primary PW3 is possible, but some is not. It's not something cured by a proxy restart. In the past, it's not been cured by a PW3 restart either. I'm a bit curious as to what endpoints are included in the total_failures count above.

[jasonacox]

This is a big answer, but I spent some time with AI trying to come up with several options as well as some potential improvements to pypowerwall / proxy.

TEDAPI Endpoint Behavior and Health Monitoring

Issue: level() Works But power() Fails

This document explains why you can get battery level data but not power data in certain situations.

What TEDAPI Endpoint Does power() Use?

The power() function retrieves data from /api/meters/aggregates, which in TEDAPI mode requires TWO underlying TEDAPI endpoints:

1. tedapi.get_config() - fetches https://{gateway_ip}/tedapi/v1 with config.json request

   • Provides meter configuration data
   • Cached for pwconfigexpire seconds (default: 5 seconds)

2. tedapi.get_status() - fetches https://{gateway_ip}/tedapi/v1 with status request

   • Provides real-time power readings from meters
   • Cached for pwcacheexpire seconds (default: 5 seconds)

The get_api_meters_aggregates() method (pypowerwall/tedapi/pypowerwall_tedapi.py, lines 346-380) calls both:

config = self.tedapi.get_config(force=force)
status = self.tedapi.get_status(force=force)
if not isinstance(config, dict) or not isinstance(status, dict):
    return None

What TEDAPI Endpoint Does level() Use?

In contrast, level() only uses ONE TEDAPI endpoint:

• tedapi.battery_level() which calls tedapi.get_status() only
• Retrieves battery percentage from the status data

This explains the behavior you're seeing:

• If get_config() is failing but get_status() is succeeding, level() will work but power() will fail
• The /csv endpoint also uses /api/meters/aggregates, so it would fail for the same reason

Understanding the Health Counters

What total_failures Tracks

The total_failures count (729 in your example) is a global counter that tracks ALL failed calls through safe_pw_call() for ANY endpoint. It's incremented when:

1. Any safe_pw_call() returns None
2. Network errors occur (ConnectionError, TimeoutError, etc.)
3. API errors or unexpected exceptions happen
4. TEDAPI returns non-200 status codes

The counter is updated in update_connection_health(success=False) (proxy/server.py, line 403).

What total_timeouts Tracks

The total_timeouts counter (729 in your example, matching total_failures) tracks:

• HTTP timeout errors
• Socket timeout errors
• Connection timeout errors
• Any TimeoutError exceptions

This suggests all 729 failures were timeout-related.

Why No Endpoint Information?

Your health output shows:

"cached_data": {"cache_size": 0, "endpoints": {}}

This empty endpoints object occurs because:

1. Most proxy endpoints use safe_pw_call() directly, not safe_endpoint_call()

   • safe_pw_call() updates global health counters only
   • safe_endpoint_call() updates per-endpoint statistics

2. Per-endpoint tracking requires safe_endpoint_call()

   • Only certain optimized endpoints use this wrapper
   • Most standard API passthrough calls don't populate endpoint-level stats

3. Cache has been cleared or expired

   • With PW_CACHE_TTL=60, cached responses expire after 60 seconds
   • If no successful calls happened recently, cache would be empty

Possible Root Causes

Given that this persists across proxy and PW3 restarts, the issue is likely:

1. TEDAPI Rate Limiting

• The PW3 may be rate-limiting get_config() calls more aggressively than get_status() calls
• Look for 429/503 status codes in debug logs
• The proxy has a 5-minute cooldown when rate limited (line 167 in tedapi/init.py)

2. Selective Endpoint Failure

• The PW3 firmware may have an issue with the config endpoint specifically
• Protobuf message parsing issues for config requests
• Network routing issues for certain TEDAPI paths

3. Cache Expiration Timing Issue

• get_config() cache expires every 5 seconds
• get_status() cache also expires every 5 seconds
• If config endpoint is slow/timing out, it may never successfully refresh

4. Connection Pool Exhaustion

• With PW_POOL_MAXSIZE=15 (default), if connections are stuck waiting for config responses
• Status calls might succeed using different pool connections

Debugging Steps

1. Enable Debug Logging

PW_DEBUG=yes

This will show exactly which TEDAPI calls are failing with detailed error messages.

2. Test Individual TEDAPI Endpoints

Check if config endpoint is the problem:

curl http://yourproxy:8675/tedapi/config

Compare with status endpoint:

curl http://yourproxy:8675/tedapi/status

Check battery-specific endpoint:

curl http://yourproxy:8675/tedapi/battery

3. Check Direct Gateway Access

If you can reach the gateway directly:

# Test status (should work based on level() working)
curl -k --user "Tesla_Energy_Device:YourGatewayPassword" \
  https://192.168.91.1/tedapi/v1 \
  -H "Content-Type: application/octet-stream" \
  --data-binary @status_request.bin

# Test config (likely timing out)
curl -k --user "Tesla_Energy_Device:YourGatewayPassword" \
  https://192.168.91.1/tedapi/v1 \
  -H "Content-Type: application/octet-stream" \
  --data-binary @config_request.bin

4. Monitor Health Endpoint During Failures

watch -n 5 'curl -s http://yourproxy:8675/health | jq'

Look for:

• is_degraded changing to true
• consecutive_failures incrementing
• last_success_age_seconds growing

5. Check Proxy Logs for Patterns

Look for:

• "Error fetching config" messages
• "Rate limited" warnings
• "Timeout" errors
• Specific HTTP status codes

6. Try Increasing Timeouts and Cache

PW_TIMEOUT=10          # Increase from 5 to 10 seconds
PW_CACHE_EXPIRE=30     # Increase cache time
PW_CACHE_TTL=120       # Allow older cached data

Workaround Options

While debugging, you can:

1. Use alternative endpoints that don't require config

   • /api/system_status/soe (battery level) - works
   • Direct meter readings if available

2. Increase cache TTL to use stale data

   PW_CACHE_TTL=300  # 5 minutes

3. Reset health counters to clear degraded state

   curl http://yourproxy:8675/health/reset

Code References

• power() implementation: pypowerwall/__init__.py, line 370
• level() implementation: pypowerwall/__init__.py, line 354
• get_api_meters_aggregates(): pypowerwall/tedapi/pypowerwall_tedapi.py, line 346
• get_api_system_status_soe(): pypowerwall/tedapi/pypowerwall_tedapi.py, line 225
• Health tracking: proxy/server.py, lines 403-431
• /health endpoint: proxy/server.py, lines 1095-1177

Possible Improvements to pypowerwall

1. Add per-endpoint tracking to more calls

   • Wrap critical API calls with safe_endpoint_call()
   • Track config vs status endpoints separately

2. Add specific TEDAPI endpoint health checks

   • Separate counters for get_config() vs get_status()
   • Expose in /health endpoint

3. Log rate limit events more prominently

   • Current cooldown messages are at ERROR level
   • Could track rate limit events separately

4. Add TEDAPI-specific debugging endpoint

   • /tedapi/health showing config/status call success rates
   • Cache hit/miss ratios
   • Last successful call timestamps per endpoint

[JohnJ9ml]

Thank you very, very much for a very thorough and detail-rich response. There is a lot for me to unpack here, but I find it very interesting at this point of my journey.

I have a rather curious situation on my end where my initial installation by Tesla was rather poorly configured. I expanded my system and had a Tesla Certified Installer come and do the work, which included expansion of the system, expansion of the storage, and reconfiguring some of the initial work that was (IMO) done "sub-optimally". (The story is an Old Man and the Sea style journey at this point.) There are now alerts that pop up from time to time, and I have a strong suspicion that I now have a bit of a connectivity issue popping up between PW3s in the installation that will have to be addressed.

I've been wondering which calls sometimes put me in this hell-ish state as a way to see if there is a difference between those that can talk purely to the lead PW3 versus those that need to call through to the entire install. This level of detail I hope will allow me to ponder this a bit more completely.

[jasonacox]

Fascinating story, @JohnJ9ml - Sorry to hear about the troubles. These systems are indeed a technological wonder, but sometimes it feels like they're trying to out-Rube-Goldberg themselves... so many dependencies and links can break. At any rate, I'll add some of these updates to my backlog, not that it will help your situation but may make it easier to troubleshoot.