Panic inside ClientContext.h since git version 83f5f29c

[fsommer1968]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: [ESP-12]
• Core Version: [209e467]
• Development Env: [Arduino IDE]
• Operating System: [Windows]

Settings in IDE

• Module: [Wemos D1 mini]
• Flash Mode: [dio]
• Flash Size: [4MB]
• lwip Variant: [Higher Bandwidth]
• Reset Method: [nodemcu]
• Flash Frequency: [40Mhz]
• CPU Frequency: [80Mhz]
• Upload Using: [OTA|SERIAL]
• Upload Speed: [other |(serial upload only)

Problem Description

Hi, after 83f5f29 a problem was introduced, that affects my sketches using BearSSL?! When I rest my local repo to 83f5f29 everything is fine again

Debug Messages

> --------------- CUT HERE FOR EXCEPTION DECODER ---------------
17:47:30.084 -> 
17:47:30.084 -> Panic src/include/ClientContext.h:384 size_t ClientContext::write(Stream&): Assertion 'stream.hasPeekBufferAPI()' failed.
17:47:30.084 -> 
17:47:30.084 -> >>>stack>>>

Decoding stack results
0x4020f1e1: ClientContext::write(Stream&) (C:\Program Files at x86)\Arduino\hardware\esp8266com\esp8266\libraries\ESP8266WiFi\src/include/ClientContext.h line 385
0x4020f234: WiFiClient::write(unsigned char const*, unsigned int) (C:\Program Files at x86)\Arduino\hardware\esp8266com\esp8266\libraries\ESP8266WiFi\src\WiFiClient.cpp line 217
0x4021d000: __esp_yield() (C:\Program Files at x86)\Arduino\hardware\esp8266com\esp8266\cores\esp8266\core_esp8266_main.cpp line 116
0x40210410: BearSSL::WiFiClientSecureCtx::_run_until(unsigned int, bool) (C:\Program Files at x86)\Arduino\hardware\esp8266com\esp8266\libraries\ESP8266WiFi\src\WiFiClientSecureBearSSL.cpp line 521
0x4023a9dc: br_ssl_hs_client_init_main at src/ssl/ssl_hs_client.c line 907
0x4023aa2c: br_ssl_hs_client_run at src/ssl/ssl_hs_client.c line 960
0x4023a0a6: br_ssl_engine_hs_reset at src/ssl/ssl_engine.c line 1319
0x402104af: BearSSL::WiFiClientSecureCtx::_wait_for_handshake() (C:\Program Files at x86)\Arduino\hardware\esp8266com\esp8266\libraries\ESP8266WiFi\src\WiFiClientSecureBearSSL.cpp line 601
0x40210682: BearSSL::WiFiClientSecureCtx::_connectSSL(char const*) (C:\Program Files at x86)\Arduino\hardware\esp8266com\esp8266\libraries\ESP8266WiFi\src\WiFiClientSecureBearSSL.cpp line 1180
0x4020f50b: WiFiClient::connect(IPAddress, unsigned short) (C:\Program Files at x86)\Arduino\hardware\esp8266com\esp8266\libraries\ESP8266WiFi\src/include/ClientContext.h line 149
0x40210719: BearSSL::WiFiClientSecureCtx::connect(char const*, unsigned short) (C:\Program Files at x86)\Arduino\hardware\esp8266com\esp8266\libraries\ESP8266WiFi\src\WiFiClientSecureBearSSL.cpp line 231
0x40221272: BearSSL::WiFiClientSecure::connect(char const*, unsigned short) (c:\program files at x86)\arduino\hardware\esp8266com\esp8266\tools\xtensa-lx106-elf\xtensa-lx106-elf\include\c++\10.2.0\bits/shared_ptr_base.h line 1324
0x40202ea5: MQTT_reconnect() at C:\Users\Frank Sommer\Documents\Arduino\RolladenAktorMQTTWeb44c/RolladenAktorMQTTWeb44c.ino line 4543
0x402162c7: esp8266::MDNSImplementation::MDNSResponder::_updateProbeStatus() (C:\Program Files at x86)\Arduino\hardware\esp8266com\esp8266\cores\esp8266/PolledTimeout.h line 158
0x4020fb98: BearSSL::WiFiClientSecureCtx::connected() (C:\Program Files at x86)\Arduino\hardware\esp8266com\esp8266\libraries\ESP8266WiFi\src\WiFiClientSecureBearSSL.cpp line 260
0x4022128c: BearSSL::WiFiClientSecure::connected() (c:\program files at x86)\arduino\hardware\esp8266com\esp8266\tools\xtensa-lx106-elf\xtensa-lx106-elf\include\c++\10.2.0\bits/shared_ptr_base.h line 1324
0x4020318c: MQTT_connect() at C:\Users\Frank Sommer\Documents\Arduino\RolladenAktorMQTTWeb44c/RolladenAktorMQTTWeb44c.ino line 5947
0x4020cdee: loop() at C:\Users\Frank Sommer\Documents\Arduino\RolladenAktorMQTTWeb44c/RolladenAktorMQTTWeb44c.ino line 6774
0x4021bb5d: String::invalidate() (C:\Program Files at x86)\Arduino\hardware\esp8266com\esp8266\cores\esp8266/WString.h line 302
0x4020c521: setup() (C:\Program Files at x86)\Arduino\hardware\esp8266com\esp8266\cores\esp8266/WString.h line 86
0x402214e0: std::_Function_handler ::_M_invoke(std::_Any_data const&, char*&&, unsigned char*&&, unsigned int&&) (c:\program files at x86)\arduino\hardware\esp8266com\esp8266\tools\xtensa-lx106-elf\xtensa-lx106-elf\include\c++\10.2.0\bits/std_function.h line 289
0x40100458: ets_post(uint8, ETSSignal, ETSParam) (C:\Program Files at x86)\Arduino\hardware\esp8266com\esp8266\cores\esp8266\core_esp8266_main.cpp line 181
0x4021d120: loop_wrapper() (C:\Program Files at x86)\Arduino\hardware\esp8266com\esp8266\cores\esp8266\core_esp8266_main.cpp line 201

[earlephilhower]

Please post a full MCVE, or there is nothing we can do since we can't reproduce.

The GIT commit you're referencing also has no relation to BearSSL or WiFi client. The failing assert is related to a passed-in Stream and added with 1c57b3408cfdc2956d3c11307b79e2e0055db038 (#7951)

[d-a-v]

@fsommer1968 can you try with #7987 ?

edit: or try with board manager alpha release v0.0.2 ?

[fsommer1968]

@d-a-v:
I checked with v0.0.2, but it does not help. I suspect a problem between PubSubClient and BearSSL as the panic occurs during connect of the MQTT client using TLS. A quick test without TLS does not lead to an error!
Indeed it does help somehow, thx so far. Is it possible that there is something with 2nd heap support, or I use 2nd heap support in a way that is not supported.
Is 2nd heap supported for use with bearssl and/or wificlient(secure) ??

[d-a-v]

Indeed it does help somehow, thx so far

Is it a confirmation that PubSubClient w/ TLS is fixed by #7987 ?
(edit: Or is it just with latest master ?)

Is it possible that there is something with 2nd heap support, or I use 2nd heap support in a way that is not supported.

Everything from the core should work when ISR-based iram/flash access is disabled, unless an iram/flash pointer is given to a function without _P() in its name (like when using .write() instead of .write_P() with such pointers).

You understand that we must be able to reproduce in order to fix properly.

[fsommer1968]

OK, I understand what I have to test. A MCVE will take few days.
N.B.: I used V0.0.2

[fsommer1968]

Hi,

this is a MCVE:

/*
 Basic ESP8266 MQTT example
 This sketch demonstrates the capabilities of the pubsub library in combination
 with the ESP8266 board/library.
 It connects to an MQTT server then:
  - publishes "hello world" to the topic "outTopic" every two seconds
  - subscribes to the topic "inTopic", printing out any messages
    it receives. NB - it assumes the received payloads are strings not binary
  - If the first character of the topic "inTopic" is an 1, switch ON the ESP Led,
    else switch it off
 It will reconnect to the server if the connection is lost using a blocking
 reconnect function. See the 'mqtt_reconnect_nonblocking' example for how to
 achieve the same result without blocking the main loop.
 To install the ESP8266 board, (using Arduino 1.6.4+):
  - Add the following 3rd party board manager under "File -> Preferences -> Additional Boards Manager URLs":
       http://arduino.esp8266.com/stable/package_esp8266com_index.json
  - Open the "Tools -> Board -> Board Manager" and click install for the ESP8266"
  - Select your ESP8266 in "Tools -> Board"
*/

#include <ESP8266WiFi.h>
#include <PubSubClient.h>

// Update these with values suitable for your network.

const char* ssid = "HACKME";
const char* password = "HACKME";
const char* mqtt_server = "test.mosquitto.org";

WiFiClientSecure espClient;
PubSubClient client(espClient);
unsigned long lastMsg = 0;
#define MSG_BUFFER_SIZE	(50)
char msg[MSG_BUFFER_SIZE];
int value = 0;

void setup_wifi() {

  delay(10);
  // We start by connecting to a WiFi network
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);

  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  randomSeed(micros());

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}

void callback(char* topic, byte* payload, unsigned int length) {
  Serial.print("Message arrived [");
  Serial.print(topic);
  Serial.print("] ");
  for (int i = 0; i < length; i++) {
    Serial.print((char)payload[i]);
  }
  Serial.println();

  // Switch on the LED if an 1 was received as first character
  if ((char)payload[0] == '1') {
    digitalWrite(BUILTIN_LED, LOW);   // Turn the LED on (Note that LOW is the voltage level
    // but actually the LED is on; this is because
    // it is active low on the ESP-01)
  } else {
    digitalWrite(BUILTIN_LED, HIGH);  // Turn the LED off by making the voltage HIGH
  }

}

void reconnect() {
  // Loop until we're reconnected
  while (!client.connected()) {
    Serial.print("Attempting MQTT connection...");
    // Create a random client ID
    String clientId = "ESP8266Client-";
    clientId += String(random(0xffff), HEX);
    // Attempt to connect
    if (client.connect(clientId.c_str())) {
      Serial.println("connected");
      // Once connected, publish an announcement...
      client.publish("outTopic", "hello world");
      // ... and resubscribe
      client.subscribe("inTopic");
    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
      // Wait 5 seconds before retrying
      delay(5000);
    }
  }
}

void setup() {
  pinMode(BUILTIN_LED, OUTPUT);     // Initialize the BUILTIN_LED pin as an output
  Serial.begin(74880);
  setup_wifi();

  //SSL setup
  espClient.setInsecure(); //no certificate check
  // SSL setup
  client.setServer(mqtt_server, 8883);
  client.setCallback(callback);
}

void loop() {

  if (!client.connected()) {
    reconnect();
  }
  client.loop();

  unsigned long now = millis();
  if (now - lastMsg > 2000) {
    lastMsg = now;
    ++value;
    snprintf (msg, MSG_BUFFER_SIZE, "hello world #%ld", value);
    Serial.print("Publish message: ");
    Serial.println(msg);
    client.publish("outTopic", msg);
  }
}

The error appears with TLS/SSL only when using PubSubClient
To trigger the error, it is enough to set mmu=4816H !

 Panic src/include/ClientContext.h:384 size_t ClientContext::write(Stream&): Assertion 'stream.hasPeekBufferAPI()' failed.
 
Stack trace:
`Decoding stack results
0x4021f9c5: br_ssl_hs_client_run at src/ssl/ssl_hs_client.c line 1372
0x4020257d: WiFiClient::write(unsigned char const*, unsigned int) at C:\Users\Frank Sommer\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\0.0.2\libraries\ESP8266WiFi\src/include/ClientContext.h line 385
0x40205a00: can_yield() at C:\Users\Frank Sommer\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\0.0.2\cores\esp8266\core_esp8266_main.cpp line 109
0x40203574: BearSSL::WiFiClientSecureCtx::_run_until(unsigned int, bool) at C:\Users\Frank Sommer\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\0.0.2\libraries\ESP8266WiFi\src\WiFiClientSecureBearSSL.cpp line 521
0x4021f2cc: br_ssl_hs_client_init_main at src/ssl/ssl_hs_client.c line 907
0x4021f31c: br_ssl_hs_client_run at src/ssl/ssl_hs_client.c line 960
0x4021e996: br_ssl_engine_hs_reset at src/ssl/ssl_engine.c line 1319
0x40203613: BearSSL::WiFiClientSecureCtx::_wait_for_handshake() at C:\Users\Frank Sommer\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\0.0.2\libraries\ESP8266WiFi\src\WiFiClientSecureBearSSL.cpp line 601
0x402037ea: BearSSL::WiFiClientSecureCtx::_connectSSL(char const*) at C:\Users\Frank Sommer\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\0.0.2\libraries\ESP8266WiFi\src\WiFiClientSecureBearSSL.cpp line 1180
0x402027e3: WiFiClient::connect(IPAddress, unsigned short) at C:\Users\Frank Sommer\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\0.0.2\libraries\ESP8266WiFi\src/include/ClientContext.h line 149
0x40203881: BearSSL::WiFiClientSecureCtx::connect(char const*, unsigned short) at C:\Users\Frank Sommer\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\0.0.2\libraries\ESP8266WiFi\src\WiFiClientSecureBearSSL.cpp line 231
0x4020840a: BearSSL::WiFiClientSecure::connect(char const*, unsigned short) at c:\users\frank sommer\appdata\local\arduino15\packages\esp8266\tools\xtensa-lx106-elf-gcc\3.0.0-newlib4.0.0-gnu23-48f7b08\xtensa-lx106-elf\include\c++\10.2.0\bits/shared_ptr_base.h line 1324
0x40204480: PubSubClient::connect(char const*, char const*, char const*, char const*, unsigned char, bool, char const*, bool) at C:\Users\Frank Sommer\Documents\Arduino\libraries\PubSubClient\src\PubSubClient.cpp line 196
0x402057d1: String::concat(char const*, unsigned int) at C:\Users\Frank Sommer\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\0.0.2\cores\esp8266\WString.cpp line 309
0x4020b92f: rand at /workdir/repo/newlib/newlib/libc/stdlib/rand.c line 79
0x4020466c: PubSubClient::connect(char const*) at C:\Users\Frank Sommer\Documents\Arduino\libraries\PubSubClient\src\PubSubClient.cpp line 166
0x4020587e: String::concat(String const&) at C:\Users\Frank Sommer\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\0.0.2\cores\esp8266/WString.h line 284
0x40201263: reconnect() at C:\Users\Frank Sommer\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\0.0.2\cores\esp8266/WString.h line 284
0x402089f5: PubSubClient::connected() at C:\Users\Frank Sommer\Documents\Arduino\libraries\PubSubClient\src\PubSubClient.cpp line 690
0x4020102c: callback(char*, unsigned char*, unsigned int) at C:\Users\FRANKS~1\AppData\Local\Temp\arduino_modified_sketch_248226/mqtt_esp8266.ino line 62
0x40201382: loop() at C:\Users\FRANKS~1\AppData\Local\Temp\arduino_modified_sketch_248226/mqtt_esp8266.ino line 122
0x40205b54: loop_wrapper() at C:\Users\Frank Sommer\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\0.0.2\cores\esp8266\core_esp8266_main.cpp line 201

[d-a-v]

Thanks 👍 this is reproducible.

edit edit²:

First findings: it is caused by this:

Arduino/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp

Line 1121 in cfbbd6f

HeapSelectIram ephemeral;

second thought: this is not really the cause, explanation is below.

A non byte-addressable IRAM pointer is internally passed to WiFiClient.
Before the Stream::send changes, that was not causing any issue maybe because 1. the pointer is 4-bvyte aligned, 2. its size too and 3. lwIP was taking it all so memcpy was not failing. So we were lucky. because the unaligned exception handler was installed.
After the changes, byte-addressability is checked and StreamConstPtr adapts its behaviour => preventing direct access (peek buffer) for that kind of address (iram, flash) even when the handler is installed.

ClientContext needs anyway a big change. It was designed to rely on a Stream-like class for its input. Since the Stream::sendchanges it relies on a Stream class (so ClientContext received minimal changes), but truth is it uses a pointer and a size in the main ::write() function. Changing the API for a regular pointer will avoid to use intermediary and nearly useless StreamConstPtr objects and will restore the lucky possibility we had before. The question is, shall we rely on this.

What shall be considered:

• fix ClientContext to not use Stream but regular pointer
• temporarily? remove this HeapSelectIram not an option, exception handler is installed in that case

for later?:

• use memcpy_P in lwIP (there are #defines for that)
• check how Stream::send implementations could tell that they can receive non byte-addressable pointers (if they always use memcpy_P).

[fsommer1968]

temporarily? remove this HeapSelectIram

Do I understand right, that as soon as 2nd Heap support is enabled BearSSL will use it ?
I vote for removing this HeapSelectIram. As mentioned in the comments: This is an example ;-) Not an option as d-a-v- explains

[d-a-v]

@fsommer,
I successfully ran your MCVE with and without the @mhightower83 / @earlephilhower 's nonx32xfer handler.
Recent changes in ClientContext were not sufficient.

#7987 is updated, can you retry with it ?
(or with alpha release v0.0.2: you must install another version of this core in the arduino ide, then re-install v0.0.2 back since it has been updated with #7987 changes)

[fsommer1968]

@d-a-v,

success! PubSubClient with TLS working again, no panic anymore.
Thanks!