Example 16 not waking from deep sleep correctly after upgrade to v2.1

[PaulZC]

Original issue was raised by @a-schneider-fmi in #36:

Since the update to version 2, I experience another issue which may or may not be related: the tracker seems not to wake from deep sleep after 2 messages have been sent. From the configuration tool, I see the following messages:

[...]
>>> Message sent! <<<
Clearing the MO buffer.
The number of messages in the MT queue is: 0
Putting the 9603N to sleep.
Getting ready to put the Apollo3 into deep sleep...
Disabling 9603N power...
Disabling the supercapacitor charger...
Powering down the GNSS...
Going into deep sleep until next WAKEINT (60 seconds).

And then it hangs. This happens with several TXINT settings (I tried 2 minutes, 5 minutes, 10 minutes).

I have tested example 12 to debug. It seems to work, at least it runs for more than half an hour. Do you have any suggestion how to investigate further?

[a-schneider-fmi]

I made some more tests with the tracker with different settings, and after some settings change it suddenly works again, even when changing the settings back to what they have been before. I have no idea what is different.

[PaulZC]

Hi Andreas,

Restoring the Artemis peripherals and pins to full function after deep sleep is a lot more complex in v2.1 than it was with v1.2 of the core. Getting that part of the code right on OpenLog Artemis took a lot of effort.

It will be a question of making sure that the AGT deep sleep and wake code is exactly the same as the OpenLog Artemis code.

Example12 is much simpler and doesn't have to worry about restoring (e.g.) Serial1, agtWire and all the pins and interrupts to full functionality after deep sleep.

I will try and investigate this when I get some 'spare' time. (It is in short supply at the moment! ;-) )

Meanwhile if you discover any more clues, please report them here.

Very best wishes,
Paul

[a-schneider-fmi]

This morning a made another test with the same configuration that worked yesterday evening (i.e. no changes, just plugged the device to power), and it stopped after the first message.

[PaulZC]

Hi Andreas,
OK - thank you. I will investigate this as soon as I can.
Very best wishes,
Paul

[a-schneider-fmi]

To test more, I tried to skip the deep sleep phase by adding the following lines in case zzz, line 2251 (i.e. after the modem is powered down):

      // TEST
      Serial.print(F("Delaying WAKEINT ("));
      Serial.print(wake_int);
      Serial.println(F(" seconds)."));
      delay(wake_int*1000);
      loop_step = loop_init;
      break; // TEST

Have I missed something? Now it hangs when reading the PHT sensor after sending two messages.

Transmitting message...
>>> Message sent! <<<
Clearing the MO buffer.
The number of messages in the MT queue is: 0
Putting the 9603N to sleep.
Getting ready to put the Apollo3 into deep sleep...
Disabling 9603N power...
Disabling the supercapacitor charger...
Delaying WAKEINT (60 seconds).


Artemis Global Tracker
Software Version: 2.0


Ready to accept configuration settings via Serial...


Getting the PHT readings...
*** Could not detect the MS8607 sensor. Trying again... ***
Pressure (mbar): 973
Temperature (C * 10^-2): 1980
Humidity (%RH * 10^-2): 5703
Getting ready to put the Apollo3 into deep sleep...
Disabling 9603N power...
Disabling the supercapacitor charger...
Delaying WAKEINT (60 seconds).


Artemis Global Tracker
Software Version: 2.0


Ready to accept configuration settings via Serial...


Getting the PHT readings...
Pressure (mbar): 973
Temperature (C * 10^-2): 1879
Humidity (%RH * 10^-2): 6014
Getting ready to put the Apollo3 into deep sleep...
Disabling 9603N power...
Disabling the supercapacitor charger...
Delaying WAKEINT (60 seconds).


Artemis Global Tracker
Software Version: 2.0


Ready to accept configuration settings via Serial...


Getting the PHT readings...
Pressure (mbar): 973
Temperature (C * 10^-2): 1784
Humidity (%RH * 10^-2): 6414
Getting ready to put the Apollo3 into deep sleep...
Disabling 9603N power...
Disabling the supercapacitor charger...
Delaying WAKEINT (60 seconds).


Artemis Global Tracker
Software Version: 2.0


Ready to accept configuration settings via Serial...


Getting the PHT readings...

[PaulZC]

Hi Andreas,

Thank you for investigating this.

If you have time, it would be good to understand exactly where your code is hanging. Could you please try adding this line in many places in the read_pressure case statement:

Serial.println(F("HERE")); Serial.flush();

The flush is important as it prevents the code from moving on until the println is complete. That way you can tell that the code hung in the preceding line.

The background to this is that v2.1 of the core does things differently compared to v1.2. The Ambiq Micro low-level calls to the Apollo3 Blue hardware are wrapped up in the Mbed OS functions, and things happen differently. The point at which the pins are configured for I2C / SPI is different for example. Maybe there is a gremlin in making repeated calls to agtWire.begin(). You can follow some of our detective work here.

Very best wishes,
Paul

[a-schneider-fmi]

I have inserted several Serial.println lines. In my tests, it hung around the line

barometricSensorOK = barometricSensor.begin(agtWire); // Begin the PHT sensor

that is either the println before that line (i.e. after setAGTWirePullups(1);) or directly after this line was the last to be actually printed out.

[PaulZC]

Excellent - thank you Andreas,
If you have time, it would be very interesting to test this in its simplest form. Please see below for the code.
It would also be interesting to see if updating your bootloader makes any difference:
Please try running the Artemis Firmware Upload GUI and click the "Update Bootloader" button
https://github.com/sparkfun/Artemis-Firmware-Upload-GUI

// Artemis Global Tracker pin definitions
#define spiCS1              4  // D4 can be used as an SPI chip select or as a general purpose IO pin
#define geofencePin         10 // Input for the ZOE-M8Q's PIO14 (geofence) pin
#define busVoltagePin       13 // Bus voltage divided by 3 (Analog in)
#define iridiumSleep        17 // Iridium 9603N ON/OFF (sleep) pin: pull high to enable the 9603N
#define iridiumNA           18 // Input for the Iridium 9603N Network Available
#define LED                 19 // White LED
#define iridiumPwrEN        22 // ADM4210 ON: pull high to enable power for the Iridium 9603N
#define gnssEN              26 // GNSS Enable: pull low to enable power for the GNSS (via Q2)
#define superCapChgEN       27 // LTC3225 super capacitor charger: pull high to enable the super capacitor charger
#define superCapPGOOD       28 // Input for the LTC3225 super capacitor charger PGOOD signal
#define busVoltageMonEN     34 // Bus voltage monitor enable: pull high to enable bus voltage monitoring (via Q4 and Q3)
#define spiCS2              35 // D35 can be used as an SPI chip select or as a general purpose IO pin
#define iridiumRI           41 // Input for the Iridium 9603N Ring Indicator

#include <Wire.h> // Needed for I2C
const byte PIN_AGTWIRE_SCL = 8;
const byte PIN_AGTWIRE_SDA = 9;
TwoWire agtWire(PIN_AGTWIRE_SDA, PIN_AGTWIRE_SCL); //Create an I2C port using pads 8 (SCL) and 9 (SDA)

#include <SparkFun_PHT_MS8607_Arduino_Library.h> //http://librarymanager/All#SparkFun_MS8607
MS8607 barometricSensor; //Create an instance of the MS8607 object

void gnssON(void) // Enable power for the GNSS
{
  am_hal_gpio_pincfg_t pinCfg = g_AM_HAL_GPIO_OUTPUT; // Begin by making the gnssEN pin an open-drain output
  pinCfg.eGPOutcfg = AM_HAL_GPIO_PIN_OUTCFG_OPENDRAIN;
  pin_config(PinName(gnssEN), pinCfg);
  delay(1);
  
  digitalWrite(gnssEN, LOW); // Enable GNSS power (HIGH = disable; LOW = enable)
}

void gnssOFF(void) // Disable power for the GNSS
{
  am_hal_gpio_pincfg_t pinCfg = g_AM_HAL_GPIO_OUTPUT; // Begin by making the gnssEN pin an open-drain output
  pinCfg.eGPOutcfg = AM_HAL_GPIO_PIN_OUTCFG_OPENDRAIN;
  pin_config(PinName(gnssEN), pinCfg);
  delay(1);
  
  digitalWrite(gnssEN, HIGH); // Disable GNSS power (HIGH = disable; LOW = enable)
}

void setup()
{
  // Let's begin by setting up the I/O pins
   
  pinMode(LED, OUTPUT); // Make the LED pin an output

  gnssOFF(); // Disable power for the GNSS
  pinMode(geofencePin, INPUT); // Configure the geofence pin as an input

  pinMode(iridiumPwrEN, OUTPUT); // Configure the Iridium Power Pin (connected to the ADM4210 ON pin)
  digitalWrite(iridiumPwrEN, LOW); // Disable Iridium Power (HIGH = enable; LOW = disable)
  pinMode(superCapChgEN, OUTPUT); // Configure the super capacitor charger enable pin (connected to LTC3225 !SHDN)
  digitalWrite(superCapChgEN, LOW); // Disable the super capacitor charger (HIGH = enable; LOW = disable)
  pinMode(iridiumSleep, OUTPUT); // Iridium 9603N On/Off (Sleep) pin
  digitalWrite(iridiumSleep, LOW); // Put the Iridium 9603N to sleep (HIGH = on; LOW = off/sleep)
  pinMode(iridiumRI, INPUT); // Configure the Iridium Ring Indicator as an input
  pinMode(iridiumNA, INPUT); // Configure the Iridium Network Available as an input
  pinMode(superCapPGOOD, INPUT); // Configure the super capacitor charger PGOOD input

  pinMode(busVoltageMonEN, OUTPUT); // Make the Bus Voltage Monitor Enable an output
  digitalWrite(busVoltageMonEN, LOW); // Set it low to disable the measurement to save power
  analogReadResolution(14); //Set resolution to 14 bit

  Serial.begin(115200);
  Serial.println(F("Artemis Global Tracker - agtWire Test"));

}

void loop()
{
 
  agtWire.begin(); // Set up the I2C pins
  Serial.println(F("agtWire begun...")); Serial.flush();
  agtWire.setClock(100000); // Use 100kHz for best performance
  Serial.println(F("agtWire clock speed set...")); Serial.flush();
  setAGTWirePullups(1); // MS8607 needs pull-ups
  Serial.println(F("agtWire pull-ups set...")); Serial.flush();
  
  bool barometricSensorOK;
  barometricSensorOK = barometricSensor.begin(agtWire); // Begin the PHT sensor
  Serial.print(F("barometricSensor.begin returned ")); Serial.println(barometricSensorOK); Serial.flush();
  if (barometricSensorOK == false)
  {
    // Send a warning message if we were unable to connect to the MS8607:
    Serial.println(F("*** Could not detect the MS8607 sensor. Trying again... ***")); Serial.flush();
    barometricSensorOK = barometricSensor.begin(agtWire); // Re-begin the PHT sensor
    if (barometricSensorOK == false)
    {
      // Send a warning message if we were unable to connect to the MS8607:
      Serial.println(F("*** MS8607 sensor not detected at default I2C address ***")); Serial.flush();
    }
  }
  
  if (barometricSensorOK == true) // If the sensor is OK
  {
    float agtPressure = barometricSensor.getPressure();
    Serial.print(F("Pressure (mbar): "));
    Serial.print(agtPressure);
    Serial.flush();
    float agtTemperature = barometricSensor.getTemperature();
    Serial.print(F(" Temperature (C): "));
    Serial.print(agtTemperature);
    Serial.flush();
    float agtHumidity = barometricSensor.getHumidity();
    Serial.print(F(" Humidity (%RH): "));
    Serial.println(agtHumidity);
    Serial.flush();
  }

  delay(60000);
}

void setAGTWirePullups(uint32_t i2cBusPullUps)
{
  //Change SCL and SDA pull-ups manually using pin_config
  am_hal_gpio_pincfg_t sclPinCfg = g_AM_BSP_GPIO_IOM1_SCL;
  am_hal_gpio_pincfg_t sdaPinCfg = g_AM_BSP_GPIO_IOM1_SDA;

  if (i2cBusPullUps == 0)
  {
    sclPinCfg.ePullup = AM_HAL_GPIO_PIN_PULLUP_NONE; // No pull-ups
    sdaPinCfg.ePullup = AM_HAL_GPIO_PIN_PULLUP_NONE;
  }
  else if (i2cBusPullUps == 1)
  {
    sclPinCfg.ePullup = AM_HAL_GPIO_PIN_PULLUP_1_5K; // Use 1K5 pull-ups
    sdaPinCfg.ePullup = AM_HAL_GPIO_PIN_PULLUP_1_5K;
  }
  else if (i2cBusPullUps == 6)
  {
    sclPinCfg.ePullup = AM_HAL_GPIO_PIN_PULLUP_6K; // Use 6K pull-ups
    sdaPinCfg.ePullup = AM_HAL_GPIO_PIN_PULLUP_6K;
  }
  else if (i2cBusPullUps == 12)
  {
    sclPinCfg.ePullup = AM_HAL_GPIO_PIN_PULLUP_12K; // Use 12K pull-ups
    sdaPinCfg.ePullup = AM_HAL_GPIO_PIN_PULLUP_12K;
  }
  else
  {
    sclPinCfg.ePullup = AM_HAL_GPIO_PIN_PULLUP_24K; // Use 24K pull-ups
    sdaPinCfg.ePullup = AM_HAL_GPIO_PIN_PULLUP_24K;
  }

  pin_config(PinName(PIN_AGTWIRE_SCL), sclPinCfg);
  pin_config(PinName(PIN_AGTWIRE_SDA), sdaPinCfg);
}

[a-schneider-fmi]

Hello Paul,
Thank you for distilling this minimal example. In my tests it doesn't hang, so unfortunately it doesn't get us closer to finding the cause for the issue.
I have already updated the bootloader, but that doesn't change anything.
Andreas

[PaulZC]

Thank you Andreas,

I will keep thinking about this - but I'm really struggling to think what the cause could be. I wish I could replicate it here - but my board is operating nicely. Would you be willing to let us replace your board if that is the only way forward? Please feel free to contact me privately about this - my email address is on my profile page.

Best wishes,
Paul

[a-schneider-fmi]

In order to test whether this may be caused by some installation issue, a colleague of mine flashed example 16 to the board with his independent installation (Arduino 1.8.12 under Windows 10, while I use Arduino 1.8.15 under Ubuntu Linux). Therewith, it hangs at a different place:

Artemis Global Tracker
Software Version: 2.0
Ready to accept configuration settings via Serial...
Getting the PHT readings...
Pressure (mbar): 1005
Temperature (C * 10^-2): 2504
Humidity (%RH * 10^-2): 2507
Powering up the GNSS...
Dynamic Model updated and saved to BBR
Clearing any existing geofences. clearGeofences returned: 1
NAVCONF saved to BBR
Waiting for a 3D GNSS fix...
A 3D fix was found!
Latitude (degrees * 10^-7): 673665821
Longitude (degrees * 10^-7): 266296369
Altitude (mm): 169537
Powering down the GNSS...
Enabling the supercapacitor charger...
Waiting for supercapacitors to charge...
Supercapacitors charged!
Giving the supercapacitors extra time to charge...
Supercapacitors charged!
Enabling 9603N power...
Beginning to talk to the 9603...

So this issue may be related to #36 yet.

[PaulZC]

Thank you Andreas,

I think I have a possible explanation for this... Please see: sparkfun/Arduino_Apollo3#423

When the Iridium modem is powered down, it is likely to pull the Serial1 RX pin low. As reported above, this can cause the code to hang. (With v1.2.1 of the core, this does not happen.)

I will of course try to produce a fix for this as soon as possible.

Very best wishes,
Paul

[PaulZC]

Hi Andreas (@a-schneider-fmi ),

I have updated the examples to include a fix for the Apollo3 Serial1 RX pin issue. Please give them a try. As usual, there are pre-compiled binaries you can upload. Or, you can compile the code yourself: please use v2.1.0 of the Apollo3 core; please update the IridiumSBDi2c library to v3.0.5.

Please let me know if this resolves your issue.

Very best wishes,
Paul

[PaulZC]

Hi Andreas,
Please use v3.0.5 of the IridiumSBDi2c library. It includes a small change to the timing of beginSerialPort.
Best wishes,
Paul

[a-schneider-fmi]

Hi Paul,
Thank you very much for the fix. In the tests I have made so far with the updated version, the tracker did not hang any more. I'll continue with more extensive tests (like a flight test) later.
Best wishes
Andreas

[PaulZC]

Thank you Andreas - that's great news,
Let's leave this issue open until you have completed your tests.
Very best wishes,
Paul

[a-schneider-fmi]

When I switch back on my custom code which uses the MCP23008 library to switch some GPIOs connected by I2C, I now get a MbedOS Fault in the user function upon the second execution which I haven't seen before:

++ MbedOS Fault Handler ++

FaultType: HardFault

Context:
R0: 1000B540
R1: 40
R2: 10001D14
R3: A3009150
R4: 1000B540
R5: 0
R6: 10001BEC
R7: 1
R8: 0
R9: 0
R10: 0
R11: 0
R12: 14
SP   : 10007CC0
LR   : 1D12D
PC   : A3009150
xPSR : 0
PSP  : 10007C58
MSP  : 1005FF70
CPUID: 410FC241
HFSR : 40000000
MMFSR: 1
BFSR : 0
UFSR : 0
DFSR : 0
AFSR : 0
Mode : Thread
Priv : Privileged
Stack: PSP

-- MbedOS Fault Handler --



++ MbedOS Error Info ++
Error Status: 0x80FF013D Code: 317 Module: 255
Error Message: Fault exception
Location: 0xA3009150
Error Value: 0x10006D98
Current Thread: main Id: 0x10005174 Entry: 0x2F361 StackSize: 0x1000 StackMem: 0x10006DF8 SP: 0x10007CC0 
For more info, visit: https://mbed.com/s/error?error=0x80FF013D&tgt=SFE_ARTEMIS_ATP
-- MbedOS Error Info --

Does this mean the MCP23008 library is not compatible (any more)? I have already inserted some Wire.end(); directives, but maybe that is not enough?

[PaulZC]

Hi Andreas,

Ah. I was worried this might happen. I guess you only see the hard fault after the AGT has woken up from deep sleep? (Is that your "second execution"?)

You will need to replicate the code which (re)configures agtWire. Formerly this was Wire1.

I'm guessing you are now using Wire to communicate with your MCP23008? Or maybe you are using qwiic as defined in Example4_ExternalPHT?

You should not need to change the pull-ups, so I think you do not need to fully replicate setAGTWirePullups but see below for details on re-enabling the pins after sleep.

The important pieces to replicate are:

1. Do not call .end()

Artemis_Global_Tracker/Software/examples/Example16_GlobalTracker/Example16_GlobalTracker.ino

Line 2386 in d3d7382

//agtWire.end(); //DO NOT Power down I2C - causes badness with v2.1 of the core: https://github.com/sparkfun/Arduino_Apollo3/issues/412

2. Do not disable the IOM:

Artemis_Global_Tracker/Software/examples/Example16_GlobalTracker/Example16_GlobalTracker.ino

Line 2408 in d3d7382

//am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM1); // agtWire I2C

which means you need to comment this line:

Artemis_Global_Tracker/Software/examples/Example16_GlobalTracker/Example16_GlobalTracker.ino

Line 2411 in d3d7382

am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM4); // Qwiic I2C

3. I suggest you disable pins 39 & 40 to prevent a current leak through those pins. You will need to re-enable them afterwards.

Add 39 and 40 to this list:

Artemis_Global_Tracker/Software/examples/Example16_GlobalTracker/Example16_GlobalTracker.ino

Lines 2417 to 2418 in d3d7382

	// Disable all unused pins - including: SCL (8), SDA (9), UART0 TX (48) and RX (49) and UART1 TX (24) and RX (25)
	const int pinsToDisable[] = {0,1,2,8,9,11,12,14,15,16,20,21,24,25,29,31,32,33,36,37,38,42,43,44,45,48,49,-1};

The code to re-enable the pins afterwards will be similar to setAGTWirePullups. Please try:

  //Restore Qwiic SCL (39) and SDA (40) manually using pin_config
  pin_config(PinName(39), g_AM_BSP_GPIO_IOM4_SCL);
  pin_config(PinName(40), g_AM_BSP_GPIO_IOM4_SDA);

I suggest you do this in wakeUp just after the UART0 pins are re-enabled. Or, you might need to do it just after your qwiic.begin()?

I hope this solves your problem!
Best wishes,
Paul

[a-schneider-fmi]

Hi Paul,
Thank you very much for the instructions. The MCP23008 library used the standard Wire object for communication, but I have now patched it to accept a pointer to a custom TwoWire object in its constructor. Then I have modified the AGT firmware to use qwiic as you described, and hand a pointer to it to the MCP constructor in the user function. With these modifications, the program works and didn't hang in my first tests. The patch to Example16_GlobalTracker.ino is:

--- a/Software/examples/Example16_GlobalTracker/Example16_GlobalTracker.ino
+++ b/Software/examples/Example16_GlobalTracker/Example16_GlobalTracker.ino
@@ -69,6 +69,8 @@
 
 */
 
+#define USE_QWIIC
+
 // Artemis Global Tracker pin definitions
 #define spiCS1              4  // D4 can be used as an SPI chip select or as a general purpose IO pin
 #define geofencePin         10 // Input for the ZOE-M8Q's PIO14 (geofence) pin
@@ -111,6 +113,11 @@ IridiumSBD modem(Serial1, iridiumSleep, iridiumRI);
 const byte PIN_AGTWIRE_SCL = 8;
 const byte PIN_AGTWIRE_SDA = 9;
 TwoWire agtWire(PIN_AGTWIRE_SDA, PIN_AGTWIRE_SCL); //Create an I2C port using pads 8 (SCL) and 9 (SDA)
+#ifdef USE_QWIIC
+const byte PIN_QWIIC_SCL = 39;
+const byte PIN_QWIIC_SDA = 40;
+TwoWire qwiic(PIN_QWIIC_SDA, PIN_QWIIC_SCL); //Create an I2C port using Artemis pads 39 (SCL) and 40 (SDA) broken out to QWIIC connector
+#endif
 
 #include <SparkFun_PHT_MS8607_Arduino_Library.h> //http://librarymanager/All#SparkFun_MS8607
 MS8607 barometricSensor; //Create an instance of the MS8607 object
@@ -414,7 +421,11 @@ void loop()
       agtWire.begin(); // Set up the I2C pins
       agtWire.setClock(100000); // Use 100kHz for best performance
       setAGTWirePullups(1); // MS8607 needs pull-ups
-      
+      #ifdef USE_QWIIC
+      qwiic.begin(); // Set up QWIIC I2C
+      setQwiicPins();
+      #endif
+
       bool barometricSensorOK;
       barometricSensorOK = barometricSensor.begin(agtWire); // Begin the PHT sensor
       if (barometricSensorOK == false)
@@ -2408,14 +2419,16 @@ void loop()
       //am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM1); // agtWire I2C
       am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM2);
       am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM3);
-      am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM4); // Qwiic I2C
+      #ifdef USE_QWIIC
+      //am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM4); // Qwiic I2C
+      #endif
       am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_IOM5);
       am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_ADC);
       //am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_UART0); // Leave UART0 on to avoid printing erroneous characters to Serial
       am_hal_pwrctrl_periph_disable(AM_HAL_PWRCTRL_PERIPH_UART1); // Serial1
     
       // Disable all unused pins - including: SCL (8), SDA (9), UART0 TX (48) and RX (49) and UART1 TX (24) and RX (25)
-      const int pinsToDisable[] = {0,1,2,8,9,11,12,14,15,16,20,21,24,25,29,31,32,33,36,37,38,42,43,44,45,48,49,-1};
+      const int pinsToDisable[] = {0,1,2,8,9,11,12,14,15,16,20,21,24,25,29,31,32,33,36,37,38,39,40,42,43,44,45,48,49,-1};
       for (int x = 0; pinsToDisable[x] >= 0; x++)
       {
         pin_config(PinName(pinsToDisable[x]), g_AM_HAL_GPIO_DISABLE);
@@ -2620,3 +2633,12 @@ void setAGTWirePullups(uint32_t i2cBusPullUps)
   pin_config(PinName(PIN_AGTWIRE_SCL), sclPinCfg);
   pin_config(PinName(PIN_AGTWIRE_SDA), sdaPinCfg);
 }
+
+#ifdef USE_QWIIC
+void setQwiicPins()
+{
+  //Restore QWIIC SCL (39) and SDA (40) manually using pin_config
+  pin_config(PinName(PIN_QWIIC_SCL), g_AM_BSP_GPIO_IOM4_SCL);
+  pin_config(PinName(PIN_QWIIC_SDA), g_AM_BSP_GPIO_IOM4_SDA);
+}
+#endif

The final version will use the broken-out I2C pins and thus agtWire instead of QWIIC.
Best wishes
Andreas

[PaulZC]

Thank you Andreas - that is music to my ears...!
I think I will add a new, separate example to capture this.
Can we close this issue now?
Very best wishes,
Paul

[a-schneider-fmi]

I will do some more tests and notify you then.

[a-schneider-fmi]

Now my tracker hangs after the first Starting modem..., i.e. I experience a manifestation of #36 ...

[PaulZC]

Hi Andreas,
Please narrow this down for me.
Does this happen with Example7 and Example21?
Does this happen with an unmodified copy of Example16 - or just your modified version?
Please make sure the only place Serial1.begin takes place is inside IridiumSBD::beginSerialPort.
Thank you for your patience and understanding,
Paul

[a-schneider-fmi]

The tracker indeed hangs with the nearly unmodified Example 16 (code including #39 but otherwise unmodified) and with Example 7 and Example 21. In my tests, the AGT hung on starting the modem when it had cold temperatures. Background is that I put it into the freezer to test #39 . The handful of times I did this test it always hung when cold and never hung after letting it warm up for an hour or so, for all three examples mentioned.

[a-schneider-fmi]

I should add that if I build and flash the old firmware using v1 of the Artemis core (but including #39 ) the tracker does not hang when cold from the freezer, but sends messages with negative temperatures.

[PaulZC]

Thank you Andreas,
I'm struggling to think why this issue with v2.1 is temperature-dependent. I have ordered a small freezer so I can try to replicate. Maybe the modem takes longer to start when cold and so holds the RX pin low for longer? Very strange!
Thank you again for your support and patience,
Paul

[a-schneider-fmi]

Maybe it's not directly caused be the low temperature, but the low temperature is correlated to the effect causing it.

[PaulZC]

Just to document how things are with the board I have here:

Running Example21 at room temperature: the yellow trace is the EXT_PWR power rail for the Iridium modem (5.3V switched by U3 and Q1); the purple trace is the Iridium modem RX1 line (RX(OUT)). From initial power-up: note that the RX1 line is already high - it goes high as soon as the code starts; note also the small glitch on the RX1 line 1.0s after the 5.3V rail goes high coinciding with the modem.begin; note also that serial characters are seen on RX1 0.5s after modem.begin is called

On the second and subsequent loop: note that RX1 is low (disabled) until the modem.begin

Here the yellow trace is ARTEMIS_D17 (modem ON/!OFF). First loop:

Second and subsequent loops:

[PaulZC]

Thinking more about the glitch on RX1 on the first loop. Is the glitch wide enough to cause the Artemis to hang?

Here it is with a shorter timebase: yellow is ARTEMIS_D17, purple is RX1

Inconclusive... It is ~500us wide and does not go fully low.

A thought experiment...

The Artemis could hang: IF Serial1 is enabled in v2.1 of the core as soon as code execution starts, and IF the glitch on RX1 corresponds to >9.5 bit periods at whatever the default baud rate happens to be.

JR (@jerabaul29) reported similar issues in Apollo3 Issue 349

If the width or depth of the glitch is temperature-dependent, that could explain this issue.

(More to follow shortly.)

[jerabaul29]

That was interesting...

I am curious of why these issues happen with the core v2 but not the core v1. Following several issues with the core v2 (including the one you linked to), I decided to stick to the core v1 instead, and things work like a charm then...

[PaulZC]

Hi Andreas (@a-schneider-fmi ),

It is easy to test if the glitch is the cause of this issue, by making sure that the RX1 pin is disabled before calling modem.begin for the first time.

If I add:

am_hal_gpio_pinconfig(PinName(D25), g_AM_HAL_GPIO_DISABLE); // Disable the RX1 pin (until modem.begin)

to the setup code in Example21:

void setup()
{
  pinMode(LED, OUTPUT); // Make the LED pin an output

  gnssOFF(); // Disable power for the GNSS
  pinMode(geofencePin, INPUT); // Configure the geofence pin as an input

  pinMode(iridiumPwrEN, OUTPUT); // Configure the Iridium Power Pin (connected to the ADM4210 ON pin)
  digitalWrite(iridiumPwrEN, LOW); // Disable Iridium Power (HIGH = enable; LOW = disable)
  pinMode(superCapChgEN, OUTPUT); // Configure the super capacitor charger enable pin (connected to LTC3225 !SHDN)
  digitalWrite(superCapChgEN, LOW); // Disable the super capacitor charger (HIGH = enable; LOW = disable)
  pinMode(iridiumSleep, OUTPUT); // Iridium 9603N On/Off (Sleep) pin
  digitalWrite(iridiumSleep, LOW); // Put the Iridium 9603N to sleep (HIGH = on; LOW = off/sleep)
  pinMode(iridiumRI, INPUT); // Configure the Iridium Ring Indicator as an input
  pinMode(iridiumNA, INPUT); // Configure the Iridium Network Available as an input
  pinMode(superCapPGOOD, INPUT); // Configure the super capacitor charger PGOOD input

  am_hal_gpio_pinconfig(PinName(D25), g_AM_HAL_GPIO_DISABLE); // Disable the RX1 pin (until modem.begin)

  // Start the console serial port

then I can guarantee RX1 is disabled (low) the first time loop is executed:

The slow rise of RX1 is interesting. It implies that the circuit inside the Iridium modem which drives the RX(OUT) pin has a slow response when ON/!OFF is pulled high. Perhaps while the internal microcontroller is starting up.

If you have time, please do try adding am_hal_gpio_pinconfig(PinName(D25), g_AM_HAL_GPIO_DISABLE); to your setup code.

Best wishes,
Paul

[a-schneider-fmi]

Hi Paul,

Thank you for your investigations. I have tested the modified Example 21 (i.e. with RX1 pin disabled as you wrote) with the deep-frozen tracker twice, and it doesn't hang.

Best regards

Andreas

[PaulZC]

Thank you Andreas - that is great news!

Let's leave this issue open until I have completed my own freezer tests.

Very best wishes,
Paul

[PaulZC]

Just for information, here's the start of my own cold test:

The freezer is a Russell Hobbs RHTTFZ1.
I've cut a 25mm hole through the door for the cable feed-through.
Silicone grommets protect the sharp edges.
Sponge fills the gap around the cables through the door, between the grommets.
Polystyrene packing fills the voids at the back and bottom of the freezer to reduce the volume and improve insulation.
The evaporators appear to be behind the left, right and top surfaces towards the front of the freezer, adjacent to the door. Routing the cables through the door was the safest option.
The AGT is double-bagged with desiccant sachets in each bag.

Start of the test: 11AM local. Example3 reports the following (the desiccant is clearly working!):

Example9 shows the antenna cable is working:

Here's the initial glitch on RX1 - same as above. The code is not calling am_hal_gpio_pinconfig(PinName(D25), g_AM_HAL_GPIO_DISABLE); in setup:

My thesis is that the glitch will become wider or deeper as the temperature falls.

More to follow...

[PaulZC]

No obvious change at ~ -20C :

[a-schneider-fmi]

In my tests, I took the cold AGT out of the freezer and put it in front of a window in order to have radio reception while making the actual tests. Making a hole in the freezer was not an option. Thus, the effects I saw may also be related to humidity or warming effects.

[PaulZC]

Thank you Andreas - understood.

There are no obvious signs of anything having changed after a few hours at ~ -22C. Example21 is still running successfully:

I'm not worried that I cannot replicate the behavior you observed. I believe we understand the core issue - and have a work-around for it. Meanwhile, my colleague is working on a proper fix for this issue. I will leave this open until I have had the opportunity to test Razvan's (@razvandragomirescu , #36 ) board.

Have a nice weekend,
Paul

[PaulZC]

After ~70 hours at ~ -22C, Example21 is still working OK for me. The glitch on the RX1 line does not appear to have changed significantly, but is perhaps 0.2V deeper / lower.

So no clear "smoking gun" here...

I will test Razvan's (@razvandragomirescu , #36 ) board as soon as it arrives.

Best wishes,
Paul

[PaulZC]

Hi Andreas (@a-schneider-fmi ),
We tested Razvan's board yesterday using three versions of Example21: one based on v1.2.1 of Apollo3; one based on v2.1.0 of Apollo3 - without commit c6f3dbc; one based on v2.1.0 of Apollo3 - with commit c6f3dbc.
Versions 1 and 3 both worked, version 2 stalled at "Starting modem".
So, I do believe we understand the root cause of this issue - and we have a successful workaround. The proper fix will be included in v2.2.0 of Apollo3.
Closing...!
Very best wishes,
Paul