Example 16 not waking from deep sleep correctly after upgrade to v2.1

[PaulZC]

Original issue was raised by @a-schneider-fmi in #36:

Since the update to version 2, I experience another issue which may or may not be related: the tracker seems not to wake from deep sleep after 2 messages have been sent. From the configuration tool, I see the following messages:

[...]
>>> Message sent! <<<
Clearing the MO buffer.
The number of messages in the MT queue is: 0
Putting the 9603N to sleep.
Getting ready to put the Apollo3 into deep sleep...
Disabling 9603N power...
Disabling the supercapacitor charger...
Powering down the GNSS...
Going into deep sleep until next WAKEINT (60 seconds).

And then it hangs. This happens with several TXINT settings (I tried 2 minutes, 5 minutes, 10 minutes).

I have tested example 12 to debug. It seems to work, at least it runs for more than half an hour. Do you have any suggestion how to investigate further?

[a-schneider-fmi]

I made some more tests with the tracker with different settings, and after some settings change it suddenly works again, even when changing the settings back to what they have been before. I have no idea what is different.

[PaulZC]

Hi Andreas,

Restoring the Artemis peripherals and pins to full function after deep sleep is a lot more complex in v2.1 than it was with v1.2 of the core. Getting that part of the code right on OpenLog Artemis took a lot of effort.

It will be a question of making sure that the AGT deep sleep and wake code is exactly the same as the OpenLog Artemis code.

Example12 is much simpler and doesn't have to worry about restoring (e.g.) Serial1, agtWire and all the pins and interrupts to full functionality after deep sleep.

I will try and investigate this when I get some 'spare' time. (It is in short supply at the moment! ;-) )

Meanwhile if you discover any more clues, please report them here.

Very best wishes,
Paul

[a-schneider-fmi]

This morning a made another test with the same configuration that worked yesterday evening (i.e. no changes, just plugged the device to power), and it stopped after the first message.

[PaulZC]

Hi Andreas,
OK - thank you. I will investigate this as soon as I can.
Very best wishes,
Paul

[a-schneider-fmi]

To test more, I tried to skip the deep sleep phase by adding the following lines in case zzz, line 2251 (i.e. after the modem is powered down):

      // TEST
      Serial.print(F("Delaying WAKEINT ("));
      Serial.print(wake_int);
      Serial.println(F(" seconds)."));
      delay(wake_int*1000);
      loop_step = loop_init;
      break; // TEST

Have I missed something? Now it hangs when reading the PHT sensor after sending two messages.

Transmitting message...
>>> Message sent! <<<
Clearing the MO buffer.
The number of messages in the MT queue is: 0
Putting the 9603N to sleep.
Getting ready to put the Apollo3 into deep sleep...
Disabling 9603N power...
Disabling the supercapacitor charger...
Delaying WAKEINT (60 seconds).


Artemis Global Tracker
Software Version: 2.0


Ready to accept configuration settings via Serial...


Getting the PHT readings...
*** Could not detect the MS8607 sensor. Trying again... ***
Pressure (mbar): 973
Temperature (C * 10^-2): 1980
Humidity (%RH * 10^-2): 5703
Getting ready to put the Apollo3 into deep sleep...
Disabling 9603N power...
Disabling the supercapacitor charger...
Delaying WAKEINT (60 seconds).


Artemis Global Tracker
Software Version: 2.0


Ready to accept configuration settings via Serial...


Getting the PHT readings...
Pressure (mbar): 973
Temperature (C * 10^-2): 1879
Humidity (%RH * 10^-2): 6014
Getting ready to put the Apollo3 into deep sleep...
Disabling 9603N power...
Disabling the supercapacitor charger...
Delaying WAKEINT (60 seconds).


Artemis Global Tracker
Software Version: 2.0


Ready to accept configuration settings via Serial...


Getting the PHT readings...
Pressure (mbar): 973
Temperature (C * 10^-2): 1784
Humidity (%RH * 10^-2): 6414
Getting ready to put the Apollo3 into deep sleep...
Disabling 9603N power...
Disabling the supercapacitor charger...
Delaying WAKEINT (60 seconds).


Artemis Global Tracker
Software Version: 2.0


Ready to accept configuration settings via Serial...


Getting the PHT readings...

[PaulZC]

Hi Andreas,

Thank you for investigating this.

If you have time, it would be good to understand exactly where your code is hanging. Could you please try adding this line in many places in the read_pressure case statement:

Serial.println(F("HERE")); Serial.flush();

The flush is important as it prevents the code from moving on until the println is complete. That way you can tell that the code hung in the preceding line.

The background to this is that v2.1 of the core does things differently compared to v1.2. The Ambiq Micro low-level calls to the Apollo3 Blue hardware are wrapped up in the Mbed OS functions, and things happen differently. The point at which the pins are configured for I2C / SPI is different for example. Maybe there is a gremlin in making repeated calls to agtWire.begin(). You can follow some of our detective work here.

Very best wishes,
Paul

[a-schneider-fmi]

I have inserted several Serial.println lines. In my tests, it hung around the line

barometricSensorOK = barometricSensor.begin(agtWire); // Begin the PHT sensor

that is either the println before that line (i.e. after setAGTWirePullups(1);) or directly after this line was the last to be actually printed out.

[PaulZC]

Excellent - thank you Andreas,
If you have time, it would be very interesting to test this in its simplest form. Please see below for the code.
It would also be interesting to see if updating your bootloader makes any difference:
Please try running the Artemis Firmware Upload GUI and click the "Update Bootloader" button
https://github.com/sparkfun/Artemis-Firmware-Upload-GUI

// Artemis Global Tracker pin definitions
#define spiCS1              4  // D4 can be used as an SPI chip select or as a general purpose IO pin
#define geofencePin         10 // Input for the ZOE-M8Q's PIO14 (geofence) pin
#define busVoltagePin       13 // Bus voltage divided by 3 (Analog in)
#define iridiumSleep        17 // Iridium 9603N ON/OFF (sleep) pin: pull high to enable the 9603N
#define iridiumNA           18 // Input for the Iridium 9603N Network Available
#define LED                 19 // White LED
#define iridiumPwrEN        22 // ADM4210 ON: pull high to enable power for the Iridium 9603N
#define gnssEN              26 // GNSS Enable: pull low to enable power for the GNSS (via Q2)
#define superCapChgEN       27 // LTC3225 super capacitor charger: pull high to enable the super capacitor charger
#define superCapPGOOD       28 // Input for the LTC3225 super capacitor charger PGOOD signal
#define busVoltageMonEN     34 // Bus voltage monitor enable: pull high to enable bus voltage monitoring (via Q4 and Q3)
#define spiCS2              35 // D35 can be used as an SPI chip select or as a general purpose IO pin
#define iridiumRI           41 // Input for the Iridium 9603N Ring Indicator

#include <Wire.h> // Needed for I2C
const byte PIN_AGTWIRE_SCL = 8;
const byte PIN_AGTWIRE_SDA = 9;
TwoWire agtWire(PIN_AGTWIRE_SDA, PIN_AGTWIRE_SCL); //Create an I2C port using pads 8 (SCL) and 9 (SDA)

#include <SparkFun_PHT_MS8607_Arduino_Library.h> //http://librarymanager/All#SparkFun_MS8607
MS8607 barometricSensor; //Create an instance of the MS8607 object

void gnssON(void) // Enable power for the GNSS
{
  am_hal_gpio_pincfg_t pinCfg = g_AM_HAL_GPIO_OUTPUT; // Begin by making the gnssEN pin an open-drain output
  pinCfg.eGPOutcfg = AM_HAL_GPIO_PIN_OUTCFG_OPENDRAIN;
  pin_config(PinName(gnssEN), pinCfg);
  delay(1);
  
  digitalWrite(gnssEN, LOW); // Enable GNSS power (HIGH = disable; LOW = enable)
}

void gnssOFF(void) // Disable power for the GNSS
{
  am_hal_gpio_pincfg_t pinCfg = g_AM_HAL_GPIO_OUTPUT; // Begin by making the gnssEN pin an open-drain output
  pinCfg.eGPOutcfg = AM_HAL_GPIO_PIN_OUTCFG_OPENDRAIN;
  pin_config(PinName(gnssEN), pinCfg);
  delay(1);
  
  digitalWrite(gnssEN, HIGH); // Disable GNSS power (HIGH = disable; LOW = enable)
}

void setup()
{
  // Let's begin by setting up the I/O pins
   
  pinMode(LED, OUTPUT); // Make the LED pin an output

  gnssOFF(); // Disable power for the GNSS
  pinMode(geofencePin, INPUT); // Configure the geofence pin as an input

  pinMode(iridiumPwrEN, OUTPUT); // Configure the Iridium Power Pin (connected to the ADM4210 ON pin)
  digitalWrite(iridiumPwrEN, LOW); // Disable Iridium Power (HIGH = enable; LOW = disable)
  pinMode(superCapChgEN, OUTPUT); // Configure the super capacitor charger enable pin (connected to LTC3225 !SHDN)
  digitalWrite(superCapChgEN, LOW); // Disable the super capacitor charger (HIGH = enable; LOW = disable)
  pinMode(iridiumSleep, OUTPUT); // Iridium 9603N On/Off (Sleep) pin
  digitalWrite(iridiumSleep, LOW); // Put the Iridium 9603N to sleep (HIGH = on; LOW = off/sleep)
  pinMode(iridiumRI, INPUT); // Configure the Iridium Ring Indicator as an input
  pinMode(iridiumNA, INPUT); // Configure the Iridium Network Available as an input
  pinMode(superCapPGOOD, INPUT); // Configure the super capacitor charger PGOOD input

  pinMode(busVoltageMonEN, OUTPUT); // Make the Bus Voltage Monitor Enable an output
  digitalWrite(busVoltageMonEN, LOW); // Set it low to disable the measurement to save power
  analogReadResolution(14); //Set resolution to 14 bit

  Serial.begin(115200);
  Serial.println(F("Artemis Global Tracker - agtWire Test"));

}

void loop()
{
 
  agtWire.begin(); // Set up the I2C pins
  Serial.println(F("agtWire begun...")); Serial.flush();
  agtWire.setClock(100000); // Use 100kHz for best performance
  Serial.println(F("agtWire clock speed set...")); Serial.flush();
  setAGTWirePullups(1); // MS8607 needs pull-ups
  Serial.println(F("agtWire pull-ups set...")); Serial.flush();
  
  bool barometricSensorOK;
  barometricSensorOK = barometricSensor.begin(agtWire); // Begin the PHT sensor
  Serial.print(F("barometricSensor.begin returned ")); Serial.println(barometricSensorOK); Serial.flush();
  if (barometricSensorOK == false)
  {
    // Send a warning message if we were unable to connect to the MS8607:
    Serial.println(F("*** Could not detect the MS8607 sensor. Trying again... ***")); Serial.flush();
    barometricSensorOK = barometricSensor.begin(agtWire); // Re-begin the PHT sensor
    if (barometricSensorOK == false)
    {
      // Send a warning message if we were unable to connect to the MS8607:
      Serial.println(F("*** MS8607 sensor not detected at default I2C address ***")); Serial.flush();
    }
  }
  
  if (barometricSensorOK == true) // If the sensor is OK
  {
    float agtPressure = barometricSensor.getPressure();
    Serial.print(F("Pressure (mbar): "));
    Serial.print(agtPressure);
    Serial.flush();
    float agtTemperature = barometricSensor.getTemperature();
    Serial.print(F(" Temperature (C): "));
    Serial.print(agtTemperature);
    Serial.flush();
    float agtHumidity = barometricSensor.getHumidity();
    Serial.print(F(" Humidity (%RH): "));
    Serial.println(agtHumidity);
    Serial.flush();
  }

  delay(60000);
}

void setAGTWirePullups(uint32_t i2cBusPullUps)
{
  //Change SCL and SDA pull-ups manually using pin_config
  am_hal_gpio_pincfg_t sclPinCfg = g_AM_BSP_GPIO_IOM1_SCL;
  am_hal_gpio_pincfg_t sdaPinCfg = g_AM_BSP_GPIO_IOM1_SDA;

  if (i2cBusPullUps == 0)
  {
    sclPinCfg.ePullup = AM_HAL_GPIO_PIN_PULLUP_NONE; // No pull-ups
    sdaPinCfg.ePullup = AM_HAL_GPIO_PIN_PULLUP_NONE;
  }
  else if (i2cBusPullUps == 1)
  {
    sclPinCfg.ePullup = AM_HAL_GPIO_PIN_PULLUP_1_5K; // Use 1K5 pull-ups
    sdaPinCfg.ePullup = AM_HAL_GPIO_PIN_PULLUP_1_5K;
  }
  else if (i2cBusPullUps == 6)
  {
    sclPinCfg.ePullup = AM_HAL_GPIO_PIN_PULLUP_6K; // Use 6K pull-ups
    sdaPinCfg.ePullup = AM_HAL_GPIO_PIN_PULLUP_6K;
  }
  else if (i2cBusPullUps == 12)
  {
    sclPinCfg.ePullup = AM_HAL_GPIO_PIN_PULLUP_12K; // Use 12K pull-ups
    sdaPinCfg.ePullup = AM_HAL_GPIO_PIN_PULLUP_12K;
  }
  else
  {
    sclPinCfg.ePullup = AM_HAL_GPIO_PIN_PULLUP_24K; // Use 24K pull-ups
    sdaPinCfg.ePullup = AM_HAL_GPIO_PIN_PULLUP_24K;
  }

  pin_config(PinName(PIN_AGTWIRE_SCL), sclPinCfg);
  pin_config(PinName(PIN_AGTWIRE_SDA), sdaPinCfg);
}

[a-schneider-fmi]

Hello Paul,
Thank you for distilling this minimal example. In my tests it doesn't hang, so unfortunately it doesn't get us closer to finding the cause for the issue.
I have already updated the bootloader, but that doesn't change anything.
Andreas

[PaulZC]

Thank you Andreas,

I will keep thinking about this - but I'm really struggling to think what the cause could be. I wish I could replicate it here - but my board is operating nicely. Would you be willing to let us replace your board if that is the only way forward? Please feel free to contact me privately about this - my email address is on my profile page.

Best wishes,
Paul

[a-schneider-fmi]

In order to test whether this may be caused by some installation issue, a colleague of mine flashed example 16 to the board with his independent installation (Arduino 1.8.12 under Windows 10, while I use Arduino 1.8.15 under Ubuntu Linux). Therewith, it hangs at a different place:

Artemis Global Tracker
Software Version: 2.0
Ready to accept configuration settings via Serial...
Getting the PHT readings...
Pressure (mbar): 1005
Temperature (C * 10^-2): 2504
Humidity (%RH * 10^-2): 2507
Powering up the GNSS...
Dynamic Model updated and saved to BBR
Clearing any existing geofences. clearGeofences returned: 1
NAVCONF saved to BBR
Waiting for a 3D GNSS fix...
A 3D fix was found!
Latitude (degrees * 10^-7): 673665821
Longitude (degrees * 10^-7): 266296369
Altitude (mm): 169537
Powering down the GNSS...
Enabling the supercapacitor charger...
Waiting for supercapacitors to charge...
Supercapacitors charged!
Giving the supercapacitors extra time to charge...
Supercapacitors charged!
Enabling 9603N power...
Beginning to talk to the 9603...

So this issue may be related to #36 yet.

[PaulZC]

Thank you Andreas,

I think I have a possible explanation for this... Please see: sparkfun/Arduino_Apollo3#423

When the Iridium modem is powered down, it is likely to pull the Serial1 RX pin low. As reported above, this can cause the code to hang. (With v1.2.1 of the core, this does not happen.)

I will of course try to produce a fix for this as soon as possible.

Very best wishes,
Paul

[PaulZC]

Hi Andreas (@a-schneider-fmi ),

I have updated the examples to include a fix for the Apollo3 Serial1 RX pin issue. Please give them a try. As usual, there are pre-compiled binaries you can upload. Or, you can compile the code yourself: please use v2.1.0 of the Apollo3 core; please update the IridiumSBDi2c library to v3.0.5.

Please let me know if this resolves your issue.

Very best wishes,
Paul

[PaulZC]

Hi Andreas,
Please use v3.0.5 of the IridiumSBDi2c library. It includes a small change to the timing of beginSerialPort.
Best wishes,
Paul