Release v2.3.0

README.md

@@ -43,52 +43,25 @@ On January 28th, 2019, Tuya started [distributing a patch](https://www.heise.de/
     # git clone https://github.com/ct-Open-Source/tuya-convert
     # cd tuya-convert
     # ./install_prereq.sh
-### flash loader firmware + backup
-    # ./start_flash.sh
-
-Follow the instructions in the start_flash script. It will install our flash loader onto the ESP and connect to the access point created by your wifi adapter.
-
-    WIFI: vtrust-flash
-    IP: 10.42.42.42
-A backup of the original firmware will be created and stored locally
-
-### Device information
-After the firmware backup procedure, the retrieved device information will be shown.
-Please make sure to write down your devices flash mode and size!
-You can show this information again by executing:
-
-    # curl http://10.42.42.42
-### BACKUP only and UNDO
-You can use the flash loader to create a backup only.
-If you want to delete the FLASH loader out of the flash again and go back to the stock software just do following:
-
-    # curl http://10.42.42.42/undo
-### FLASH loader to user2
-The FLASH loader only allows flashing the third party firmware if the loader is running in the userspace user2 starting from 0x81000.
-This will flash the FLASH loader in user2 if it is not already there.
-It will destroy your ability to undo and go back to the original firmware
-
-    # curl http://10.42.42.42/flash2
 
 ### FLASH third-party firmware
 BE SURE THE FIRMWARE FITS YOUR DEVICE!
-1. Place or link your binary file to ./files/thirdparty.bin.
+1. Place your binary file in the `/files/` directory or use one of the included firmware images.
 
 	Currently a Tasmota [v7.0.0.3](https://github.com/arendst/Tasmota/releases) `tasmota-wifiman.bin` build is included. You can update to a [current version](http://thehackbox.org/tasmota) via OTA after the Tuya-Convert process completes successfully. Please note that while we include this for your convenience, we are not affiliated with the Tasmota project and cannot provide support for post installation issues. Please refer to [the respective project](https://github.com/arendst/Tasmota) for configuration and support.
-	
+
 	An ESPurna [1.13.5](https://github.com/xoseperez/espurna/releases/tag/1.13.5) binary is also included (`espurna-base.bin`). Like before, the binary included does not have any specific hardware defined. Once flashed using Tuya-Convert you can update to the device-specific version via any of the means that ESPurna provides (OTA, web interface update, update via telnet or MQTT). Please refer to the [ESPurna project page](http://espurna.io) for more info and support.
 
 	Binary requirements:
-	* full binary including first-stage bootloader
+	* full binary including first-stage bootloader (tested with Arduino eboot and Open-RTOS rBoot)
 	* maximum filesize 512KB for first flash
 
 2. Start flashing process
 
-        # curl http://10.42.42.42/flash3
-
-	Alternatively you can request a certain file to be requested and flashed by the device:
-
-        # curl http://10.42.42.42/flash3?url=http://10.42.42.1/files/certain_file.bin
+	Execute `./start_flash.sh` and follow the instructions.
+	It will install our flash loader onto the ESP and connect to the access point created by your wifi adapter.
+	A backup of the original firmware will be automatically downloaded and stored locally.
+	You can then proceed to flash your desired firmware or revert to the stock firmware.
 
 3. Initial Configuration
 

scripts/fake-registration-server.py

@@ -113,7 +113,7 @@ def post(self):
         print(self.request.headers)
         if payload:
             try:
-                decrypted_payload = unpad(AES.new(options.secKey, AES.MODE_ECB).decrypt(binascii.unhexlify(payload))).decode()
+                decrypted_payload = unpad(AES.new(options.secKey.encode(), AES.MODE_ECB).decrypt(binascii.unhexlify(payload))).decode()
                 if decrypted_payload[0] != "{":
                     raise ValueError("payload is not JSON")
                 print("payload", decrypted_payload)

scripts/firmware_picker.sh

@@ -0,0 +1,80 @@
+#!/bin/bash
+
+MAGIC=$(printf "\xe9")
+
+while true; do
+	echo
+	echo "Available options:"
+	echo "  0) return to stock"
+	index=0
+	for file in ../files/*.bin; do
+		# skip null glob
+		[[ -e $file ]] || continue
+		# get short name
+		filename=$(basename "$file")
+		# skip files too large or too small
+		filesize=$(stat -c%s "$file")
+		[[ "$filesize" -gt 0x1000 && "$filesize" -le 0x80000 ]] || continue
+		# skip files without magic byte
+		[[ $(head -c 1 "$file") == "$MAGIC" ]] || continue
+		echo "  $((++index))) flash $filename"
+		options[$index]="$filename"
+		# only show first 9 options, accessible with a single keypress
+		if (( index == 9 )); then
+			break
+		fi
+	done
+	echo "  q) quit; do nothing"
+	echo -n "Please select 0-$index: "
+	while true; do
+		read -n 1 -r
+		echo
+		if [[ "$REPLY" =~ ^[0-9]$ && "$REPLY" -ge 0 && "$REPLY" -le $index ]]; then
+			break
+		fi
+		if [[ "$REPLY" =~ ^[Qq]$ ]]; then
+			echo "Leaving device as is..."
+			exit
+		fi
+		echo -n "Invalid selection, please select 0-$index: "
+	done
+
+	if [[ "$REPLY" == 0 ]]; then
+		if curl -s http://10.42.42.42/undo; then
+			echo "Disconnect the device to prevent it from repeating the upgrade"
+			echo "You will need to put the device back into pairing mode and register to use again"
+		else
+			echo "Could not reach the device!"
+		fi
+		break
+	fi
+
+	selection="${options[$REPLY]}"
+	read -p "Are you sure you want to flash $selection? This is the point of no return [y/N] " -n 1 -r
+	echo
+	[[ "$REPLY" =~ ^[Yy]$ ]] || continue
+
+	echo "Attempting to flash $selection, this may take a few seconds..."
+	RESULT=$(curl -s "http://10.42.42.42/flash?url=http://10.42.42.1/files/$selection") ||
+	echo "Could not reach the device!"
+
+	echo "$RESULT"
+	if [[ "$RESULT" =~ failed || -z "$RESULT" ]]; then
+		read -p "Do you want to try something else? [y/N] " -n 1 -r
+		echo
+		[[ "$REPLY" =~ ^[Yy]$ ]] || break
+	else
+		if [[ "$selection" == "tasmota.bin" ]]; then
+			echo "Look for a tasmota-xxxx SSID to which you can connect and configure"
+			echo "Be sure to configure your device for proper function!"
+		elif [[ "$selection" == "espurna.bin" ]]; then
+			echo "Look for an ESPURNA-XXXXXX SSID to which you can connect and configure"
+			echo "Default password is \"fibonacci\""
+			echo "Be sure to upgrade to your device specific firmware for proper function!"
+		fi
+		echo
+		echo "HAVE FUN!"
+		break
+	fi
+done
+

scripts/mq_pub_15.py

@@ -21,8 +21,8 @@
 from Crypto.Cipher import AES
 pad = lambda s: s + (16 - len(s) % 16) * chr(16 - len(s) % 16)
 unpad = lambda s: s[:-ord(s[len(s) - 1:])]
-encrypt = lambda msg, key: AES.new(key, AES.MODE_ECB).encrypt(pad(msg))
-decrypt = lambda msg, key: AES.new(key, AES.MODE_ECB).decrypt(unpad(msg))
+encrypt = lambda msg, key: AES.new(key.encode(), AES.MODE_ECB).encrypt(pad(msg))
+decrypt = lambda msg, key: unpad(AES.new(key.encode(), AES.MODE_ECB).decrypt(msg))
 
 def iot_dec(message, local_key):
 	message_clear = decrypt(base64.b64decode(message[19:]), local_key)

scripts/psk-frontend.py

@@ -9,6 +9,7 @@
 from hashlib import md5
 from binascii import hexlify, unhexlify
 
+IDENTITY_PREFIX = "BAohbmd6aG91IFR1"
 
 def listener(host, port):
     sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
@@ -24,10 +25,15 @@ def client(host, port):
 
 def gen_psk(identity, hint):
     print("ID: %s" % hexlify(identity))
+    # sometimes the device only sends part of the prefix
+    # since it is always the same, we can correct it
+    if identity[1:17] != IDENTITY_PREFIX:
+        print("Prefix: %s" % identity[1:17])
+    identity = IDENTITY_PREFIX + identity[17:]
     key = md5(hint[-16:]).digest()
-    iv = md5(identity[1:]).digest()
+    iv = md5(identity).digest()
     cipher = AES.new(key, AES.MODE_CBC, iv)
-    psk = cipher.encrypt(identity[1:33])
+    psk = cipher.encrypt(identity[:32])
     print("PSK: %s" % hexlify(psk))
     return psk
 

scripts/setup_ap.sh

@@ -32,7 +32,10 @@ setup () {
 		--address=/#/$GATEWAY
 
 	echo "Starting AP on $WLAN..."
-	sudo hostapd hostapd.conf -i $WLAN
+
+	# Read hostapd.conf with interface from stdin for
+	# backward compatibility (hostapd < v2.6). See #398
+	printf "$(cat hostapd.conf)\ninterface=$WLAN" | sudo hostapd /dev/stdin
 }
 
 cleanup () {

scripts/setup_checks.sh

@@ -82,6 +82,19 @@ check_port () {
 	fi
 }
 
+check_blacklist () {
+	if [ -e /etc/modprobe.d/blacklist-rtl8192cu.conf ]; then
+		echo "Detected /etc/modprobe.d/blacklist-rtl8192cu.conf"
+		echo "This has been known to cause kernel panic in hostapd"
+		echo "See https://github.com/ct-Open-Source/tuya-convert/issues/373"
+		read -p "Do you wish to remove this file? [y/N] " -n 1 -r
+		echo
+		if [[ $REPLY =~ ^[Yy]$ ]]; then
+			sudo rm /etc/modprobe.d/blacklist-rtl8192cu.conf
+		fi
+	fi
+}
+
 check_eula
 check_config
 check_port udp 53 "resolve DNS queries"
@@ -92,4 +105,5 @@ check_port udp 6666 "detect unencrypted Tuya firmware"
 check_port udp 6667 "detect encrypted Tuya firmware"
 check_port tcp 1883 "run MQTT"
 check_port tcp 8886 "run MQTTS"
+check_blacklist
 

scripts/tuya-discovery.py

@@ -0,0 +1,53 @@
+#!/usr/bin/env python3
+# encoding: utf-8
+"""
+tuya-discovery.py
+Created by kueblc on 2019-11-13.
+Discover Tuya devices on the LAN via UDP broadcast
+"""
+
+import asyncio
+import json
+
+from Crypto.Cipher import AES
+pad = lambda s: s + (16 - len(s) % 16) * chr(16 - len(s) % 16)
+unpad = lambda s: s[:-ord(s[len(s) - 1:])]
+encrypt = lambda msg, key: AES.new(key, AES.MODE_ECB).encrypt(pad(msg))
+decrypt = lambda msg, key: unpad(AES.new(key, AES.MODE_ECB).decrypt(msg))
+
+from hashlib import md5
+udpkey = md5(b"yGAdlopoPVldABfn").digest()
+decrypt_udp = lambda msg: decrypt(msg, udpkey)
+
+class TuyaDiscovery(asyncio.DatagramProtocol):
+	def datagram_received(self, data, addr):
+		# remove message frame
+		data = data[20:-8]
+		# decrypt if encrypted
+		try:
+			data = decrypt_udp(data)
+		except:
+			pass
+		# parse json
+		try:
+			data = json.loads(data)
+		except:
+			pass
+		print(addr[0], data)
+
+def main():
+	loop = asyncio.get_event_loop()
+	listener = loop.create_datagram_endpoint(TuyaDiscovery, local_addr=('0.0.0.0', 6666))
+	encrypted_listener = loop.create_datagram_endpoint(TuyaDiscovery, local_addr=('0.0.0.0', 6667))
+	loop.run_until_complete(listener)
+	print("Listening for Tuya broadcast on UDP 6666")
+	loop.run_until_complete(encrypted_listener)
+	print("Listening for encrypted Tuya broadcast on UDP 6667")
+	try:
+		loop.run_forever()
+	except KeyboardInterrupt:
+		loop.stop()
+
+if __name__ == "__main__":
+	main()
+

start_flash.sh

@@ -1,10 +1,10 @@
 #!/bin/bash
 bold=$(tput bold)
 normal=$(tput sgr0)
-screen_minor=`screen --version | cut -d . -f 2`
-if [ $screen_minor -gt 5 ]; then
+screen_minor=$(screen --version | cut -d . -f 2)
+if [ "$screen_minor" -gt 5 ]; then
     screen_with_log="sudo screen -L -Logfile"
-elif [ $screen_minor -eq 5 ]; then
+elif [ "$screen_minor" -eq 5 ]; then
     screen_with_log="sudo screen -L"
 else
     screen_with_log="sudo screen -L -t"
@@ -13,14 +13,14 @@ fi
 
 ./stop_flash.sh >/dev/null
 
-pushd scripts >/dev/null
+pushd scripts >/dev/null || exit
 
 . ./setup_checks.sh
 
 echo "======================================================"
 echo -n "  Starting AP in a screen"
 $screen_with_log smarthack-wifi.log -S smarthack-wifi -m -d ./setup_ap.sh
-while ! ping -c 1 -W 1 -n $GATEWAY &> /dev/null; do
+while ! ping -c 1 -W 1 -n "$GATEWAY" &> /dev/null; do
 	printf .
 done
 echo
@@ -31,18 +31,21 @@ echo "  Starting Mosquitto in a screen"
 $screen_with_log smarthack-mqtt.log -S smarthack-mqtt -m -d mosquitto -v
 echo "  Starting PSK frontend in a screen"
 $screen_with_log smarthack-psk.log -S smarthack-psk -m -d ./psk-frontend.py -v
+echo "  Starting Tuya Discovery in a screen"
+$screen_with_log smarthack-udp.log -S smarthack-udp -m -d ./tuya-discovery.py
 echo
 REPLY=y
 while [[ $REPLY =~ ^[Yy]$ ]]; do
 echo "======================================================"
 echo
 echo "IMPORTANT"
 echo "1. Connect any other device (a smartphone or something) to the WIFI $AP"
-echo "   This step is IMPORTANT otherwise the smartconfig will not work!"
+echo "   This step is IMPORTANT otherwise the smartconfig may not work!"
 echo "2. Put your IoT device in autoconfig/smartconfig/pairing mode (LED will blink fast). This is usually done by pressing and holding the primary button of the device"
+echo "   Make sure nothing else is plugged into your IoT device while attempting to flash."
 echo "3. Press ${bold}ENTER${normal} to continue"
-read x
-echo ""
+read -r
+echo
 echo "======================================================"
 
 echo "Starting smart config pairing procedure"
@@ -71,45 +74,40 @@ pkill -f smartconfig/main.py && echo "Stopping smart config"
 
 echo "Fetching firmware backup"
 sleep 2
-timestamp=`date +%Y%m%d_%H%M%S`
-mkdir -p "../backups/$timestamp"
-pushd "../backups/$timestamp" >/dev/null
+timestamp=$(date +%Y%m%d_%H%M%S)
+backupfolder="../backups/$timestamp"
+mkdir -p "$backupfolder"
+pushd "$backupfolder" >/dev/null || exit
 curl -JO http://10.42.42.42/backup
 
 echo "======================================================"
 echo "Getting Info from IoT-device"
 curl -s http://10.42.42.42 | tee device-info.txt
-popd >/dev/null
+popd >/dev/null || exit
 
 echo "======================================================"
-echo "Please make sure to note the correct SPI flash mode!"
-echo "Installing an alternative firmware with the wrong flash mode will leave the ESP unable to boot!"
-echo
-echo "Next steps:"
-echo "1. To go back to the orginal software"
-echo "   # curl http://10.42.42.42/undo"
+echo "Ready to flash third party firmware!"
 echo
-echo "2. Be sure the conversion software runs in user2"
-echo "   # curl http://10.42.42.42/flash2"
+echo "For your convenience, the following firmware images are already included in this repository:"
+echo "  Tasmota v7.0.0.3 (wifiman)"
+echo "  ESPurna 1.13.5 (base)"
 echo
-echo "3. Flash a third party firmware to the device"
-echo "BE SURE THE FIRMWARE FITS THE DEVICE AND USES THE CORRECT FLASH MODE!"
+echo "You can also provide your own image by placing it in the /files directory"
+echo "Please ensure the firmware fits the device and includes the bootloader"
 echo "MAXIMUM SIZE IS 512KB"
-echo "put or link it to ./files/thirdparty.bin"
-echo "A build of Tasmota v7.0.0.3 is already included in this repository."
-echo "   # curl http://10.42.42.42/flash3"
-echo "If you want to flash the included ESPurna 1.13.5 image use this command:"
-echo "   # curl http://10.42.42.42/flash3?url=http://10.42.42.1/files/espurna-base.bin"
-echo "Alternatively let the device download and flash a file via HTTP:"
-echo "   # curl http://10.42.42.42/flash3?url=http://10.42.42.1/files/thirdparty.bin"
-echo
-echo "HAVE FUN!"
+
+./firmware_picker.sh
+
 echo "======================================================"
 read -p "Do you want to flash another device? [y/N] " -n 1 -r
 echo
+
+sudo mv *.log "$backupfolder/"
 done
 
 echo "Exiting..."
 
-popd >/dev/null
+popd >/dev/null || exit
+
+./stop_flash.sh >/dev/null