Skip to content

Commit c70a38c

Browse files
authored
[tfjs-react-native] RN alpha release (#1944)
FEATURE
1 parent 49154ef commit c70a38c

16 files changed

+181
-74
lines changed

tfjs-react-native/README.md

+122-19
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,21 @@
11
# Platform Adapter for React Native
22

3-
Status: __Early development__. This is still an unpublished experimental package.
4-
5-
## Adapter Docs
6-
7-
TODO
3+
This package provides a TensorFlow.js platform adapter for react native. It
4+
provides GPU accelerated execution of TensorFlow.js supporting all major modes
5+
of tfjs usage, include:
6+
- Support for both model inference and training
7+
- GPU support with WebGL via expo-gl.
8+
- Support for loading models pretrained models (tfjs-models) from the web.
9+
- IOHandlers to support loading models from asyncStorage and models
10+
that are compiled into the app bundle.
11+
12+
## Status
13+
This package is currently an **alpha release**. We welcome react native developers
14+
to try it and give us feedback.
815

916
## Setting up a React Native app with tfjs-react-native
1017

11-
These instructions assume that you are generally familiar with [react native](https://facebook.github.io/react-native/) developement. This library has only been tested with React Native 0.58.X & 0.59.X. React Native 0.60 is not supported.
18+
These instructions **assume that you are generally familiar with [react native](https://facebook.github.io/react-native/) developement**. This library has only been tested with React Native 0.58.X & 0.59.X. React Native 0.60 is not supported.
1219

1320
### Step 1. Create your react native app.
1421

@@ -28,11 +35,7 @@ Depending on which workflow you used to set up your app you will need to install
2835
- Expo Managed App
2936
- Install and configure [expo-gl](https://github.com/expo/expo/tree/master/packages/expo-gl)
3037

31-
32-
Note if in a _managed_ expo application these libraries should be present and you should be able to skip this step.
33-
34-
Install and configure [react-native-unimodules](https://github.com/unimodules/react-native-unimodules)
35-
Install and configure [expo-gl-cpp](https://github.com/expo/expo/tree/master/packages/expo-gl-cpp) and [expo-gl](https://github.com/expo/expo/tree/master/packages/expo-gl)
38+
> If you are in a _managed_ expo application these libraries should be present and you should be able to skip this step.
3639
3740
> After this point, if you are using XCode to build for ios, you should use a ‘.workspace’ file instead of the ‘.xcodeproj’
3841
@@ -42,7 +45,7 @@ Edit your `metro.config.js` to look like the following. Changes are noted in
4245
the comments below.
4346

4447
```js
45-
// Change 1
48+
// Change 1 (import the blacklist utility)
4649
const blacklist = require('metro-config/src/defaults/blacklist');
4750

4851
module.exports = {
@@ -58,25 +61,125 @@ module.exports = {
5861
// Change 2 (add 'bin' to assetExts)
5962
assetExts: ['bin', 'txt', 'jpg'],
6063
sourceExts: ['js', 'json', 'ts', 'tsx', 'jsx'],
61-
// Change 3
64+
// Change 3 (add platform_node to blacklist)
6265
blacklistRE: blacklist([/platform_node/])
6366
},
6467
};
6568
```
6669

67-
6870
### Step 4: Install TensorFlow.js and tfjs-react-native
6971

7072
- Install @tensorflow/tfjs - `npm install @tensorflow/tfjs`
71-
- Install @tensorflow/tfjs-react-native - coming soon
73+
- Install @tensorflow/tfjs-react-native - `npm install @tensorflow/tfjs-react-native@alpha`
74+
75+
### Step 4.5: (Optional) Install and configure async-storage
76+
77+
- If you want use `asyncStorageIO` (see below) to save and load models, add [async-storage](https://github.com/react-native-community/async-storage) to your project.
7278

7379
### Step 5: Test that it is working
7480

75-
TODO: Add some sample code.
81+
Before using tfjs in a react native app, you need to call `tf.ready()` and wait for it to complete. This is an **async function** so you might want to do this in a `componentDidMount` or before the app is rendered.
82+
83+
The example below uses a flag in the App state to indicate that TensorFlow is ready.
84+
85+
86+
```js
87+
import * as tf from '@tensorflow/tfjs';
88+
import '@tensorflow/tfjs-react-native';
89+
90+
export class App extends React.Component {
91+
constructor(props) {
92+
super(props);
93+
this.state = {
94+
isTfReady: false,
95+
};
96+
}
97+
98+
async componentDidMount() {
99+
// Wait for tf to be ready.
100+
await tf.ready();
101+
// Signal to the app that tensorflow.js can now be used.
102+
this.setState({
103+
isTfReady: true,
104+
});
105+
}
106+
107+
108+
render() {
109+
//
110+
}
111+
}
112+
113+
```
114+
115+
After gathering feedback in the alpha release we will add an example to the [tensorflow/tfjs-examples](https://github.com/tensorflow/tfjs-examples) repository.
116+
117+
For now you can take a look at [`integration_rn59/App.tsx`](integration_rn59/App.tsx) for an example of what using tfjs-react-native looks like.
118+
The [Webcam demo folder](integration_rn70/components/webcam) has an example of a style transfer app.
119+
120+
![style transfer app initial screen](images/rn-styletransfer_1.jpg)
121+
![style transfer app initial screen](images/rn-styletransfer_2.jpg)
122+
![style transfer app initial screen](images/rn-styletransfer_3.jpg)
123+
![style transfer app initial screen](images/rn-styletransfer_4.jpg)
124+
125+
126+
## API Docs
127+
128+
`tfjs-react-native` exports a number of utility functions:
129+
130+
### asyncStorageIO(modelKey: string)
131+
132+
```js
133+
async function asyncStorageExample() {
134+
// Define a model
135+
const model = tf.sequential();
136+
model.add(tf.layers.dense({units: 5, inputShape: [1]}));
137+
model.add(tf.layers.dense({units: 1}));
138+
model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});
139+
140+
// Save the model to async storage
141+
await model.save(asyncStorageIO('custom-model-test'));
142+
// Load the model from async storage
143+
await tf.loadLayersModel(asyncStorageIO('custom-model-test'));
144+
}
145+
```
146+
147+
The `asyncStorageIO` function returns an io handler that can be used to save an load models
148+
to and from AsyncStorage.
149+
150+
### bundleResourceIO(modelArchitecture: Object, modelWeights: number)
151+
152+
```js
153+
const modelJson = require('../path/to/model.json');
154+
const modelWeights = require('../path/to/model_weights.bin');
155+
async function bundleResourceIOExample() {
156+
const model =
157+
await tf.loadLayersModel(bundleResourceIO(modelJson, modelWeights));
158+
159+
const res = model.predict(tf.randomNormal([1, 28, 28, 1])) as tf.Tensor;
160+
}
161+
```
162+
163+
The `bundleResourceIO` function returns an IOHandler that is able to **load** models
164+
that have been bundled with the app (apk or ipa) at compile time. It takes two
165+
parameters.
166+
167+
1. modelArchitecture: This is a JavaScript object (and notably not a string). This is
168+
because metro will automatically resolve `require`'s for JSON file and return parsed
169+
JavaScript objects.
170+
171+
2. modelWeights: This is the numeric id returned by the metro bundler for the binary weights file
172+
via `require`. The IOHandler will be able to load the actual data from the bundle package.
173+
174+
`bundleResourceIO` only supports non sharded models at the moment. It also cannot save models. Though you
175+
can use the asyncStorageIO handler to save to AsyncStorage.
176+
76177

77-
For now take a look at `integration_rn59/App.tsx` for an example of what using tfjs-react-native looks like.
178+
### fetch(path: string, init?: RequestInit, options?: tf.io.RequestDetails)
78179

79-
### Optional Steps
180+
tfjs react native exports a custom fetch function that is able to correctly load binary files into
181+
`arrayBuffer`'s. The first two parameters are the same as regular [`fetch`](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API). The 3rd paramater is an optional custom `options` object, it currently has one option
80182

81-
If you want use the `AsyncStorageHandler` to save and load models, add [async-storage](https://github.com/react-native-community/async-storage) to your project.
183+
- options.isBinary: A boolean indicating if this is request for a binary file.
82184

185+
This is needed because the response from `fetch` as currently implemented in React Native does not support the `arrayBuffer()` call.
37.8 KB
Loading
38.3 KB
Loading
50.7 KB
Loading
54.9 KB
Loading

tfjs-react-native/integration_rn59/components/diagnostic.tsx

+4-2
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ import { Button, SafeAreaView, StyleSheet, ScrollView, View, StatusBar } from 'r
2020

2121
import * as tf from '@tensorflow/tfjs';
2222
import { Run } from './run';
23-
import { simpleOpRunner, precisionTestRunner, mobilenetRunner, localModelRunner, trainModelRunner } from './ml';
23+
import { simpleOpRunner, precisionTestRunner, mobilenetRunner, localModelRunner, trainModelRunner, saveModelRunner } from './ml';
2424

2525
interface ScreenProps {
2626
returnToMain: () => void;
@@ -75,10 +75,12 @@ export class Diagnostic extends React.Component<ScreenProps> {
7575
getRunner={precisionTestRunner} numRuns={1}></Run>
7676
<Run label='mobilenet'
7777
getRunner={mobilenetRunner} numRuns={1}></Run>
78-
<Run label='local model run'
78+
<Run label='bundleStorageIO'
7979
getRunner={localModelRunner} numRuns={1}></Run>
8080
<Run label='train model'
8181
getRunner={trainModelRunner} numRuns={1}></Run>
82+
<Run label='asyncStorareIO'
83+
getRunner={saveModelRunner} numRuns={1}></Run>
8284
</View>
8385
</View>
8486
</ScrollView>

tfjs-react-native/integration_rn59/components/ml.ts

+1-1
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@ export async function localModelRunner() {
7171
await tf.loadLayersModel(bundleResourceIO(modelJson, modelWeights));
7272

7373
return async () => {
74-
const res = model.predict(tf.randomNormal([1, 28, 28, 1])) as tf.Tensor;
74+
const res = model.predict(tf.randomNormal([1, 10])) as tf.Tensor;
7575
const data = await res.data();
7676
return JSON.stringify(data);
7777
};

tfjs-react-native/integration_rn59/components/webcam/webcam_demo.tsx

+3-1
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ export class WebcamDemo extends React.Component<ScreenProps,ScreenState> {
4646
super(props);
4747
this.state = {
4848
mode: 'results',
49-
cameraType: Camera.Constants.Type.front,
49+
cameraType: Camera.Constants.Type.back,
5050
isLoading: true,
5151
};
5252
this.styler = new StyleTranfer();
@@ -158,6 +158,7 @@ export class WebcamDemo extends React.Component<ScreenProps,ScreenState> {
158158
resultImage = await this.stylize(contentImage, styleImage),
159159
this.setState({
160160
styleImage,
161+
contentImage,
161162
resultImage,
162163
mode: 'results',
163164
isLoading: false,
@@ -175,6 +176,7 @@ export class WebcamDemo extends React.Component<ScreenProps,ScreenState> {
175176
resultImage = await this.stylize(contentImage, styleImage);
176177
this.setState({
177178
contentImage,
179+
styleImage,
178180
resultImage,
179181
mode: 'results',
180182
isLoading: false,

tfjs-react-native/integration_rn59/package.json

+2-2
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,8 @@
1111
"dependencies": {
1212
"@react-native-community/async-storage": "^1.6.1",
1313
"@tensorflow-models/mobilenet": "^1.0.1",
14-
"@tensorflow/tfjs": "^1.2.7",
15-
"@tensorflow/tfjs-react-native": "file:../",
14+
"@tensorflow/tfjs": "^1.2.8",
15+
"@tensorflow/tfjs-react-native": "0.1.0-alpha.2",
1616
"expo-camera": "^6.0.0",
1717
"expo-gl": "^6.0.0",
1818
"expo-gl-cpp": "^6.0.0",

tfjs-react-native/integration_rn59/yarn.lock

+28-26
Original file line numberDiff line numberDiff line change
@@ -853,15 +853,15 @@
853853
resolved "https://registry.yarnpkg.com/@tensorflow-models/mobilenet/-/mobilenet-1.0.1.tgz#e0dfdfd4941bae780dfe09e927ebdeea00926d12"
854854
integrity sha512-VaxVRCmJLK09yp2Qn/LvT29joZPT4aCBrSD4DwIkWghfJF7M/DGqxsiZSnjnUnYjzkZdNv7JCVJ4EY1PCmufQw==
855855

856-
"@tensorflow/[email protected].7":
857-
version "1.2.7"
858-
resolved "https://registry.yarnpkg.com/@tensorflow/tfjs-converter/-/tfjs-converter-1.2.7.tgz#3a1cf6d636667586010c4df79e861293ce10c4c8"
859-
integrity sha512-7MRH21zNgOOYGuztcrZwx6eXhg8gn/QvIZWljsmTJ33fnIyWLSAIHXsp5WgWalfAsj40yU92TnxaIyRuPeD/mw==
856+
"@tensorflow/[email protected].8":
857+
version "1.2.8"
858+
resolved "https://registry.yarnpkg.com/@tensorflow/tfjs-converter/-/tfjs-converter-1.2.8.tgz#86fa47be3e92a90d4191956f08015a17b93c3ef9"
859+
integrity sha512-weHzkNRVgnY9TcbA3XTneNgCyuIXLjF8ks8YbFA+81i2w6qO90xiAdWtP2YmR+F9K9S4WR3bSSB0AQKZAp+mPQ==
860860

861-
"@tensorflow/[email protected].7":
862-
version "1.2.7"
863-
resolved "https://registry.yarnpkg.com/@tensorflow/tfjs-core/-/tfjs-core-1.2.7.tgz#522328de16470aa9f7c15b91e4b68616f425002a"
864-
integrity sha512-RsXavYKMc0MOcCmOyD7HE8am1tWlDGXl0nJbsdib7ubmvMuH6KnrZ302eTYV7k1RMq+/ukkioJmCcw13hopuHQ==
861+
"@tensorflow/[email protected].8":
862+
version "1.2.8"
863+
resolved "https://registry.yarnpkg.com/@tensorflow/tfjs-core/-/tfjs-core-1.2.8.tgz#d6873b88522f8cf25d34c10afd095866578d7d92"
864+
integrity sha512-lWV4vAnXAAmahXpCWBwdGGW9HO6iNw9pUeVYih7pDXeJahMk3OJs6SgjRNhwn+ldsGwRoorR0/RHg0yNLmqWxQ==
865865
dependencies:
866866
"@types/offscreencanvas" "~2019.3.0"
867867
"@types/seedrandom" "2.4.27"
@@ -870,34 +870,36 @@
870870
node-fetch "~2.1.2"
871871
seedrandom "2.4.3"
872872

873-
"@tensorflow/[email protected].7":
874-
version "1.2.7"
875-
resolved "https://registry.yarnpkg.com/@tensorflow/tfjs-data/-/tfjs-data-1.2.7.tgz#06f5315a045ca680e721688e64976a418f43f979"
876-
integrity sha512-Q274DhAHkN6wJNEeNkKj7p++lueVd0QZXWTIVoyuzN7o25dJooStjomYjZimWIi1f81/3m2AoMtfA7y0uPGDiA==
873+
"@tensorflow/[email protected].8":
874+
version "1.2.8"
875+
resolved "https://registry.yarnpkg.com/@tensorflow/tfjs-data/-/tfjs-data-1.2.8.tgz#04325ad01167b0168ca8e34e85bc6a475aa5e8a0"
876+
integrity sha512-KUJy55mRx33OodlDKy6Emt2vvV9EadumZ6JzbtZCf1ADK5ddRpRLhCrEmGauCk+Ay1Zj+p6MYx6CLYqhJsENFQ==
877877
dependencies:
878878
"@types/node-fetch" "^2.1.2"
879879
node-fetch "~2.1.2"
880880

881-
"@tensorflow/[email protected].7":
882-
version "1.2.7"
883-
resolved "https://registry.yarnpkg.com/@tensorflow/tfjs-layers/-/tfjs-layers-1.2.7.tgz#4d9ad8966b5f1833ddabd2581c282cd288a69968"
884-
integrity sha512-qc9qRmJsizRon2HyIKKtWnZOQFMKdaWDeLlHyZUkrOn7YxqjFvxLs9BXsjKR1IuBliOljEe+ZEePDpasUtYLng==
881+
"@tensorflow/[email protected].8":
882+
version "1.2.8"
883+
resolved "https://registry.yarnpkg.com/@tensorflow/tfjs-layers/-/tfjs-layers-1.2.8.tgz#6d696dcfa530708b1fdb0cd7614a0d2a1c8ae58a"
884+
integrity sha512-NMcA44U0rvVpo/VUTMpqkBP3jbvNhRXV7K8KovLi/WZ9FG/u5XIbIqFkcT3QXBDUyjDUNFToYQM/59dtnaut9Q==
885885

886-
"@tensorflow/tfjs-react-native@file:..":
887-
version "0.1.0"
886+
"@tensorflow/[email protected]":
887+
version "0.1.0-alpha.2"
888+
resolved "https://registry.yarnpkg.com/@tensorflow/tfjs-react-native/-/tfjs-react-native-0.1.0-alpha.2.tgz#83e4efd6b8ff4e454df6cb3ccc56330b7547b5be"
889+
integrity sha512-VRatwALllGbJuo/8dov9LmGcmwcNItQwc8YEnQDLnEw4x5rpCSfcigvKmm0J4EcdFHgzh65j0ODzbO00QK9Z1g==
888890
dependencies:
889891
base64-js "^1.3.0"
890892
buffer "^5.2.1"
891893

892-
"@tensorflow/tfjs@^1.2.7":
893-
version "1.2.7"
894-
resolved "https://registry.yarnpkg.com/@tensorflow/tfjs/-/tfjs-1.2.7.tgz#5da167a56cda8d0fb6aa2f7cf1b1d6b36a99a7f7"
895-
integrity sha512-C5wCPic5kWDHLEoDCJsuypfge/vw9CVYpHAOzM57PXhVdYBNQRFNi2mA4KhB6jg8gs/e9u5Ei4lA1I0rhPorAQ==
894+
"@tensorflow/tfjs@^1.2.8":
895+
version "1.2.8"
896+
resolved "https://registry.yarnpkg.com/@tensorflow/tfjs/-/tfjs-1.2.8.tgz#441b78294f1e2f7174c5773f82424b8d963f0f60"
897+
integrity sha512-pNgLMWNpzWn8AIaTV+1MOjg0a99IBgGjDLLhgDBJ6m6/mzfDdFv2TxcYtF0kdF4asSd4cHaupuELeOzSF2coFw==
896898
dependencies:
897-
"@tensorflow/tfjs-converter" "1.2.7"
898-
"@tensorflow/tfjs-core" "1.2.7"
899-
"@tensorflow/tfjs-data" "1.2.7"
900-
"@tensorflow/tfjs-layers" "1.2.7"
899+
"@tensorflow/tfjs-converter" "1.2.8"
900+
"@tensorflow/tfjs-core" "1.2.8"
901+
"@tensorflow/tfjs-data" "1.2.8"
902+
"@tensorflow/tfjs-layers" "1.2.8"
901903

902904
"@types/babel__core@^7.1.0":
903905
version "7.1.2"

tfjs-react-native/karma.conf.js

+3-1
Original file line numberDiff line numberDiff line change
@@ -40,8 +40,9 @@ const karmaTypescriptConfig = {
4040

4141
const baseConfig = {
4242
frameworks: ['jasmine', 'karma-typescript'],
43+
autoWatch: true,
4344
files: [
44-
'./src/**/*.ts',
45+
{pattern: 'src/**/*.ts'},
4546
],
4647
preprocessors: {
4748
'src/**/*.ts': ['karma-typescript'],
@@ -54,6 +55,7 @@ const browserstackConfig = {
5455
...baseConfig,
5556
reporters: ['dots'],
5657
singleRun: true,
58+
autoWatch: false,
5759
hostname: 'bs-local.com',
5860
browserStack: {
5961
username: process.env.BROWSERSTACK_USERNAME,

tfjs-react-native/package.json

+5-9
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,11 @@
11
{
2-
"name": "@tensorflow/tfjs-platform-react-native",
3-
"version": "0.1.0",
2+
"name": "@tensorflow/tfjs-react-native",
3+
"version": "0.1.0-alpha.2",
44
"description": "TensorFlow.js platform implementation for React Native",
55
"main": "dist/index.js",
66
"types": "dist/index.d.ts",
7-
"jsnext:main": "dist/tf-react-native.esm.js",
8-
"module": "dist/tf-react-native.esm.js",
9-
"unpkg": "dist/tf-react-native.min.js",
10-
"jsdelivr": "dist/tf-react-native.min.js",
117
"license": "Apache-2.0",
12-
"private": true,
8+
"keywords": ["tensorflow", "tensorflowjs", "machine learning", "deep learning", "neural networks", "react native"],
139
"scripts": {
1410
"publish-local": "rimraf dist/ && yarn build && rollup -c && yalc push",
1511
"build": "tsc",
@@ -21,7 +17,7 @@
2117
},
2218
"devDependencies": {
2319
"@react-native-community/async-storage": "^1.4.2",
24-
"@tensorflow/tfjs-core": "^1.2.7",
20+
"@tensorflow/tfjs-core": ">=1.2.8",
2521
"@types/base64-js": "^1.2.5",
2622
"@types/jasmine": "~2.5.53",
2723
"@types/react-native": "^0.60.2",
@@ -52,7 +48,7 @@
5248
},
5349
"peerDependencies": {
5450
"@react-native-community/async-storage": "^1.4.2",
55-
"@tensorflow/tfjs-core": ">=1.2.7",
51+
"@tensorflow/tfjs-core": ">=1.2.8",
5652
"expo-gl": "^5.0.1",
5753
"react-native": ">= 0.58.0 < 0.60.0"
5854
}

0 commit comments

Comments
 (0)