You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
1.[Senzing APIs Bare Metal Usage](#senzing-apis-bare-metal-usage)
12
-
1.[Configuration](#configuration)
13
-
2.[Usage](#usage)
14
-
1.[Docker Usage](#docker-usage)
15
-
1.[Configuration](#configuration-1)
16
-
2.[Usage](#usage-1)
17
-
1.[Items of Note](#items-of-note)
18
-
1.[With Info](#with-info)
19
-
2.[Parallel Processing](#parallel-processing)
20
-
3.[Scalability](#scalability)
21
-
4.[Randomize Input Files](#randomize-input-files)
22
-
5.[Purging Senzing Repository Between Examples](#purging-senzing-repository-between-examples)
23
-
6.[Input Load File Sizes](#input-load-file-sizes)
16
+
1.[Legend]
17
+
1.[Warning]
18
+
1.[Senzing Engine Configuration]
19
+
1.[Senzing APIs Bare Metal Usage]
20
+
1.[Configuration]
21
+
2.[Usage]
22
+
1.[Docker Usage]
23
+
1.[Configuration]
24
+
2.[Usage]
25
+
1.[Items of Note]
26
+
1.[With Info]
27
+
2.[Parallel Processing]
28
+
3.[Scalability]
29
+
4.[Randomize Input Data]
30
+
5.[Purging Senzing Repository Between Examples]
31
+
6.[Input Load File Sizes]
24
32
25
33
### Legend
26
34
@@ -30,81 +38,89 @@ Succinct examples of how you might use the Senzing APIs for operational tasks.
30
38
1.:pencil2: - A "pencil" icon means that the instructions may need modification before performing.
31
39
1.:warning: - A "warning" icon means that something tricky is happening, so pay attention.
32
40
33
-
34
41
## Warning
35
42
36
-
:warning::warning::warning:__Only run the code snippets against a test Senzing database instance.__ Running the snippets adds and deletes data, and some snippets purge the entire database of currently ingested data. It is recommended to create a separate test Senzing project if you are using a bare metal Senzing install, or if using Docker a separate Senzing database to use only with the snippets. If you are getting started and are unsure please contact [Senzing Support](https://senzing.zendesk.com/hc/en-us/requests/new). :warning::warning::warning:
43
+
:warning::warning::warning:**Only run the code snippets against a test Senzing database instance.** Running the snippets adds and deletes data, and some snippets purge the entire database of currently ingested data. It is recommended to create a separate test Senzing project if you are using a bare metal Senzing install, or if using Docker a separate Senzing database to use only with the snippets. If you are getting started and are unsure please contact [Senzing Support]. :warning::warning::warning:
37
44
38
45
## Senzing Engine Configuration
39
46
40
47
A JSON configuration string is used by the snippets to specify initialization parameters to the Senzing engine:
The JSON configuration string is set via the environment variable `SENZING_ENGINE_CONFIGURATION_JSON`.
58
63
59
64
## Senzing APIs Bare Metal Usage
60
-
You may already have installed the Senzing APIs and created a Senzing project by following the [Quickstart Guide](https://senzing.zendesk.com/hc/en-us/articles/115002408867-Quickstart-Guide). If not, and you would like to install the Senzing APIs directly on a machine, follow the steps in the[ Quickstart Guide](https://senzing.zendesk.com/hc/en-us/articles/115002408867-Quickstart-Guide). Be sure to review the API [Quickstart Roadmap](https://senzing.zendesk.com/hc/en-us/articles/115001579954-API-Quickstart-Roadmap), especially the [System Requirements](https://senzing.zendesk.com/hc/en-us/articles/115010259947).
65
+
66
+
You may already have installed the Senzing APIs and created a Senzing project by following the [Quickstart Guide]. If not, and you would like to install the Senzing APIs directly on a machine, follow the steps in the [Quickstart Guide]. Be sure to review the API [Quickstart Roadmap], especially the [System Requirements].
61
67
62
68
### Configuration
63
69
64
-
When using a bare metal install, the initialization parameters used by the Senzing Python utilities are maintained within ```<project_path>/etc/G2Module.ini```.
70
+
When using a bare metal install, the initialization parameters used by the Senzing Python utilities are maintained within `<project_path>/etc/G2Module.ini`.
65
71
66
72
🤔To convert an existing Senzing project G2Module.ini file to a JSON string use one of the following methods:
:pencil2:`<project_path>` in the above examples should point to your project.
100
-
101
-
116
+
102
117
## Docker Usage
103
118
104
-
The included Dockerfile leverages the [Senzing API runtime](https://github.com/Senzing/senzingapi-runtime) image to provide an environment to run the code snippets.
119
+
The included Dockerfile leverages the [Senzing API runtime] image to provide an environment to run the code snippets.
105
120
106
-
### Configuration
107
-
When used with a container, the JSON configuration is relative to the paths within the container. The JSON configuration should look like:
121
+
### Configuration for Docker usage
122
+
123
+
When used with a container, the JSON configuration is relative to the paths within the container. The JSON configuration should look like:
108
124
109
125
```json
110
126
{
@@ -121,65 +137,109 @@ The included Dockerfile leverages the [Senzing API runtime](https://github.com/S
121
137
122
138
✏️You only need to modify the `CONNECTION` string to point to your Senzing database.
123
139
124
-
### Usage
140
+
### Usage for Dccker usage
141
+
125
142
1. Clone this repository
126
-
2. Export the engine configuration environment variable
143
+
1. Export the engine configuration environment variable
✏️You only need to modify the `CONNECTION` string to point to your Senzing database.
146
168
147
169
## Items of Note
148
-
170
+
149
171
### With Info
172
+
150
173
A feature of Senzing is the capability to pass changes from data manipulation API calls to downstream systems for analysis, consolidation and replication. Any API that can change the outcome of entity resolution have a "WithInfo" version of the API. For example, addRecord and addRecordWithInfo. The "WithInfo" version of the API returns a response message detailing any entities that were affected by the API. In the following example (from addRecordWithInfo) a single entity with the ID 7903 was affected.
174
+
151
175
```json
152
176
{
153
-
"DATA_SOURCE": "TEST",
154
-
"RECORD_ID": "10945",
155
-
"AFFECTED_ENTITIES": [
156
-
{
157
-
"ENTITY_ID": 7903,
158
-
"LENS_CODE": "DEFAULT"
159
-
}
160
-
],
161
-
"INTERESTING_ENTITIES": []
177
+
"DATA_SOURCE": "TEST",
178
+
"RECORD_ID": "10945",
179
+
"AFFECTED_ENTITIES": [
180
+
{
181
+
"ENTITY_ID": 7903,
182
+
"LENS_CODE": "DEFAULT"
183
+
}
184
+
],
185
+
"INTERESTING_ENTITIES": []
162
186
}
163
187
```
164
-
The AFFECTED_ENTITIES object contains a list of all entity IDs affected. Separate processes can query the affected entities and synchronize changes and information to downstream systems. For additional information see [Real-time replication and analytics](https://senzing.zendesk.com/hc/en-us/articles/4417768234131--Advanced-Real-time-replication-and-analytics).
188
+
189
+
The AFFECTED_ENTITIES object contains a list of all entity IDs affected. Separate processes can query the affected entities and synchronize changes and information to downstream systems. For additional information see [Real-time replication and analytics].
165
190
166
191
### Parallel Processing
192
+
167
193
Many of the example tasks demonstrate concurrent execution with threads. The entity resolution process involves IO operations, the use of concurrent processes and threads when calling the Senzing APIs provides scalability and performance. If using multiple processes, each process should have its own instance of a Senzing engine, for example G2Engine. Each engine object can support multiple threads.
168
194
169
195
### Scalability
170
-
Many of the examples demonstrate using multiple threads to utilize the resources available on the machine. Consider loading data into Senzing and increasing the load rate, loading (and other tasks) can be horizontally scaled by utilizing additional machines.
171
196
172
-
If a single very large load file and 3 machines were available for performing data load, the file can be split into 3 with each machine running the sample code or your own application. Horizontal scaling such as this does require the Senzing database to have the capacity to accept the additional workload and not become the bottleneck.
197
+
Many of the examples demonstrate using multiple threads to utilize the resources available on the machine. Consider loading data into Senzing and increasing the load rate, loading (and other tasks) can be horizontally scaled by utilizing additional machines.
198
+
199
+
If a single very large load file and 3 machines were available for performing data load, the file can be split into 3 with each machine running the sample code or your own application. Horizontal scaling such as this does require the Senzing database to have the capacity to accept the additional workload and not become the bottleneck.
173
200
174
201
### Randomize Input Data
175
-
When providing your own input file(s) to the snippets or your own applications and processing data manipulation tasks (adding, deleting, replacing), it is important to randomize the file(s) or other input methods when running multiple threads. If source records that pertain to the same entity are clustered together, multiple processes or threads could all be trying to work on the same entity concurrently. This causes contention and overhead resulting in slower performance. To prevent this contention always randomize input data.
176
202
177
-
You may be able to randomize your input files during ETL and mapping the source data to the [Senzing Entity Specification](https://senzing.zendesk.com/hc/en-us/articles/231925448-Generic-Entity-Specification). Otherwise utilities such as [shuf](https://man7.org/linux/man-pages/man1/shuf.1.html) or [terashuf](https://github.com/alexandres/terashuf) for large files can be used.
203
+
When providing your own input file(s) to the snippets or your own applications and processing data manipulation tasks (adding, deleting, replacing), it is important to randomize the file(s) or other input methods when running multiple threads. If source records that pertain to the same entity are clustered together, multiple processes or threads could all be trying to work on the same entity concurrently. This causes contention and overhead resulting in slower performance. To prevent this contention always randomize input data.
204
+
205
+
You may be able to randomize your input files during ETL and mapping the source data to the [Senzing Entity Specification]. Otherwise utilities such as [shuf] or [terashuf] for large files can be used.
178
206
179
207
### Purging Senzing Repository Between Examples
208
+
180
209
When trying out different examples you may notice consecutive tasks complete much faster than an initial run. For example, running a loading task for the first time without the data in the system will be representative of load rate. If the same example is subsequently run again without purging the system it will complete much faster. This is because Senzing knows the records already exist in the system and it skips them.
181
210
182
-
To run the same example again and see representative performance, first [purge](Python/Tasks/Initialization/PurgeRepository.py) the Senzing repository of the loaded data. Some examples don't require purging between running them, an example would be the deleting examples that require data to be ingested first. See the usage notes for each task category for an overview of how to use the snippets.
211
+
To run the same example again and see representative performance, first [purge] the Senzing repository of the loaded data. Some examples don't require purging between running them, an example would be the deleting examples that require data to be ingested first. See the usage notes for each task category for an overview of how to use the snippets.
183
212
184
213
### Input Load File Sizes
185
-
There are different sized load files within the [Data](Resources/Data/) path that can be used to decrease or increase the volume of data loaded depending on the specification of your hardware. The files are named loadx.json, where the x specifies the number of records in the file.
214
+
215
+
There are different sized load files within the [Data] path that can be used to decrease or increase the volume of data loaded depending on the specification of your hardware. The files are named loadx.json, where the x specifies the number of records in the file.
0 commit comments