[FAI-13688] Run source airbyte connector docker container #101
Conversation
| AutoRemove: true, | ||
| Init: true, | ||
| }, | ||
| }; |
There was a problem hiding this comment.
confirmed the cpu / mem / log settings.
A screenshot of docker inspect the container
|
|
||
| // Write the container ID to the cidfile | ||
| const cidfilePath = `tmp-${timestamp}-src_cid`; | ||
| writeFileSync(cidfilePath, container.id); |
There was a problem hiding this comment.
i followed the current script to have a cid file. is it actually helpful?
| maxMemory?: number; // unit: MB | ||
| maxCpus?: number; // unit: CPU | ||
| maxLogSize?: string; // default: 10m (10MB) | ||
| additionalOptions?: any; |
There was a problem hiding this comment.
the dockerode lib doesn't take the docker options as a string instead it take a typed structure. i think they are corresponding to this https://docs.docker.com/reference/api/engine/version/v1.43/#tag/Container/operation/ContainerCreate
so i break down to a few variables and have an free form to take options that are not pre-defined by us. I would assume this won't be used often but might be useful when we or users need debugging
| }); | ||
| // Attach the stderr to termincal stderr, and stdout to the output stream | ||
| const stream = await container.attach({stream: true, stdout: true, stderr: true}); | ||
| container.modem.demuxStream(stream, stdoutStream, process.stderr); |
There was a problem hiding this comment.
stdout -> captured and returned in this function
stderr -> terminal
| @@ -14,6 +14,7 @@ export const SRC_CONFIG_FILENAME = `${FILENAME_PREFIX}_src_config.json`; | |||
| export const DST_CONFIG_FILENAME = `${FILENAME_PREFIX}_dst_config.json`; | |||
| export const SRC_CATALOG_FILENAME = `${FILENAME_PREFIX}_src_catalog.json`; | |||
| export const DST_CATALOG_FILENAME = `${FILENAME_PREFIX}_dst_catalog.json`; | |||
| export const DEFAULT_STATE_FILE = 'state.json'; | |||
There was a problem hiding this comment.
I wonder if we should make the default state file name something that includes the src and dst image names, so it doesn't just keep overwriting the same state.json file every time.
There was a problem hiding this comment.
i thought we replace the state file every time?
this state file is in the temp folder that has differ name every run. and it will replace the state file that is outside the temp folder at the end of the run
|
Description
This PR doesn't include process the output of the source connector.