Skip to content

Commit f6b16a7

Browse files
committed
Fixes for tests
1 parent b437bfa commit f6b16a7

File tree

3 files changed

+268
-25
lines changed

3 files changed

+268
-25
lines changed

.rules/new_models_best_practice.mdc

Lines changed: 109 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,10 +37,118 @@ Each example should contain the following files:
3737

3838
## 4. Testing
3939

40-
- Tests should be runnable with `pytest` directly from the repository root or inside the examples Docker container.
40+
- Tests should be runnable with `pytest` directly from the repository root or inside the example's Docker container.
4141
- Mock Label Studio API interactions whenever possible to avoid requiring a running server during tests.
4242
- Aim for good coverage of `fit()` and `predict()` logic to catch regressions.
4343

44+
### 4.1. Running Tests in Docker Containers
45+
46+
For ML backends that require specific dependencies or environments, Docker containers provide consistent testing environments. Here's the recommended workflow:
47+
48+
#### Setup and Build
49+
```bash
50+
# Navigate to your example directory
51+
cd label_studio_ml/examples/<your_example>
52+
53+
# Build the Docker container (without --no-cache for faster builds)
54+
docker compose -f docker-compose.yml build
55+
56+
# Start the container in background
57+
docker compose -f docker-compose.yml up -d
58+
```
59+
60+
#### Install Test Dependencies
61+
Most containers won't have pytest installed by default. Install it:
62+
```bash
63+
# Install pytest and coverage tools
64+
docker compose -f docker-compose.yml exec -T <service_name> pip install pytest pytest-cov
65+
```
66+
67+
#### Run Tests
68+
```bash
69+
# Run all tests with verbose output and coverage
70+
docker compose -f docker-compose.yml exec -T <service_name> pytest -vvv --cov --cov-report=xml:/tmp/coverage.xml
71+
72+
# Run specific test files
73+
docker compose -f docker-compose.yml exec -T <service_name> pytest -vvv tests/test_model.py
74+
75+
# Run specific test methods
76+
docker compose -f docker-compose.yml exec -T <service_name> pytest -vvv tests/test_model.py::TestClass::test_method
77+
78+
# Collect available tests without running them
79+
docker compose -f docker-compose.yml exec -T <service_name> pytest --collect-only tests/
80+
```
81+
82+
#### Example: TimeSeries Segmenter
83+
```bash
84+
# Complete testing workflow for timeseries_segmenter
85+
cd label_studio_ml/examples/timeseries_segmenter
86+
87+
# Build and start container
88+
docker compose -f docker-compose.yml build
89+
docker compose -f docker-compose.yml up -d
90+
91+
# Install test dependencies
92+
docker compose -f docker-compose.yml exec -T timeseries_segmenter pip install pytest pytest-cov
93+
94+
# Run full test suite
95+
docker compose -f docker-compose.yml exec -T timeseries_segmenter pytest -vvv --cov --cov-report=xml:/tmp/coverage.xml tests/test_segmenter.py
96+
97+
# Cleanup
98+
docker compose -f docker-compose.yml down
99+
```
100+
101+
#### Troubleshooting Docker Tests
102+
103+
**Issue: Test files not found or outdated**
104+
- Solution: Rebuild the container if test files were modified after the image was built
105+
- Use `docker compose build` (without `--no-cache` unless absolutely necessary)
106+
107+
**Issue: Import errors in container**
108+
- Solution: Ensure all dependencies are in `requirements.txt` and properly installed
109+
- Check that the test file has correct import paths (relative vs absolute)
110+
111+
**Issue: Environment variables not set**
112+
- Solution: Use `patch.dict(os.environ, {...})` in tests or set them in docker-compose.yml
113+
- Override instance attributes directly for test configurations
114+
115+
**Issue: Mock function signature mismatches**
116+
- Solution: Check the actual method signatures and use `side_effect` with lambda functions when needed
117+
- Ensure mock functions match the expected parameter names and types
118+
119+
#### Best Practices for Docker Testing
120+
121+
1. **Consistent Environment**: Use the same base image for development and testing
122+
2. **Fast Iteration**: Avoid `--no-cache` unless dependencies changed
123+
3. **Test Isolation**: Use temporary directories and cleanup fixtures
124+
4. **Comprehensive Logging**: Enable verbose pytest output (`-vvv`) for debugging
125+
5. **Coverage Reports**: Generate coverage reports to ensure thorough testing
126+
6. **Service Names**: Use descriptive service names in docker-compose.yml for clarity
127+
128+
#### Test Documentation in Code
129+
130+
Each test method should include comprehensive docstrings explaining:
131+
- What functionality is being tested
132+
- Expected inputs and outputs
133+
- Critical validations being performed
134+
- Edge cases being handled
135+
136+
Example:
137+
```python
138+
def test_model_training_workflow(self):
139+
"""Test complete end-to-end machine learning pipeline.
140+
141+
This test validates:
142+
- Full training workflow with real data
143+
- Training metrics generation (accuracy, F1-score, loss)
144+
- Model convergence and learning validation
145+
- Prediction generation on trained model
146+
147+
Critical validation: The complete ML pipeline works from training
148+
to prediction, producing valid Label Studio annotations.
149+
"""
150+
```
151+
44152
## 5. Examples
45153

46154
- You can use as an implementation example `label_studio_ml/examples/yolo/`. It's well written and can be a model to follow.

label_studio_ml/examples/timeseries_segmenter/model.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -127,7 +127,7 @@ def _get_labeling_params(self) -> Dict:
127127

128128
def _read_csv(self, task: Dict, path: str) -> pd.DataFrame:
129129
logger.debug(f"Reading CSV data from path: {path}")
130-
csv_str = self.preload_task_data(task, path)
130+
csv_str = self.preload_task_data(task, value=path)
131131
df = pd.read_csv(io.StringIO(csv_str))
132132
logger.debug(f"CSV loaded with shape: {df.shape}")
133133
return df

0 commit comments

Comments
 (0)