Some test solutions do not return outputs in sandbox running, while they output successfully on CLI.

[stovecat]

• Environment: Dockerhub from Error - bazel run #9 (comment) (Thanks to @LaudemPax)
• Successfully ran solve_example.cc
• Tested on both Python3.8.10 and Python3.10.4

I built a custom code based on solve_example.cc to evaluate an arbitrary test problem.

However, l found that some of the test solutions passed yet some did not.

After a round of debugging, l confirmed that some of the test solutions do not return any output through stdout.

My conjecture is that, for some reason, the sandbox fails to bind stdout of the running code time to time.

Here is one of the test solution code for 1580_A. Portal (test problem index=20, solution index=105):

import sys;input=sys.stdin.readline
T, = map(int, input().split())
for _ in range(T):
    N, M = map(int, input().split())
    X = [[0]*(M+1)]
    for _ in range(N):
        X.append([0]+[int(c) for c in input().strip()])
    Y = [[0]*(M+1) for _ in range(N+1)]
    for i in range(N+1):
        for j in range(M+1):
            Y[i][j] = X[i][j]
    for i in range(1, N+1):
        for j in range(1, M+1):
            X[i][j] += - X[i-1][j-1] + X[i][j-1] + X[i-1][j]
    R = 10**18
    L = []
    for i in range(5, N+1):
        for j in range(4, M+1):
            x = i-4
            y = j-3
            r = X[i-1][j-1]-X[x][j-1]-X[i-1][y]+X[x][y]
            rrr = (X[i-1][j]+X[i][j-1]-X[i-1][j-1]-r-(X[x-1][j]+X[i][y-1]-X[x-1][y-1]))
            r2 = rrr-Y[x][j]-Y[i][y]-Y[x][y]
            rr = r+(2*(i-x+1-2)+2*(j-y+1-2))-r2
            L.append((rr, x, y))
    L.sort(key=lambda x:x[0])
    for i in range(min(len(L), 23)):
        _, x, y = L[i]
#        print(x, y)
        for i in range(x+4, N+1):
            for j in range(y+3, M+1):
                r = X[i-1][j-1]-X[x][j-1]-X[i-1][y]+X[x][y]
                rrr = (X[i-1][j]+X[i][j-1]-X[i-1][j-1]-r-(X[x-1][j]+X[i][y-1]-X[x-1][y-1]))
                r2 = rrr-Y[x][j]-Y[i][y]-Y[x][y]
                rr = r+(2*(i-x+1-2)+2*(j-y+1-2))-r2
                if R > rr:
                    R = rr
    print(R)

I inserted std::cout in between #L404 and #405, for logging test_result->stdout and test_result->stderr.

Here is the log for running the sandbox:

test_result->stdout:
test_result->stderr:
test_result->stdout:
test_result->stderr:
test_result->stdout:
test_result->stderr:
test_result->stdout:
test_result->stderr:
Compilation succeeded
Test 0 failed.
Test 1 failed.
Test 2 failed.
Test 3 failed.
Test 4 did not run.
Test 5 did not run.
Test 6 did not run.
Test 7 did not run.
Test 8 did not run.
Test 9 did not run.

When I run this solution directly on my CLI, it successfully returns the output. (Input: '1\n9 9\n001010001\n101110100\n000010011\n100000001\n101010101\n110001111\n000001111\n111100000\n000110000\n'; Ouput: '5\n')
(Note that I used the same Python path for both sandbox running and CLI running.)

$ python code.py
1
9 9
001010001
101110100
000010011
100000001
101010101
110001111
000001111
111100000
000110000
5
$

For comparison, I also share a passed solution for 1579_G. Minimal Coverage (test problem index=19, solution index=9):

import sys
input = sys.stdin.readline

for _ in range(int(input())):
    n = int(input())
    a = list(map(int, input().split()))
    dp = [[2100]*2100 for _ in range(n)]
    dp[0][a[0]] = a[0]
    
    for i in range(1, n):
        for j in range(2100):
            nj = max(j-a[i], 0)
            dp[i][nj] = min(dp[i][nj], dp[i-1][j]+max(a[i]-j, 0))
            
            if j+a[i]<2100:
                dp[i][j+a[i]] = min(dp[i][j+a[i]], dp[i-1][j]+max(a[i]-(dp[i-1][j]-j), 0))
    
    print(min(dp[-1]))

test_result->stdout: 3
3
9
9
7
8

test_result->stderr:
test_result->stdout: 3
3
14
9
7
8

test_result->stderr:
test_result->stdout: 3
3
9
9
7
8

test_result->stderr:
test_result->stdout: 3
6
9
9
7
8

test_result->stderr:
test_result->stdout: 3
3
9
9
7
7

test_result->stderr:
test_result->stdout: 3
6
9
12
7
8

test_result->stderr:
test_result->stdout: 3
2
9
9
7
7

test_result->stderr:
test_result->stdout: 3
3
23
9
7
8

test_result->stderr:
test_result->stdout: 4
3
14
9
7
8

test_result->stderr:
test_result->stdout: 3
6
9
12
12
8

test_result->stderr:
Compilation succeeded
Test 0 passed.
Test 1 passed.
Test 2 passed.
Test 3 passed.
Test 4 passed.
Test 5 passed.
Test 6 passed.
Test 7 passed.
Test 8 passed.
Test 9 passed.

This solution successfully run on CLI as well (Input: '6\n2\n1 3\n3\n1 2 3\n4\n6 2 3 9\n4\n6 8 4 5\n7\n1 2 4 6 7 7 3\n8\n8 6 5 1 2 2 3 6\n'; Expected ouput: '3\n3\n9\n9\n7\n8\n').

$ python code.py
6
2
1 3
3
1 2 3
4
6 2 3 9
4
6 8 4 5
7
1 2 4 6 7 7 3
8
8 6 5 1 2 2 3 6
3
3
9
9
7
8
$

I checked stdin with the same way (insert std::cout in between of #L313 and #L314) I did for stdout, but it seems ok as both failed and passed solutions print inputs.

I tried to find the distinctions between failed solutions and passed solutions (e.g., the way they take inputs/outputs), but it was hard to point out the differences.

I leave the test problem indices for the passed and the failed solutions as follows:
(Remaining indices do not have Python solution.)
[Failed]

0, 3, 9, 14, 16, 17, 18, 20, 30, 31, 36, 37, 38, 39, 40, 41, 52, 54, 56, 60, 62, 63, 74, 75, 77, 79, 92, 95, 101, 103, 104, 105, 107, 108, 113, 116, 117, 122, 123, 128, 129, 131, 134, 135, 137, 138, 143, 144, 145, 146, 149, 157, 159, 160, 161, 162, 163, 164

[Passed]

8, 10, 11, 13, 15, 19, 26, 27, 28, 29, 33, 34, 44, 45, 46, 47, 48, 50, 51, 53, 55, 57, 58, 59, 61, 65, 67, 76, 78, 87, 88, 89, 90, 93, 94, 96, 98, 99, 100, 102, 106, 109, 114, 115, 119, 120, 121, 127, 132, 133, 136, 139, 140, 141, 142, 147, 148, 150, 151, 152, 154, 155, 156, 158

Throughout the Python solutions, I used the first ones, e.g., I got the index of the failed solution above as testset[20]['solutions']['language'].index(3).

P.S. As there is a possibility of sandbox discrepancy issue mentioned in #14 (comment), I checked that:

• All the CodeContests test problems are from Codeforces, where Codeforces do not allow any external library (See https://codeforces.com/blog/entry/44806 and https://codeforces.com/blog/entry/70890).
• There was no compilation failed nor SANDBOX VIOLATION.

Thus it seems that this issue is independent from the sandbox discrepancy issue, as all the test solutions are implemented by internal libraries only and all solutions got no violation.

Thank you in advance.

[felixgimeno]

I tried to reproduce the mentioned example for "1580_A. Portal" using Python 3.9 but it succeeds for me.
Can you try installing python3.9 and rerunning with --python3_path=/usr/bin/python3.9 --python3_library_paths=/usr/lib/python3.9 ?