Skip to content

feat: add getViews and categorize table types #3327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 12 commits into
base: 5.x
Choose a base branch
from
+90 −11
Open

feat: add getViews and categorize table types #3327

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
c32dd99
feat: exclude system collections, add engine collection type to getTa…
masterbater Mar 24, 2025
196405b
chore: remove comment
masterbater Mar 24, 2025
3162280
feat: separate virtual view and table
masterbater Mar 24, 2025
ae5187a
chore: cs fix
masterbater Mar 24, 2025
12b6d1f
chore: remove modified dev environment
masterbater Mar 24, 2025
825a4fe
fix: restore aggregate getTables
masterbater Mar 24, 2025
19f1cdd
fix: follow suggestions
masterbater Mar 24, 2025
7a774e9
fix: remove stats, comparison directly in if
masterbater Mar 24, 2025
44a57f5
fix: follow master jmikola suggestions
masterbater Mar 25, 2025
ef91b44
chore: fix grammar comment
masterbater Mar 25, 2025
4890eca
feat: categorize by collection type, drop database in dropAllTables
masterbater Mar 25, 2025
0e14a68
chore: add test to ensure normal collections are excluded from getViews
masterbater Mar 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion phpunit.xml.dist
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
</testsuite>
</testsuites>
<php>
<env name="MONGODB_URI" value="mongodb://mongodb/?directConnection=true"/>
<env name="MONGODB_URI" value="mongodb://127.0.0.1/?directConnection=true"/>
<env name="MONGODB_DATABASE" value="unittest"/>
<env name="SQLITE_DATABASE" value=":memory:"/>
<env name="QUEUE_CONNECTION" value="database"/>
Expand Down
68 changes: 62 additions & 6 deletions src/Schema/Builder.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
use function sort;
use function sprintf;
use function str_ends_with;
use function str_starts_with;
use function substr;
use function usort;

Expand Down Expand Up @@ -142,13 +143,25 @@ public function dropAllTables()
}
}

/** @param string|null $schema Database name */
/** @param string|null $schema Database name */
public function getTables($schema = null)
{
$db = $this->connection->getDatabase($schema);
$collections = [];

foreach ($db->listCollectionNames() as $collectionName) {
foreach ($db->listCollections() as $collectionInfo) {
$collectionName = $collectionInfo->getName();

// Skip system collections
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The system collections are actual collections. In the case of system.profile, that might actually be something an application would want to query.

I'm not certain it makes sense to exclude these from all output through the integration.

if (str_starts_with($collectionName, 'system.')) {
continue;
}

// Skip views it doesnt suport aggregate
if ($collectionInfo->getType() === 'view') {
continue;
}

$stats = $db->selectCollection($collectionName)->aggregate([
['$collStats' => ['storageStats' => ['scale' => 1]]],
['$project' => ['storageStats.totalSize' => 1]],
Expand All @@ -165,9 +178,42 @@ public function getTables($schema = null)
];
}

usort($collections, function ($a, $b) {
return $a['name'] <=> $b['name'];
});
usort($collections, fn ($a, $b) => $a['name'] <=> $b['name']);

return $collections;
}

/** @param string|null $schema Database name */
public function getViews($schema = null)
{
$db = $this->connection->getDatabase($schema);
$collections = [];

foreach ($db->listCollections() as $collectionInfo) {
$collectionName = $collectionInfo->getName();

// Skip system collections
if (str_starts_with($collectionName, 'system.')) {
continue;
}

// Skip normal type collection
if ($collectionInfo->getType() !== 'view') {
continue;
}

$collections[] = [
'name' => $collectionName,
'schema' => $db->getDatabaseName(),
'schema_qualified_name' => $db->getDatabaseName() . '.' . $collectionName,
'size' => null,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is relevant, the information can be provided via $collStats (MongoDB 6.2+). I'm not sure that'd be worth the overhead to obtain this for all collections/views, though.

'comment' => null,
'collation' => null,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is actually relevant, the information should be available via the options field for each collection info result. See: listCollections

'engine' => null,
];
}

usort($collections, fn ($a, $b) => $a['name'] <=> $b['name']);

return $collections;
}
Expand Down Expand Up @@ -196,6 +242,9 @@ public function getTableListing($schema = null, $schemaQualified = false)

$collections = array_merge(...array_values($collections));

// Exclude system collections before sorting
$collections = array_filter($collections, fn ($name) => ! str_starts_with($name, 'system.'));

sort($collections);

return $collections;
Expand Down Expand Up @@ -346,7 +395,14 @@ protected function getAllCollections()
{
$collections = [];
foreach ($this->connection->getDatabase()->listCollections() as $collection) {
$collections[] = $collection->getName();
$name = $collection->getName();

// Skip system collections
if (str_starts_with($name, 'system.')) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will preserve system collection from being dropped 👍🏻

Even for the gridfs bucket metadata, I think that's good to not remove this collection when dropAllTables is called.
https://www.mongodb.com/docs/manual/reference/system-collections/#database-specific-collections

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

system.buckets stores data for time series collections. It's unrelated to GridFS, which stores its data in <bucketName>.fs (metadata) and <bucketName.files> (chunk data) collections.

I'm not sure how dropAllTables() is used, but I don't see why you'd need system collections to persist if everything else is being removed. Moreover, if Blueprint::drop() (called by dropAllTables()) is going to invoke Collection::drop() for each collection, you could conceivably just drop the entire database to be more efficient.

continue;
}

$collections[] = $name;
}

return $collections;
Expand Down
72 changes: 71 additions & 1 deletion tests/SchemaTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,17 @@
use function collect;
use function count;
use function sprintf;
use function str_starts_with;

class SchemaTest extends TestCase
{
public function tearDown(): void
{
$database = $this->getConnection('mongodb')->getMongoDB();
$database = $this->getConnection('mongodb')->getDatabase();
assert($database instanceof Database);
$database->dropCollection('newcollection');
$database->dropCollection('newcollection_two');
$database->dropCollection('test_view');

parent::tearDown();
}
Expand Down Expand Up @@ -397,6 +399,9 @@ public function testGetTables()
DB::connection('mongodb')->table('newcollection_two')->insert(['test' => 'value']);
$dbName = DB::connection('mongodb')->getDatabaseName();

// Create a view (this creates system.views)
DB::connection('mongodb')->getDatabase()->createCollection('test_view', ['viewOn' => 'newcollection']);

$tables = Schema::getTables();
$this->assertIsArray($tables);
$this->assertGreaterThanOrEqual(2, count($tables));
Expand All @@ -413,24 +418,67 @@ public function testGetTables()
$this->assertEquals($dbName . '.newcollection', $table['schema_qualified_name']);
$found = true;
}

// Ensure system collections are excluded
$this->assertFalse(str_starts_with($table['name'], 'system.'));
}

if (! $found) {
$this->fail('Collection "newcollection" not found');
}
}

public function testGetViews()
{
DB::connection('mongodb')->table('newcollection')->insert(['test' => 'value']);
DB::connection('mongodb')->table('newcollection_two')->insert(['test' => 'value']);
$dbName = DB::connection('mongodb')->getDatabaseName();

DB::connection('mongodb')->getDatabase()->createCollection('test_view', ['viewOn' => 'newcollection']);

$tables = Schema::getViews();

$this->assertIsArray($tables);
$this->assertGreaterThanOrEqual(1, count($tables));
$found = false;
foreach ($tables as $table) {
$this->assertArrayHasKey('name', $table);
$this->assertArrayHasKey('size', $table);
$this->assertArrayHasKey('schema', $table);
$this->assertArrayHasKey('schema_qualified_name', $table);

if ($table['name'] === 'test_view') {
$this->assertEquals($dbName, $table['schema']);
$this->assertEquals($dbName . '.test_view', $table['schema_qualified_name']);
$found = true;
}

// Ensure system collections are excluded
$this->assertFalse(str_starts_with($table['name'], 'system.'));
}

if (! $found) {
$this->fail('Collection "test_view" not found');
}
}

public function testGetTableListing()
{
DB::connection('mongodb')->table('newcollection')->insert(['test' => 'value']);
DB::connection('mongodb')->table('newcollection_two')->insert(['test' => 'value']);

// Create a view (this creates system.views)
DB::connection('mongodb')->getDatabase()->createCollection('test_view', ['viewOn' => 'newcollection']);

$tables = Schema::getTableListing();

$this->assertIsArray($tables);
$this->assertGreaterThanOrEqual(2, count($tables));
$this->assertContains('newcollection', $tables);
$this->assertContains('newcollection_two', $tables);

// Ensure system collections are excluded
$this->assertNotContains('system.views', $tables);
}

public function testGetTableListingBySchema()
Expand All @@ -454,6 +502,28 @@ public function testGetTableListingBySchema()
$this->assertContains('newcollection_two', $tables);
}

public function testSystemCollectionsArePresentButFiltered()
{
// Create a view to trigger system.views collection
DB::connection('mongodb')->getDatabase()->createCollection('test_view', ['viewOn' => 'newcollection']);

// Get all collections directly from MongoDB
$allCollections = DB::connection('mongodb')->getDatabase()->listCollectionNames();

// Ensure the system.views collection exists in MongoDB
$this->assertContains('system.views', $allCollections);

// Ensure Schema::getTables does NOT include system collections
$tables = Schema::getTables();
foreach ($tables as $table) {
$this->assertFalse(str_starts_with($table['name'], 'system.'));
}

// Ensure Schema::getTableListing does NOT include system collections
$tableListing = Schema::getTableListing();
$this->assertNotContains('system.views', $tableListing);
}

public function testGetColumns()
{
$collection = DB::connection('mongodb')->table('newcollection');
Expand Down
Loading