Skip to content

[feat](iceberg) Support Google Authentication for Iceberg REST catalog#62988

Open
tuantran0910 wants to merge 2 commits intoapache:masterfrom
tuantran0910:feat/iceberg-rest-google-auth
Open

[feat](iceberg) Support Google Authentication for Iceberg REST catalog#62988
tuantran0910 wants to merge 2 commits intoapache:masterfrom
tuantran0910:feat/iceberg-rest-google-auth

Conversation

@tuantran0910
Copy link
Copy Markdown

What problem does this PR solve?

Issue Number: close #62987

Currently, Doris's Iceberg REST catalog only supports none and oauth2 security types. Users running Google Cloud Lakehouse with Iceberg REST catalogs need a way to authenticate using Google's native GoogleAuthManager via Application Default Credentials (ADC).

Solution

This PR adds a new google security type to IcebergRestProperties that configures the Iceberg REST client to use org.apache.iceberg.gcp.auth.GoogleAuthManager for authentication, following the official Google Cloud Lakehouse documentation.

Main changes:

  • Added GOOGLE to the Security enum in IcebergRestProperties
  • Added addGoogleProperties() method that sets rest.auth.type to the fully qualified GoogleAuthManager class name
  • Added new connector properties: iceberg.rest.io-impl, iceberg.rest.google.user-project, iceberg.gcs.oauth2.token
  • Added explicit iceberg-gcp and google-cloud-storage dependencies to fe-core/pom.xml
  • Added 5 new test cases covering Google auth configuration

How to use

CREATE CATALOG lakehouse PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "rest",
    "iceberg.rest.uri" = "https://biglake.googleapis.com/iceberg/v1/restcatalog",
    "iceberg.rest.security.type" = "google",
    "iceberg.rest.io-impl" = "org.apache.iceberg.gcp.gcs.GCSFileIO",
    "iceberg.rest.google.user-project" = "my-billing-project",
    "warehouse" = "gs://my-bucket/warehouse/"
);

Release note

Features

  • [Iceberg] Support Google Authentication (GoogleAuthManager) for Iceberg REST catalog

…alog

Add support for Google Cloud Lakehouse Iceberg REST catalog by
introducing a new `google` security type that leverages Iceberg's
built-in GoogleAuthManager for authentication via Application Default
Credentials.

New properties:
- iceberg.rest.security.type = google
- iceberg.rest.io-impl for FileIO implementation
- iceberg.rest.google.user-project for billing project
- iceberg.gcs.oauth2.token for GCS storage access

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@hello-stephen
Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@tuantran0910
Copy link
Copy Markdown
Author

run buildall

@tuantran0910
Copy link
Copy Markdown
Author

/review

@tuantran0910
Copy link
Copy Markdown
Author

run buildall

@tuantran0910
Copy link
Copy Markdown
Author

/review

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-H: Total hot run time: 29590 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit be06db44195f17be6f293ab6611ed574dfd6b4ba, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17773	3865	3857	3857
q2	q3	10717	873	597	597
q4	4661	460	343	343
q5	7448	1340	1153	1153
q6	185	168	137	137
q7	905	953	743	743
q8	9310	1375	1276	1276
q9	5607	5347	5300	5300
q10	6243	2081	1791	1791
q11	464	264	259	259
q12	628	420	295	295
q13	18115	3265	2773	2773
q14	294	283	259	259
q15	q16	911	864	792	792
q17	913	1041	704	704
q18	6402	5777	5646	5646
q19	1223	1252	1103	1103
q20	543	396	283	283
q21	4804	2352	1960	1960
q22	481	399	319	319
Total cold run time: 97627 ms
Total hot run time: 29590 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4720	4629	4828	4629
q2	q3	4637	4777	4199	4199
q4	2072	2135	1390	1390
q5	4984	4993	5267	4993
q6	193	169	132	132
q7	2057	1793	1660	1660
q8	3334	3090	3117	3090
q9	8365	8431	8389	8389
q10	4471	4558	4231	4231
q11	619	418	400	400
q12	731	735	522	522
q13	3221	3657	2879	2879
q14	299	306	269	269
q15	q16	789	926	706	706
q17	1311	1313	1294	1294
q18	7938	7141	7087	7087
q19	1183	1163	1138	1138
q20	2233	2266	1945	1945
q21	6051	5485	4810	4810
q22	549	491	423	423
Total cold run time: 59757 ms
Total hot run time: 54186 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-DS: Total hot run time: 170619 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit be06db44195f17be6f293ab6611ed574dfd6b4ba, data reload: false

query5	4314	671	517	517
query6	332	222	209	209
query7	4223	586	323	323
query8	321	248	225	225
query9	8816	3966	3993	3966
query10	448	338	304	304
query11	5782	2391	2231	2231
query12	188	131	128	128
query13	1286	608	454	454
query14	5993	5394	5086	5086
query14_1	4362	4390	4366	4366
query15	216	202	185	185
query16	1014	462	455	455
query17	1171	746	618	618
query18	2465	474	357	357
query19	220	200	161	161
query20	140	131	130	130
query21	210	136	118	118
query22	13593	13598	14811	13598
query23	17155	16509	16277	16277
query23_1	16251	16299	16217	16217
query24	7435	1769	1317	1317
query24_1	1354	1353	1389	1353
query25	576	491	420	420
query26	1294	325	160	160
query27	2778	635	329	329
query28	4387	1959	1958	1958
query29	1002	618	508	508
query30	302	234	195	195
query31	1096	1075	935	935
query32	83	71	70	70
query33	540	334	279	279
query34	1162	1125	639	639
query35	764	769	658	658
query36	1310	1408	1161	1161
query37	146	99	85	85
query38	3207	3067	3013	3013
query39	957	922	878	878
query39_1	878	913	866	866
query40	236	153	134	134
query41	62	59	58	58
query42	112	105	107	105
query43	315	331	284	284
query44	
query45	202	204	189	189
query46	1029	1199	722	722
query47	2352	2296	2257	2257
query48	391	441	310	310
query49	637	510	431	431
query50	706	289	213	213
query51	4280	4266	4175	4175
query52	104	104	97	97
query53	250	273	208	208
query54	306	265	250	250
query55	93	89	86	86
query56	297	306	333	306
query57	1450	1395	1325	1325
query58	289	267	266	266
query59	1552	1600	1424	1424
query60	348	338	331	331
query61	156	160	155	155
query62	677	616	571	571
query63	243	195	207	195
query64	2423	812	687	687
query65	
query66	1751	515	384	384
query67	30009	29919	29919	29919
query68	
query69	470	338	305	305
query70	1009	973	996	973
query71	307	282	270	270
query72	3247	2928	2624	2624
query73	827	771	405	405
query74	5080	4881	4717	4717
query75	2803	2712	2346	2346
query76	2292	1142	750	750
query77	435	430	367	367
query78	12896	12953	12275	12275
query79	1487	970	748	748
query80	802	567	469	469
query81	479	275	237	237
query82	1270	159	121	121
query83	338	274	257	257
query84	257	141	107	107
query85	906	529	442	442
query86	428	328	327	327
query87	3429	3328	3192	3192
query88	3545	2658	2657	2657
query89	438	384	336	336
query90	1789	190	178	178
query91	174	168	137	137
query92	81	77	78	77
query93	967	972	563	563
query94	589	353	301	301
query95	670	370	429	370
query96	985	786	319	319
query97	2725	2708	2608	2608
query98	242	230	226	226
query99	1104	1129	1018	1018
Total cold run time: 253077 ms
Total hot run time: 170619 ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support Google Authentication for Iceberg REST Catalog

2 participants