Added support for add column and drop column #41

subkanthi · 2025-06-30T18:49:34Z

closes: #39

subkanthi · 2025-07-01T19:53:27Z

ice/src/main/java/com/altinity/ice/cli/internal/cmd/AddColumn.java

+    Types.NestedField field;
+
+    switch (type.toLowerCase()) {
+      case "string":


couldn't find any mapper from string to iceberg Types

subkanthi · 2025-07-01T21:06:16Z

ice/src/main/java/com/altinity/ice/cli/internal/cmd/AddColumn.java

+    return new ColumnSpec(columnName, field.type(), comment);
+  }
+
+  private static Types.NestedField parseColumnType(String name, String type, String comment) {


maybe this needs to be moved to a common class for field mapping.

shyiko

I think we should go with @aadant's proposal #39 (comment). Separate add-column/delete-column commands instead of single alter-table means you can't combine multiple changes in a single transaction.

… as arguments.

subkanthi · 2025-07-10T21:03:43Z

alter-table flowers.iris --operations '[{"drop": "age"}]'
2025-07-10 15:41:35 [main] INFO c.a.i.c.i.cmd.AlterTable > Dropping column 'age' from table: flowers.iris
2025-07-10 15:41:36 [main] INFO c.a.i.c.i.cmd.AlterTable > Successfully applied 1 operations to table: flowers.iris

Invalid operation

alter-table flowers.iris --operations '[{"rename": "age"}]'
2025-07-10 15:54:17 [main] ERROR com.altinity.ice.cli.Main > Fatal
java.lang.IllegalArgumentException: Invalid operation. Supported operations: add, drop
        at com.altinity.ice.cli.internal.cmd.AlterTable.validateOperation(AlterTable.java:117)
        at com.altinity.ice.cli.internal.cmd.AlterTable.run(AlterTable.java:65)
        at com.altinity.ice.cli.Main.alterTable(Main.java:459)
        at com.altinity.ice.cli.Main.lambda$main$0(Main.java:551) [9 skipped]
        at com.altinity.ice.cli.Main.main(Main.java:558) [1 skipped]

shyiko · 2025-07-11T19:41:29Z

ice/src/main/java/com/altinity/ice/cli/Main.java

+              names = {"--operations"},
+              required = true,
+              description =
+                  "JSON array of operations: [{'add': 'age:int:User age'}, {'drop': 'col_name'}]")


It would be best to stick to yaml/json for this as column names may contain colons, future operations may not be so easy to express in colon-separated values, etc.

It would also be nice if operation/files names would follow https://iceberg.apache.org/docs/latest/spark-ddl/#alter-table-rename-column naming.

alter-table flowers.iris --operations '[{"operation": "add column", "column_name": "age", "type": "int", "comment": "user age"}]' 2025-07-16 16:12:24 [main] INFO c.a.i.c.i.cmd.AlterTable > Adding column 'age' to table: flowers.iris 2025-07-16 16:12:24 [main] INFO c.a.i.c.i.cmd.AlterTable > Successfully applied 1 operations to table: flowers.iris

ice alter-table flowers.iris --operations '[{"operation": "drop column", "column_name": "age"}]' 2025-07-16 16:12:48 [main] INFO c.a.i.c.i.cmd.AlterTable > Dropping column 'age' from table: flowers.iris 2025-07-16 16:12:49 [main] INFO c.a.i.c.i.cmd.AlterTable > Successfully applied 1 operations to table: flowers.iris

alter-table flowers.iris --operations '[{"operation": "drop column", "column_name": "age"}, {"operation": "add column", "column_name": "marital_status", "type": "string", "comment": "Marital Status"}]' 2025-07-16 16:35:47 [main] INFO c.a.i.c.i.cmd.AlterTable > Dropping column 'age' from table: flowers.iris 2025-07-16 16:35:47 [main] INFO c.a.i.c.i.cmd.AlterTable > Adding column 'marital_status' to table: flowers.iris 2025-07-16 16:35:47 [main] INFO c.a.i.c.i.cmd.AlterTable > Successfully applied 2 operations to table: flowers.iris

…tions name

shyiko · 2025-08-18T17:57:20Z

Can you please confirm that this scenario works:

insert parquet file #1 containing N columns
run ice alter-table to replace last column with a different one (i.e. delete X + add Y)
insert parquet file #2 that contains a new set of columns. confirm it gets inserted and data from both files can still be queried

…-to-an-existing-table

subkanthi · 2025-08-20T02:25:19Z

Can you please confirm that this scenario works:

insert parquet file #1 containing N columns

run ice alter-table to replace last column with a different one (i.e. delete X + add Y)

insert parquet file #2 that contains a new set of columns. confirm it gets inserted and data from both files can still be queried

CREATE TABLE employees (id INTEGER, name VARCHAR, department VARCHAR, yearsInService INTEGER);
INSERT INTO employees VALUES (1,'John Smith','Engineering',5),(2,'Sarah Johnson','Marketing',3),(3,'Mike Davis','Engineering',7),(4,'Emma Wilson','HR',2),(5,'James Brown','Sales',4),(6,'Lisa Anderson','Engineering',6),(7,'Robert Taylor','Marketing',1),(8,'Maria Garcia','HR',3),(9,'David Martinez','Sales',5),(10,'Jennifer Lee','Engineering',8),(11,'Chris Wilson','Engineering',4),(12,'Amanda White','Sales',2),(13,'Kevin Thompson','Marketing',6),(14,'Michelle Harris','HR',3),(15,'Brian Clark','Engineering',9),(16,'Nancy Lewis','Sales',1),(17,'Steven Walker','Marketing',5),(18,'Laura Hall','HR',7),(19,'Gary Allen','Engineering',3),(20,'Sandra Young','Sales',4);
COPY employees TO 'employees_with_years.parquet' (FORMAT PARQUET);
SELECT * FROM 'employees_with_years.parquet' LIMIT 5;
D CREATE TABLE employees (
      id INTEGER,
      name VARCHAR,
      department VARCHAR,
      salary DECIMAL(10,2)
  );

D copy employees to 'employees.parquet' (FORMAT PARQUET);

ice  insert flowers.employees -p   file://employees.parquet

ice alter-table flowers.employees --operations '[{"operation": "drop column", "column_name": "salary"}, {"operation": "add column", "column_name": "yearsInService", "type": "int", "comment": "Years in service"}]'


D DROP TABLE IF EXISTS employees;
D CREATE TABLE employees (id INTEGER, name VARCHAR, department VARCHAR, yearsInService INTEGER);
D insert into employees values(55, 'John Foo', 'Engineering', 44);
D copy employees to 'employees_with_years.parquet' (format parquet);


**Exception**

insert flowers.employees -p   file://employees_with_years.parquet
2025-08-19 21:22:31 [52179791-79] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous GET v1/config
2025-08-19 21:22:31 [52179791-77] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous HEAD v1/namespaces/flowers/tables/employees
2025-08-19 21:22:31 [52179791-47] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous GET v1/namespaces/flowers/tables/employees
2025-08-19 21:22:31 [52179791-77] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous GET v1/namespaces/flowers/tables/employees
2025-08-19 21:22:32 [52179791-47] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous POST v1/namespaces/flowers/tables/employees/metrics
2025-08-19 21:22:32 [-6-thread-1] INFO c.a.i.c.internal.cmd.Insert > file://employees_with_years.parquet: processing
2025-08-19 21:22:32 [-6-thread-1] INFO c.a.i.c.internal.cmd.Insert > file://employees_with_years.parquet: jvm: heap: init=1008M,used=23M,committed=80M,free=15984M,max=16008M, gc pressure: 100.00%(27ms), off-heap: init=7M,used=51M,committed=54M, open fds: 10
2025-08-19 21:22:32 [main] ERROR com.altinity.ice.cli.Main > Fatal
java.io.IOException: Error processing file(s)
        at com.altinity.ice.cli.internal.cmd.Insert.run(Insert.java:230)
        at com.altinity.ice.cli.Main.insert(Main.java:364)
        at com.altinity.ice.cli.Main.lambda$main$0(Main.java:605) [9 skipped]
        at com.altinity.ice.cli.Main.main(Main.java:612) [1 skipped]
Caused by: java.io.IOException: Error processing file://employees_with_years.parquet
        at com.altinity.ice.cli.internal.cmd.Insert.lambda$run$3(Insert.java:215)
 [4 skipped]
Caused by: org.apache.iceberg.exceptions.BadRequestException: file://employees_with_years.parquet's schema doesn't match table's schema
        at com.altinity.ice.cli.internal.cmd.Insert.processFile(Insert.java:399)
        at com.altinity.ice.cli.internal.cmd.Insert.lambda$run$3(Insert.java:194)
        ... 4 common frames omitted

    MessageType type = metadata.getFileMetaData().getSchema();
    Schema fileSchema = ParquetSchemaUtil.convert(type); // nameMapping applied (when present)
    if (!sameSchema(table, fileSchema)) {
      throw new BadRequestException(
          String.format("%s's schema doesn't match table's schema", file));
    }

Added support for add column and drop column

2d7a0d6

subkanthi linked an issue Jun 30, 2025 that may be closed by this pull request

Add a column to an existing table #39

Open

subkanthi commented Jul 1, 2025

View reviewed changes

subkanthi marked this pull request as ready for review July 1, 2025 21:14

shyiko requested changes Jul 8, 2025

View reviewed changes

subkanthi added 2 commits July 10, 2025 13:59

Merged changes from main.

a4746bd

Changed add-column, drop-column to alter-table that takes a JSON list…

701a98e

… as arguments.

subkanthi requested a review from shyiko July 11, 2025 15:11

Removed AddColumn, DropColumn classes.

26e15c1

shyiko reviewed Jul 11, 2025

View reviewed changes

subkanthi added 5 commits July 14, 2025 09:11

Replaced semicolon logic to split value with JSON keys. changed opera…

9651e88

…tions name

Changed JSON for add/drop column to include operation JSON key.

c4d10b1

Replace hard coded strings in AlterTable

0d01d9e

Fixed unit test.

f76cbf3

Fixed CLI description.

530f16f

subkanthi requested a review from shyiko July 16, 2025 21:14

subkanthi removed a link to an issue Aug 15, 2025

Add a column to an existing table #39

Open

Merge branch 'master' of github.com:Altinity/ice into 39-add-a-column…

6ade046

…-to-an-existing-table

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Added support for add column and drop column #41

Added support for add column and drop column #41

Uh oh!

subkanthi commented Jun 30, 2025 •

edited

Loading

Uh oh!

subkanthi Jul 1, 2025

Uh oh!

subkanthi Jul 1, 2025

Uh oh!

shyiko left a comment •

edited

Loading

Uh oh!

subkanthi commented Jul 10, 2025

Uh oh!

shyiko Jul 11, 2025

Uh oh!

subkanthi Jul 16, 2025 •

edited

Loading

Uh oh!

shyiko commented Aug 18, 2025 •

edited

Loading

Uh oh!

subkanthi commented Aug 20, 2025 •

edited

Loading

Uh oh!

Uh oh!

Added support for add column and drop column #41

Are you sure you want to change the base?

Added support for add column and drop column #41

Uh oh!

Conversation

subkanthi commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

subkanthi Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

subkanthi Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

shyiko left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

subkanthi commented Jul 10, 2025

Uh oh!

shyiko Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

subkanthi Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shyiko commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

subkanthi commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

subkanthi commented Jun 30, 2025 •

edited

Loading

shyiko left a comment •

edited

Loading

subkanthi Jul 16, 2025 •

edited

Loading

shyiko commented Aug 18, 2025 •

edited

Loading

subkanthi commented Aug 20, 2025 •

edited

Loading