Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Spark] Implement Purge table for SparkCatalog #3015

Open
wants to merge 82 commits into
base: master
Choose a base branch
from
Open

[Spark] Implement Purge table for SparkCatalog #3015

wants to merge 82 commits into from

Commits on Mar 15, 2024

  1. [PAIMON-1401] [Feature] Implement Purge table for SparkCatalog

    @zhu3pang
    zhu3pang committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    5bf3fc0 View commit details
    Browse the repository at this point in the history

Commits on Mar 29, 2024

  1. [hotfix] Fix unstable testRandomCdcEventsUnawareBucket (apache#3014)

    @yuzelin @zhu3pang
    yuzelin authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    db88321 View commit details
    Browse the repository at this point in the history

  2. [flink] Refactor: flink tests use common CatalogITCaseBase (apache#3016)

    @yuzelin @zhu3pang
    yuzelin authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    94a15a0 View commit details
    Browse the repository at this point in the history

  3. [core] Support compiled class cache in CodeGenUtils 

    This closes apache#2976
    @xiangyuf @zhu3pang
    xiangyuf authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    dd0ef63 View commit details
    Browse the repository at this point in the history

  4. [core] Optimize compiled class cache in CodeGenUtils

    @JingsongLi @zhu3pang
    JingsongLi authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    dbdcffd View commit details
    Browse the repository at this point in the history

  5. [core] AvroBulkFormat should not get file-size by file-io repeatly (a… 

    …pache#3019)
    @leaves12138 @zhu3pang
    leaves12138 authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    3939cd3 View commit details
    Browse the repository at this point in the history

  6. [hive] Cache TableSchema into Configuration to avoid loading read sch… 

    …eme file in PaimonSerDe (apache#2946)
    @wg1026688210 @zhu3pang
    wg1026688210 authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    ae4ecc0 View commit details
    Browse the repository at this point in the history

  7. [core] Support configuring lock in paimon catalog (apache#2933) 

    * [core] Support configuring lock in paimon catalog
    @FangYongs @zhu3pang
    FangYongs authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    4508589 View commit details
    Browse the repository at this point in the history

  8. [codegen] EqualiserCodeGenerator supports ARRAY<ROW> (apache#3023)

    @yuzelin @zhu3pang
    yuzelin authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    c647733 View commit details
    Browse the repository at this point in the history

  9. [core] Unified Naming of JdbcCatalog Fields (apache#3025)

    @sunxiaojian @zhu3pang
    sunxiaojian authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    a30275f View commit details
    Browse the repository at this point in the history

  10. [hotfix] upgrade json-path and avro version (apache#2987)

    @zhuangchong @zhu3pang
    zhuangchong authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    d0f684e View commit details
    Browse the repository at this point in the history

  11. [core] Dv table supports value filter pushdown (apache#3024)

    @Zouxxyy @zhu3pang
    Zouxxyy authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    55304dc View commit details
    Browse the repository at this point in the history

  12. [hive] Fix hive writer in tez-mr (apache#2954)

    @Alibaba-HZY @zhu3pang
    Alibaba-HZY authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    edbc555 View commit details
    Browse the repository at this point in the history

  13. [hive] Add license for TezUtil

    @JingsongLi @zhu3pang
    JingsongLi authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    aea5509 View commit details
    Browse the repository at this point in the history

  14. [core] Reduce schema read for readers (apache#3021)

    @JingsongLi @zhu3pang
    JingsongLi authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    c2b2cc8 View commit details
    Browse the repository at this point in the history

  15. [core] Add more validation for sequence field during create table. (a… 

    …pache#3022)
    @hzjhjjyy @zhu3pang
    hzjhjjyy authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    b86a8a6 View commit details
    Browse the repository at this point in the history

  16. [doc] Updated Spark quickstart documentation for using --packages (ap… 

    …ache#2897)
    @cxzl25 @zhu3pang
    cxzl25 authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    5712563 View commit details
    Browse the repository at this point in the history

  17. [core] Use var length encoding for row position (apache#3031)

    @JingsongLi @zhu3pang
    JingsongLi authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    66eb936 View commit details
    Browse the repository at this point in the history

  18. [oss] use multi-release mechanism to ship jaxb (apache#2658)

    @zhoulii @zhu3pang
    zhoulii authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    a0817bd View commit details
    Browse the repository at this point in the history

  19. [core] Data files with delete records should not be upgraded directly… 

    … to max level (apache#2962)
    @tsreaper @zhu3pang
    tsreaper authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    731f3ef View commit details
    Browse the repository at this point in the history

  20. [Flink] Introduce range strategy for sort compaction. (apache#2749)

    @wg1026688210 @zhu3pang
    wg1026688210 authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    b88ba74 View commit details
    Browse the repository at this point in the history

  21. [spark] spark sql support get/list function (apache#2880)

    @waywtdcc @zhu3pang
    waywtdcc authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    e76d242 View commit details
    Browse the repository at this point in the history

  22. [docs] Trino docs update for sharing file system. (apache#3035)

    @leaves12138 @zhu3pang
    leaves12138 authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    1e22661 View commit details
    Browse the repository at this point in the history

  23. [core] Fix dv table with partial-update and aggregate (apache#3036)

    @Zouxxyy @zhu3pang
    Zouxxyy authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    b8ebe59 View commit details
    Browse the repository at this point in the history

  24. [core] Introduce BatchTableCommit.truncateTable (apache#3037)

    @JingsongLi @zhu3pang
    JingsongLi authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    5dacfb5 View commit details
    Browse the repository at this point in the history

  25. [core] orc/parquet reader obtain the fileSize from metadata (apache#2918 

    )
    @wg1026688210 @zhu3pang
    wg1026688210 authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    ede603e View commit details
    Browse the repository at this point in the history

  26. [core] Optimize write performance (apache#3039)

    @JingsongLi @zhu3pang
    JingsongLi authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    abc5b8c View commit details
    Browse the repository at this point in the history

  27. [core] Reduce getFileSize for avro reader (apache#3040)

    @JingsongLi @zhu3pang
    JingsongLi authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    5d5dcc4 View commit details
    Browse the repository at this point in the history

  28. [core] support create tag based on tag (apache#3044)

    @Aitozi @zhu3pang
    Aitozi authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    4df3140 View commit details
    Browse the repository at this point in the history

  29. [flink] Remove compatibility utils that are used for flink-1.14 (apac… 

    …he#3054)
    @yuzelin @zhu3pang
    yuzelin authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    b721428 View commit details
    Browse the repository at this point in the history

  30. [core] Add validation for fields prefix (apache#3052)

    @Zouxxyy @zhu3pang
    Zouxxyy authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    453d1e0 View commit details
    Browse the repository at this point in the history

  31. [core] add initial map size to solve load hash index slowly (apache#3051 

    )
    @Stephen0421 @zhu3pang
    Stephen0421 authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    77f050c View commit details
    Browse the repository at this point in the history

  32. [core] Introduce Int2ShortHashMap.Builder to to accelerate init

    @JingsongLi @zhu3pang
    JingsongLi authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    ea88e87 View commit details
    Browse the repository at this point in the history

  33. [core] support to drop partition when delete tag (apache#3042)

    @Aitozi @zhu3pang
    Aitozi authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    32378ad View commit details
    Browse the repository at this point in the history

  34. [core] Optimize first_row batch read (apache#3055)

    @JingsongLi @zhu3pang
    JingsongLi authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    3ebe41a View commit details
    Browse the repository at this point in the history

  35. [flink] Bump flink version to 1.19 (apache#3049)

    @yuzelin @zhu3pang
    yuzelin authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    961bd64 View commit details
    Browse the repository at this point in the history

  36. [orc] ORC Support ZStandard compression using zstd-jni (apache#3056)

    @zyl891229 @zhu3pang
    zyl891229 authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    e001946 View commit details
    Browse the repository at this point in the history

  37. [license] Add license to orc copied files

    @JingsongLi @zhu3pang
    JingsongLi authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    ba8e999 View commit details
    Browse the repository at this point in the history

  38. [doc] Introduce orc options documentation page (apache#3061)

    @JingsongLi @zhu3pang
    JingsongLi authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    a9ace38 View commit details
    Browse the repository at this point in the history

  39. [doc] Add flink 1.19 to the documentation. (apache#3064)

    @zhuangchong @zhu3pang
    zhuangchong authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    e38916e View commit details
    Browse the repository at this point in the history

  40. [hotfix] Remove redundant and unused test code (apache#3066)

    @yuzelin @zhu3pang
    yuzelin authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    df25ef2 View commit details
    Browse the repository at this point in the history

  41. [doc] Fix version in trino documentation (apache#3072)

    @Zouxxyy @zhu3pang
    Zouxxyy authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    332a994 View commit details
    Browse the repository at this point in the history

  42. [core] Skip rewrite when delete row count is 0 with dv (apache#3070)

    @Zouxxyy @zhu3pang
    Zouxxyy authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    92f0ad5 View commit details
    Browse the repository at this point in the history

  43. [hotfix] Add project root path to files to facilitate compiling modul… 

    …es in child dirs (apache#3063)
    @Zouxxyy @zhu3pang
    Zouxxyy authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    7559efa View commit details
    Browse the repository at this point in the history

  44. [flink] Compact procedure does not process 'ALL' word (apache#3065)

    @leaves12138 @zhu3pang
    leaves12138 authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    2aad966 View commit details
    Browse the repository at this point in the history

  45. [fix] Fix unstable test TagAutoCreationTest#testModifyTagPeriod (apac… 

    …he#3073)
    @yuzelin @zhu3pang
    yuzelin authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    6575b81 View commit details
    Browse the repository at this point in the history

  46. [spark] Minor refactor SparkWriter to separate commit (apache#3067)

    @YannByron @zhu3pang
    YannByron authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    5579dee View commit details
    Browse the repository at this point in the history

  47. [flink] Fix that compact actions haven't handled scan parallelism (ap… 

    …ache#3077)
    @yuzelin @zhu3pang
    yuzelin authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    afa1b10 View commit details
    Browse the repository at this point in the history

  48. [core] JdbcCatalog supports custom configuration for the length of lo… 

    …ck table (apache#3069)
    @sunxiaojian @zhu3pang
    sunxiaojian authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    f16ccdd View commit details
    Browse the repository at this point in the history

  49. [doc] Introduce the minimum version of Maven. (apache#3082)

    @wg1026688210 @zhu3pang
    wg1026688210 authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    5bc04c6 View commit details
    Browse the repository at this point in the history

  50. [doc] Apache Paimon became top level project. (apache#3085)

    @leaves12138 @zhu3pang
    leaves12138 authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    ee4fa5d View commit details
    Browse the repository at this point in the history

  51. [incubating] Remove incubator disclaimer

    @JingsongLi @zhu3pang
    JingsongLi authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    ece3375 View commit details
    Browse the repository at this point in the history

  52. [flink] Compact procedure supports named arguments (apache#3078)

    @yuzelin @zhu3pang
    yuzelin authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    e745c2a View commit details
    Browse the repository at this point in the history

  53. [cdc] Update the latest dynamic options to the table schema file when… 

    … synchronizing an existing paimon table using the cdc action (apache#3062)
    @zhuangchong @zhu3pang
    zhuangchong authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    5c78be1 View commit details
    Browse the repository at this point in the history

  54. [flink] Fix performance issue in ContinuousFileSplitEnumerator (apach… 

    …e#3071)
    @yuzelin @zhu3pang
    yuzelin authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    de61a18 View commit details
    Browse the repository at this point in the history

  55. [Refactor] Minor refactor update the latest dynamic options using the… 

    … cdc action. (apache#3090)
    @yuzelin @zhu3pang
    yuzelin authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    0939a82 View commit details
    Browse the repository at this point in the history

  56. [cdc] Extract common code (apache#3050)

    @zhangjun0x01 @zhu3pang
    zhangjun0x01 authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    8ce013c View commit details
    Browse the repository at this point in the history

  57. [cdc] Fix Mysql sync_table unable to synchronize newly added tables w… 

    …ith regular expression (apache#2851)
    @huyuanfeng2018 @zhu3pang
    huyuanfeng2018 authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    7e33038 View commit details
    Browse the repository at this point in the history

  58. [hotfix] Apply spotless to fix checkstyle violation in flink-cdc modu… 

    …le (apache#3092)
    @yuzelin @zhu3pang
    yuzelin authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    effc57e View commit details
    Browse the repository at this point in the history

  59. [test] Append only table can't test failing IO re-insert yet (apache#… 

    …3089)
    @leaves12138 @zhu3pang
    leaves12138 authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    2e65cc5 View commit details
    Browse the repository at this point in the history

  60. [core] Introduce SplitGroup for SplitGenerator to optimize more rawFi… 

    …les (apache#3059)
    @Zouxxyy @zhu3pang
    Zouxxyy authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    8ce82a7 View commit details
    Browse the repository at this point in the history

  61. [hotfix] Fix unstable test MySqlSyncTableActionITCase#testSyncShards (a… 

    …pache#3096)
    @yuzelin @zhu3pang
    yuzelin authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    f2a4e38 View commit details
    Browse the repository at this point in the history

  62. [core] Refactor FileStorePathFactory to clean test methods (apache#3093)

    @JingsongLi @zhu3pang
    JingsongLi authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    c928967 View commit details
    Browse the repository at this point in the history

  63. [core] Add TableFormatBenchmark to test format only

    @JingsongLi @zhu3pang
    JingsongLi authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    096c433 View commit details
    Browse the repository at this point in the history

  64. [spark][core] Support input_file_name UDF (apache#3094)

    @YannByron @zhu3pang
    YannByron authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    cbe121b View commit details
    Browse the repository at this point in the history

  65. [core] Unify FileRecordReader and reduce file access (apache#3098)

    @JingsongLi @zhu3pang
    JingsongLi authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    7f27f04 View commit details
    Browse the repository at this point in the history

  66. [core] Support common jdbc catalog lock for filesystem catalog. (apac… 

    …he#3076)
    @sunxiaojian @zhu3pang
    sunxiaojian authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    59d9ec7 View commit details
    Browse the repository at this point in the history

  67. [core] Fix OrphanFilesClean has a chance to delete normal file (apach… 

    …e#3075)
    @wxplovecc @zhu3pang
    wxplovecc authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    78b1ace View commit details
    Browse the repository at this point in the history

  68. [catalog] Refactor Catalog Factory and revert jdbc lock in FileSystem… 

    …Catalog (apache#3099)
    @JingsongLi @zhu3pang
    JingsongLi authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    4c17a3a View commit details
    Browse the repository at this point in the history

  69. [core] Shade roaringbitmap dependency into paimon-common (apache#3100)

    @Zouxxyy @zhu3pang
    Zouxxyy authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    a0809a2 View commit details
    Browse the repository at this point in the history

  70. [cdc] Fix cdc job mistakenly changes immutable options of existing ta… 

    …ble (apache#3095)
    @zhuangchong @zhu3pang
    zhuangchong authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    ac57e65 View commit details
    Browse the repository at this point in the history

  71. [core] Support dv with avro format (apache#3105)

    @Zouxxyy @zhu3pang
    Zouxxyy authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    c265c6f View commit details
    Browse the repository at this point in the history

  72. [core] Introduce File index format for data skipping (apache#3068)

    @leaves12138 @zhu3pang
    leaves12138 authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    22832e9 View commit details
    Browse the repository at this point in the history

  73. [test] Fix flaky test in `KeyValueFileStoreScanTest.testWithValueFilt… 

    …er` (apache#3109)
    @Zouxxyy @zhu3pang
    Zouxxyy authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    a5eff5f View commit details
    Browse the repository at this point in the history

  74. [doc] Introduce deletion vector documentation page (apache#3107)

    @Zouxxyy @zhu3pang
    Zouxxyy authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    e110f57 View commit details
    Browse the repository at this point in the history

  75. [core] Rename to compactDvIndexFiles in FileStoreCommitImpl

    @JingsongLi @zhu3pang
    JingsongLi authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    4b99a3f View commit details
    Browse the repository at this point in the history

  76. [test] Fix test in KeyValueFileStoreScanTest, make it more accurate (a… 

    …pache#3111)
    @leaves12138 @zhu3pang
    leaves12138 authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    5fa79b8 View commit details
    Browse the repository at this point in the history

  77. [format] Set orc.compression.zstd.level to 3 by default (apache#3114)

    @zyl891229 @zhu3pang
    zyl891229 authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    9a89253 View commit details
    Browse the repository at this point in the history

  78. [core] add deletedFiles in NewFilesIncrement (apache#3117)

    @YannByron @zhu3pang
    YannByron authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    a9ff40a View commit details
    Browse the repository at this point in the history

  79. [cdc] Fix that kafka message value might be null which causes NPE (ap… 

    …ache#3113)
    @yuzelin @zhu3pang
    yuzelin authored and zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    fa9c6e7 View commit details
    Browse the repository at this point in the history

  80. [PAIMON-1401] [Feature] Implement Purge table for SparkCatalog, add s… 

    …ome UTs
    @zhu3pang
    zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    ea3797d View commit details
    Browse the repository at this point in the history

  81. Merge remote-tracking branch 'origin/master' into PAIMON-1401

    @zhu3pang
    zhu3pang committed Mar 29, 2024
    Configuration menu
    Copy the full SHA
    9db339a View commit details
    Browse the repository at this point in the history