Release date: March 14, 2024

This is a patch release, containing an important bug fix while decoding the Prometheus remote write proto https (#3505).

Future critical bug fixes were delivered at v0.7.2. It's highly recommended to upgrade to v0.7.2 if you're using v0.7.

Breaking changes​

• fix!: remove error message from http header to avoid panic by @sunng87 in #3506

Bug fixes​

• fix: add support for influxdb basic auth by @shuiyisong in #3437
• fix: fix incorrect COM_STMT_PREPARE reply by @WenyXu in #3463
• fix: impl RecordBatchStream method explicitly by @shuiyisong in #3482
• fix: make max-txn-ops limit valid by @fengjiachun in #3481
• fix: fix f64 has no sufficient precision during parsing by @WenyXu in #3483
• fix: freeze data buffer in shard by @evenyag in #3468
• fix: allow passing extra table options by @evenyag in #3484
• fix(common-time): allow building nanos timestamp from parts split from i64::MIN by @zhongzc in #3493
• fix: adjust fill behavior of range query by @Taylor-lagrange in #3489
• fix: decoding prometheus remote write proto doesn't reset the value by @waynexia in #3505
• fix: correctly generate sequences when the value is pre-existed by @MichaelScofield in #3502

Improvements​

• feat: impl some "set"s to adapt to some client apps by @MichaelScofield in #3443
• feat: update dashboard to v0.4.8 by @ZonaHex in #3450
• feat: max-txn-ops option by @fengjiachun in #3458
• feat(influxdb): add db query param support for v2 write api by @etolbakov in #3445
• feat: support first_value/last_value in range query by @Taylor-lagrange in #3448
• feat: clamp function by @waynexia in #3465
• feat(fuzz): validate columns by @WenyXu in #3485
• feat: to_timezone function by @tisonkun in #3470
• feat(flow): accumulator for aggr func by @discord9 in #3396
• feat(flow): plan def by @discord9 in #3490
• feat: improve prom write requests decode performance by @v0y4g3r in #3478
• feat(fuzz): add insert target by @zhongzc in #3499
• feat(fuzz): add alter table target by @WenyXu in #3503
• feat: support decode gzip if influxdb write specify it by @tisonkun in #3494
• feat: Partition memtables by time if compaction window is provided by @evenyag in #3501

Others​

• build(deps): upgrade opendal to 0.45.1 by @tisonkun in #3432
• chore: add bin opt to build cmd by @WenyXu in #3440
• chore: specify binary name by @WenyXu in #3449
• test: add fuzz test for create table by @WenyXu in #3441
• refactor: separate the quote char and value by @WenyXu in #3455
• perf: Reduce decode overhead during pruning keys in the memtable by @evenyag in #3415
• chore: remove repetitive words by @gcmutator in #3469
• refactor: introduce new Output with OutputMeta by @shuiyisong in #3466
• refactor: make http api returns non-200 status code by @crwen in #3473
• ci: use another mirror for etcd image by @waynexia in #3486
• perf: more benchmarks for memtables by @evenyag in #3491
• refactor: validate constraints eagerly by @tisonkun in #3472
• ci: attempt to setup docker cache for etcd by @sunng87 in #3488
• chore: bump version to v0.7.1 by @evenyag in #3510
• docs: add v0.7 TSBS benchmark result by @evenyag in #3512
• ci: add bin options to reduce build burden by @WenyXu in #3518

Contributors​

We would like to thank the following contributors from the GreptimeDB community:

Eugene Tolbakov, JeremyHi, LFC, Lei HUANG, Ning Sun, Ruihang Xia, WU Jingdi, Weny Xu, Yingwen, Zhenchi, ZonaHe, crwen, discord9, gcmutator, shuiyisong, tison

Release date: March 12, 2024

This is a feature release, read the release blog to learn more detail about the feature and performance updates.

v0.7 represents a crucial leap toward achieving production readiness; it implements production-ready features for cloud-native monitoring scenarios.

Highlights

• Metric Engine: handles a large number of small tables, making it particularly suitable for cloud-native monitoring.
• Region Migration: enhances the user experience and simplifies region migrations with straightforward SQL commands.
• Inverted Index: dramatically improves the efficiency of locating data segments relevant to user queries, significantly reducing the IO operations needed for scanning data files and thus accelerating the query process.

Breaking changes​

• Switch prom remote write to metric engine by @waynexia in #3198
• Rename initialize_region_in_background to init_regions_in_background by @WenyXu in #3216
• New partition grammar in SQL parser by @waynexia in #3347

Bug fixes​

• fix: change back GREPTIME_DB_HEADER_NAME header key name by @shuiyisong in #3184
• fix(index): S3 EntityTooSmall error by @zhongzc in #3192
• fix: remove name matcher from processed matcher list by @waynexia in #3213
• fix: fix default value cannot accept negative number by @WenyXu in #3217
• fix: only register region keeper while creating physical table by @WenyXu in #3223
• fix: fix MockInstance rebuild issue by @WenyXu in #3218
• fix: security update for shlex and h2 by @sunng87 in #3227
• fix: fix create table ddl return incorrect table id by @WenyXu in #3232
• fix: init parquet reader metrics twice by @evenyag in #3242
• fix: IntermediateWriter closes underlying writer twice by @waynexia in #3248
• fix: decouple columns in projection and prune by @waynexia in #3253
• fix(Copy From): fix incorrect type casts by @WenyXu in #3264
• fix: SQL insertion and column default constraint aware of timezone by @killme2008 in #3266
• fix: cli export database default value by @tisonkun in #3259
• fix: use fe_opts after setup_frontend_plugins in standalone by @shuiyisong in #3275
• fix: fix incorrect StatusCode parsing by @WenyXu in #3281
• fix(util): join_path function should not trim leading / by @dalprahcd in #3280
• fix: correct the case sensitivity behavior for PromQL by @waynexia in #3296
• fix: $TARGET_BIN not found when docker run the image by @daviderli614 in #3297
• fix(index): sanitize S3 upload buffer size by @zhongzc in #3300
• fix: commit_short sqlness test case by @tisonkun in #3313
• fix: bump libgit2-sys from 0.16.1+1.7.1 to 0.16.2+1.7.2 by @dependabot in #3316
• fix: split write metadata request by @fengjiachun in #3311
• fix(index): encode string type to original data to enable fst regex to work by @zhongzc in #3324
• fix: disable ansi control char when stdout is redirected by @waynexia in #3332
• fix: typo in lint config by @waynexia in #3358
• fix: treat "0" and "1" as valid boolean values. by @MichaelScofield in #3370
• fix: remove unused imports in memtable_util.rs by @v0y4g3r in #3376
• fix: resets dict builder keys counter and avoid unnecessary pruning by @evenyag in #3386
• fix: throw errors instead of panic by @WenyXu in #3391
• fix: some read metrics by @v0y4g3r in #3404
• fix: partition region id by @fengjiachun in #3414
• fix: show table names not complete from information_schema by @killme2008 in #3417
• fix: mitigate memory spike during startup by @niebayes in #3418
• fix: complete interceptors for all frontend entry by @shuiyisong in #3428

Improvements​

• feat: add modulo function by @etolbakov in #3147
• feat: support HTTP&gRPC&pg set timezone by @Taylor-lagrange in #3125
• feat(mito): enable inverted index by @zhongzc in #3158
• feat: upgrade pgwire to 0.19 by @sunng87 in #3157
• feat: let tables API return a stream by @fengjiachun in #3170
• feat: add tests-fuzz crate by @WenyXu in #3173
• feat: tables stream with CatalogManager by @fengjiachun in #3180
• feat: adds date_format function by @killme2008 in #3167
• feat: auto config cache size according to memory size by @QuenKar in #3165
• feat: precise filter for mito parquet reader by @waynexia in #3178
• feat: adds parse options for SQL parser by @killme2008 in #3193
• feat(tests-fuzz): add CreateTableExprGenerator & AlterTableExprGenerator by @WenyXu in #3182
• feat: make procedure able to return output by @WenyXu in #3201
• feat: copy database from by @v0y4g3r in #3164
• feat: introduce information schema provider cache by @WenyXu in #3208
• feat: lazy initialize vector builder on write by @waynexia in #3210
• feat: make query be aware of timezone setting by @killme2008 in #3175
• feat: add create alter table expr translator by @WenyXu in #3203
• feat: read column and region info from state cache by @waynexia in #3222
• feat: enable concurrent write by @WenyXu in #3214
• feat: add Arrow IPC output format for http rest api by @sunng87 in #3177
• feat: change Range Query’s default align behavior aware of timezone by @killme2008 in #3219
• feat: update dashboard to v0.4.7 by @ZonaHex in #3229
• feat: http sql api return schema on empty resultset by @sunng87 in #3237
• feat: add pg create alter table expr translator by @WenyXu in #3206
• feat: read metadata from write cache by @QuenKar in #3224
• feat: batch create ddl by @fengjiachun in #3194
• feat: add insert/select generator & translator by @WenyXu in #3240
• feat: return request outdated error on handling alter by @waynexia in #3239
• feat: don't map semantic type in metric engine by @waynexia in #3243
• feat: create tables in batch on prom write by @fengjiachun in #3246
• feat: Only allow inserts and deletes operations to be executed in parallel by @fengjiachun in #3257
• feat: basic types for Dataflow Framework by @discord9 in #3186
• feat: initial configuration for grafana dashboard by @waynexia in #3263
• feat: support fraction part in timestamp by @discord9 in #3272
• feat: use simple filter to prune memtable by @waynexia in #3269
• feat: Basic Definitions for Expression&Functions for Dataflow by @discord9 in #3267
• feat: support cache for batch_get in CachedMetaKvBackend by @fengys1996 in #3277
• feat: put all filter exprs in a filter plan separately by @waynexia in #3288
• feat: administration functions by @killme2008 in #3236
• feat(mito): adjust seg size of inverted index to finer granularity instead of row group level by @zhongzc in #3289
• feat(mito): add options to ignore building index for specific column ids by @zhongzc in #3295
• feat: organize tracing on query path by @waynexia in #3310
• feat: impl partitions and region_peers information schema by @killme2008 in #3278
• feat: batch get physical table routes by @fengjiachun in #3319
• feat: Defines structs in the merge tree memtable by @evenyag in #3326
• feat(metric-engine): set index options for data region by @zhongzc in #3330
• feat: data buffer and related structs by @v0y4g3r in #3329
• feat: Implement KeyDictBuilder for the merge tree memtable by @evenyag in #3334
• feat: replace pk index with pk_weight during freeze by @v0y4g3r in #3343
• feat: merge tree data parts by @v0y4g3r in #3346
• feat: Defines more structs and methods for a partitioned merge tree by @evenyag in #3348
• feat(flow): impl ScalarExpr&Scalar Function by @discord9 in #3283
• feat: impl migrate_region and procedure_state SQL function by @killme2008 in #3325
• feat: add isnull function by @KKould in #3360
• feat: skip filling NULL for put and delete requests by @waynexia in #3364
• feat: impl merge reader for DataParts by @v0y4g3r in #3361
• feat: Implement write and fork for the new memtable by @evenyag in #3357
• feat: distinguish between different read paths by @v0y4g3r in #3369
• feat: Add freeze and fork method to the memtable by @evenyag in #3374
• feat: merge tree dedup reader by @v0y4g3r in #3375
• feat: Implement iter for the new memtable by @evenyag in #3373
• feat: Implement dedup for the new memtable and expose the config by @evenyag in #3377
• feat: change how region id maps to region worker by @waynexia in #3384
• feat: make tls certificates/keys reloadable (part 1) by @sunng87 in #3335
• feat(grafana): enable shared tooltip, add raft engine throughput by @waynexia in #3387
• feat: Implement partition eviction and only add value size to write buffer size by @evenyag in #3393
• feat: enable zstd compression and encodings in merge tree data part by @v0y4g3r in #3380
• feat(flow): impl for MapFilterProject by @discord9 in #3359
• feat: flush or compact table and region functions by @killme2008 in #3363
• feat: Use a partition level map to look up pk index by @evenyag in #3400
• feat(index): measure memory usage in global instead of single-column and add metrics by @zhongzc in #3383
• feat: enable ArrowFlight compression by @tisonkun in #3403
• feat: zero copy on split rows by @fengjiachun in #3407
• feat: Support automatic DNS lookup for kafka bootstrap servers by @J0HN50N133 in #3379
• feat: add configuration for tls watch option by @sunng87 in #3395
• feat: employ sparse key encoding for shard lookup by @v0y4g3r in #3410
• feat: support Create Table ... Like by @KKould in #3372
• feat: tableref cache by @fengjiachun in #3420
• feat: add verbose support for tql explain/analyze by @etolbakov in #3390
• feat: reduce a clone of string by @fengjiachun in #3422
• feat: Correct server metrics and add more metrics for scan by @evenyag in #3426
• feat: support tracing rule sampler by @Taylor-lagrange in #3405
• feat: decode prom requests to grpc by @v0y4g3r in #3425
• feat: implement multi-dim partition rule by @waynexia in #3409

Contributors​

We would like to thank the following contributors from the GreptimeDB community:

Cancai Cai, Eugene Tolbakov, Hudson C. Dalprá, JeremyHi, JohnsonLee, Kould, LFC, Lei HUANG, Ning Sun, Ruihang Xia, SteveLauC, WU Jingdi, Wei, Weny Xu, Yingwen, Yiran, Zhenchi, ZonaHe, dennis zhuang, dimbtp, discord9, fys, liyang, niebayes, shuiyisong, tison