v0.9.0
· 11 min read
Release date: July 16, 2024
👍 Highlights
- Log Engine: GreptimeDB is now a unified time-series database for both metrics, logs, and events (trace in plan).
- Remote WAL is significantly improved and is now recommended to turn on.
- Table View: You can now
CREATE VIEW
on tables and treat them as logical table. - Short interval literal for user experience.
- InfluxDB Merge Read: You can now use the
merge_mode=last_non_null
table option to allow SELECT result merges all the last non-null fields. - gRPC TLS: You can now set up the
grpc.tls
config to turn on TLS for gRPC connection. - Manually Compact in Strict Window: You can now use
SELECT COMPACT_TABLE("monitor", "strict_window", "3600");
to manually schedule a compaction task, splitting SST files strictly into time windows. The third option is the time window's width in seconds.
Breaking changes
- refactor!: unify
FrontendOptions
andDatanodeOptions
by usingGrpcOptions
by @realtaobo in #4088 - feat!: reduce sorted runs during compaction by @v0y4g3r in #3702
- fix!: forbid to change information_schema by @CookiePieWw in #4233
- feat!: remove alias
metasrv-addr
by @discord9 in #4239 - feat!: Set merge mode while creating table in influx handler by @evenyag in #4299
🚀 Features
- feat: log ingestion support by @paomian in #4014
- feat(pipeline): transform support on_failure by @yuanbohan in #4123
- feat(pipeline): gsub processor by @yuanbohan in #4121
- feat: enable metasrv to accept flownode's heartbeats by @fengjiachun in #4160
- feat: update default size of bgworkers, add hbworkers by @sunng87 in #4129
- feat: implement the
OrderedBatchProducer
by @WenyXu in #4134 - feat(flow): support datafusion scalar function by @discord9 in #4142
- feat: Implement memtable range by @evenyag in #4162
- feat: update dashboard to v0.5.1 by @ZonaHex in #4171
- feat: register flow node by @fengjiachun in #4166
- feat: show database options by @killme2008 in #4174
- feat: use the write runtime to handle the heartbeats by @fengjiachun in #4177
- feat: PREPARE and EXECUTE statement from mysql client by @CookiePieWw in #4125
- feat: introduce chaos crds by @WenyXu in #4173
- feat: add num_rows and num_row_groups to manifest by @waynexia in #4183
- feat: make RegionScanner aware of PartitionRange by @waynexia in #4170
- feat: update dashboard to v0.5.2 by @ZonaHex in #4185
- feat(puffin): support lz4 compression for footer by @zhongzc in #4194
- feat: heartbeat task&peer lookup in proc by @discord9 in #4179
- feat(puffin): introduce puffin manager trait by @zhongzc in #4195
- feat(sql): improve interval expression, support shortened version by @etolbakov in #4182
- feat: Dedup strategy that keeps the last not null field by @evenyag in #4184
- feat(puffin): implement CachedPuffinWriter by @zhongzc in #4203
- feat: FlownodeClient by @discord9 in #4206
- feat: bulk memtable codec by @v0y4g3r in #4163
- feat(puffin): implement CachedPuffinReader by @zhongzc in #4209
- feat: validate partition rule on create table by @waynexia in #4213
- feat(servers): make http timeout and body limit optional by @fengjiachun in #4217
- feat(sql): add casting support for shortened intervals by @etolbakov in #4220
- feat: Implements
merge_mode
region options by @evenyag in #4208 - feat: output multiple partition in MergeScanExec by @waynexia in #4227
- feat: add build info for flow heartbeat task by @fengjiachun in #4228
- feat: register & deregister region failure detectors actively by @WenyXu in #4223
- feat: forgot collect flownode clusterinfo handler by @fengjiachun in #4236
- feat: basic implement of
matches
fn by @waynexia in #4222 - feat: flow cli for distributed by @discord9 in #4226
- feat(puffin): implement MokaCacheManager by @zhongzc in #4211
- feat(puffin): complete dir support by @zhongzc in #4240
- feat: introduce the interface of
RemoteJobScheduler
by @zyy17 in #4181 - feat(fulltext_index): introduce creator by @zhongzc in #4249
- feat: dbeaver mysql compatibility, use statement and information_schema.tables by @sunng87 in #4218
- feat: provide a simple way to create metaclient by @fengjiachun in #4257
- feat: implement naive fuzz test for region migration by @WenyXu in #4252
- feat: update dashboard to v0.5.3 by @ZonaHex in #4262
- feat: introduce
FlowRouteValue
by @WenyXu in #4263 - feat(fulltext_index): integrate puffin manager with inverted index applier by @zhongzc in #4266
- feat: handle AND/OR and priority in matches fn by @waynexia in #4270
- feat: store peer info in
TableFlowValue
by @WenyXu in #4280 - feat(fuzz): enhance condition check of region migration finish by @WenyXu in #4283
- feat: delete pipeline by @shuiyisong in #4156
- feat: add path prefix label to object storage metrics by @sunng87 in #4277
- feat: add naive region failover test for metric table by @WenyXu in #4269
- feat: expose merge_mode option by @evenyag in #4289
- feat: refine scan metrics logging by @evenyag in #4296
- feat: make flow distributed work&tests by @discord9 in #4256
- feat(remote wal): set default compression to LZ4 by @WenyXu in #4294
- feat(sql): add iso-8601 format support for intervals by @etolbakov in #4291
- feat(fulltext_index): integrate full-text indexer with sst writer by @zhongzc in #4302
- feat: impl show table status by @killme2008 in #4303
- feat: support inserting into binary value through string by @CookiePieWw in #4197
- feat(inverted_index): inverted index cache by @v0y4g3r in #4309
- feat(fulltext_index): allow enable full-text index in SQL and gRPC way by @zhongzc in #4310
- feat: handle parentheses with unary ops by @waynexia in #4290
- feat: enhanced the retry logic by adding a random noise by @WenyXu in #4320
- feat: flownode use
Inserter
to write to database by @discord9 in #4323 - feat: add
TimeSeriesRowSelector
hint by @evenyag in #4327 - feat: show create view and creating view with columns by @killme2008 in #4086
- feat: customize copy to parquet parameter by @v0y4g3r in #4328
- feat: impl drop view by @killme2008 in #4231
- feat: support
text/plain
format for log ingestion by @shuiyisong in #4300 - feat: impl optimizer rule to handle last_value case by @waynexia in #4357
- feat: Implement reader that returns the last row of each series by @evenyag in #4354
- feat(index): distinguish different types of index metrics by @zhongzc in #4337
- feat: adding information_schema.views table by @lyang24 in #4342
- feat: support show views statement by @lyang24 in #4360
- feat: tweak error and status codes by @killme2008 in #4359
- feat(fulltext_index): integrate full-text indexer with parquet reader by @zhongzc in #4348
- feat: flow perf&fix df func call by @discord9 in #4347
- feat: add a cache for last value result in row group by @evenyag in #4369
- feat: add PruneReader for optimized row filtering by @v0y4g3r in #4370
- feat: introduce 'pg_catalog.pg_type' by @J0HN50N133 in #4332
- feat: improve datafusion external error and mysql error by @killme2008 in #4362
- feat: Add caching for last row reader and expose cache manager by @v0y4g3r in #4375
🐛 Bug Fixes
- fix(sqlness): catch different format timestamp by @WenyXu in #4149
- fix: region logical regions after catching up by @WenyXu in #4176
- fix(flow): fix call df func bug&sqlness test by @discord9 in #4165
- fix:
region_peers
returns same region_id for multi logical tables by @realtaobo in #4190 - fix(sql): improve compound signed number processing by @etolbakov in #4200
- fix: align workflows again for the troublesome GHA by @waynexia in #4196
- fix: format error correctly by @WenyXu in #4204
- fix: wrong frontend registration address by @killme2008 in #4199
- fix: add
serialize_ignore_column_ids()
to fix deserialize region options failed from json string by @zyy17 in #4229 - fix(puffin): fix dependency by @v0y4g3r in #4267
- fix(fuzz): generate valid string by @WenyXu in #4281
- fix: enhance ColumnOption::DefaultValue formatting for string values by @WenyXu in #4287
- fix: enable space string in yaml value by @shuiyisong in #4286
- fix: align pre-commit config with make file by @shuiyisong in #4292
- fix: call df_func with literal by @discord9 in #4265
- fix: prepare inserting with column defaults not work, #4244 by @killme2008 in #4272
- fix: deregister failure detector in region migration by @WenyXu in #4293
- fix(ci): remove sqlness state in success by @waynexia in #4313
- fix: test_fulltext_intm_path by @zhongzc in #4314
- fix: error on show databases in non-default catalog by @sunng87 in #4316
- fix: panic while reading information_schema. KEY_COLUMN_USAGE by @killme2008 in #4318
- fix: support unary operator in default value, partition rule and prepare statement by @waynexia in #4301
- fix(sqlness): relax start time regex to match various precisions by @waynexia in #4326
- fix: permission denied is 403 by @sunng87 in #4350
- fix(config): enable file engine by default by @WenyXu in #4345
- fix: build info should use build time env var by @tisonkun in #4343
- fix: remove path label for cache store by @sunng87 in #4336
- fix: scan hint checks order asc by @evenyag in #4365
🚜 Refactor
- refactor: add
Compactor
trait to abstract the compaction by @zyy17 in #4097 - refactor: make region manifest checkpoint ran in background by @MichaelScofield in #4133
- refactor: Decouple dedup and merge by @evenyag in #4139
- refactor: make
RegionOptions
andMergeOutput
serializable by @zyy17 in #4180 - refactor: add
region_dir
in CompactionRegion by @zyy17 in #4187 - refactor: migrate region failover implementation to region migration by @WenyXu in #4172
- refactor: add
SerializedPickerOutput
and field modification ofCompactorRequest
by @zyy17 in #4198 - refactor(flow): make
from_substrait_*
async& worker handle refactor by @discord9 in #4210 - refactor: expose
DatanodeBuilder::build_object_store_manager()
andMitoConfig::sanitize()
by @zyy17 in #4212 - refactor: use ObjectStoreManagerRef type in open_compaction_region() and add related unit test by @zyy17 in #4238
- refactor: add interceptor after Influxdb lines are converted to grpc row insert by @MichaelScofield in #4225
- refactor: add
RemoteCompaction
error by @zyy17 in #4251 - refactor: change InvertedIndexWriter method signature to offsets to f… by @v0y4g3r in #4250
- refactor(puffin): adjust generic parameters by @zhongzc in #4279
- refactor: use rwlock for modifiable session data by @sunng87 in #4232
- refactor(inverted_index): integrate puffin manager with sst indexer by @zhongzc in #4285
- refactor: split match arms in prom_expr_to_plan into smaller methods by @waynexia in #4317
- refactor: do not print error log on PlanQuery error by @sunng87 in #4322
- refactor: Remove the StandaloneKafkaConfig struct by @irenjj in #4253
- refactor: LastRowReader to use LastRowSelector by @v0y4g3r in #4374
📚 Documentation
- docs: add guide for tsbs benchmark by @waynexia in #4151
- docs: remove outdated docs by @evenyag in #4205
- docs: remove cargo test workspace command by @waynexia in #4325
- docs(config): add enable_region_failover option to configuration by @WenyXu in #4355
⚡ Performance
- perf: optimize RecordBatch to HttpOutput conversion by @waynexia in #4178
- perf(puffin): not to stage uncompressed blob by @zhongzc in #4333
- perf: fine–tuned plan steps by @waynexia in #4258
🧪 Testing
- test: wait until checkpoint finish by @evenyag in #4202
- test: add e2e test for region failover by @WenyXu in #4188
- test: replace unstable output of last value test by @evenyag in #4371
- test: more sleep when flow insert makes it serial by @discord9 in #4373
- test(flow): ignore flow tests for now by @discord9 in #4377
⚙️ Miscellaneous Tasks
- ci: align docs with develop by @waynexia in #4152
- chore: bump datafusion version to fix
last_value
regression by @MichaelScofield in #4169 - chore: enhance add pipeline http api return data by @paomian in #4167
- chore: highlight our committers in CONTRIBUTING.md by @tisonkun in #4189
- chore: add AUTHOR.md file by @tisonkun in #4241
- ci: update centos yum source and specify cargo-binstall version by @zyy17 in #4248
- chore: reduce insertion size of fuzz test by @WenyXu in #4243
- chore(ci): add timeout (60min) for fuzz tests by @WenyXu in #4255
- ci: push latest greptimedb nigthtly build image by @zyy17 in #4260
- chore: add missing s for
--metasrv-addr
by @discord9 in #4278 - ci: retry on error during installing operator by @WenyXu in #4295
- chore: remove original region failover implementation by @WenyXu in #4237
- chore: bump OpenDAL to 0.47.2 by @WenyXu in #4297
- ci: retry on error or timeout during installing operator by @WenyXu in #4308
- chore: disable TraceLayer on_failure log by @paomian in #4315
- ci: add flownode in docker compose by @zyy17 in #4306
- chore: update project slogan by @tisonkun in #4361
- chore: bump version to v0.9.0 by @evenyag in #4376
◀️ Revert
Build
- build(deps): Upgrade OpenDAL to 0.47 by @tisonkun in #4224
- build(deps): upgrade opendal to 0.47.3 by @tisonkun in #4307
- build(deps): switch to upstream by @tisonkun in #4319
All Contributors
We would like to thank the following contributors from the GreptimeDB community:
@CookiePieWw, @J0HN50N133, @MichaelScofield, @WenyXu, @ZonaHex, @discord9, @etolbakov, @evenyag, @fengjiachun, @irenjj, @killme2008, @lyang24, @paomian, @realtaobo, @shuiyisong, @sunng87, @tisonkun, @v0y4g3r, @waynexia, @yuanbohan, @zhongzc, @zyy17