v0.10.2

Release date: November 26, 2024

👍 Highlights

• Vector type: GreptimeDB supports vector data types to optimize edge scenarios, such as IoT in vehicles, enabling efficient storage and computation of vector data for real-time perception and AI applications in smart driving.
• Free index: Inverted indexes are no longer tied to primary keys, allowing users to create them on any column, enhancing query flexibility and efficiency across different scenarios.
• Alter table options: Enhancements include setting TTL for databases and tables, modifying compaction parameters, and enabling/disabling full-text indexes on columns.
• Loki remote write: GreptimeDB now supports the Loki remote write protocol, allowing users to log data in Loki format through Grafana tools.
• Performance Optimization: Up to 10x improvement for queries fetching the latest N records by timestamp (ORDER BY timestamp DESC LIMIT N).
• Remove the option: region_engine.mito.max_background_jobs and add three new options: region_engine.mito.max_background_flushes, region_engine.mito.max_background_compactions, and region_engine.mito.max_background_purges for more detailed configuration.

Breaking changes

• fix!: replace timeout_millis and connect_timeout_millis with Duration in DatanodeClientOptions by @WenyXu in #4867
• feat!: Divide flush and compaction job pool by @evenyag in #4871

🚀 Features

• feat: yields empty batch after reading a range by @evenyag in #4845
• feat: update dashboard to v0.6.0 by @ZonaHex in #4861
• feat: Sort within each PartitionRange by @discord9 in #4847
• feat: Add functionality to the Opentelemetry write interface to extract fields from attr to top-level data. by @paomian in #4859
• feat: Limit CPU in runtime (#3685) by @ActivePeter in #4782
• feat: introduce the PluginOptions by @WenyXu in #4835
• feat: add json_path_match udf by @Kev1n8 in #4864
• feat: optimizer rule for windowed sort by @waynexia in #4874
• feat(index): support building inverted index for the field column on Mito by @zhongzc in #4887
• feat: add json datatype for grpc protocol by @WenyXu in #4897
• feat: Support altering table TTL by @v0y4g3r in #4848
• feat: adds the number of rows and index files size to region_statistics table by @killme2008 in #4909
• feat: implement parse_query api by @sunng87 in #4860
• feat: enhance windowed-sort optimizer rule by @waynexia in #4910
• feat: get row group time range from cached metadata by @evenyag in #4869
• feat: simple limit impl in PartSort by @waynexia in #4922
• feat: heartbeat_flush_threshold option by @fengjiachun in #4924
• feat: support to insert json data via grpc protocol by @WenyXu in #4908
• feat: add more geo functions by @sunng87 in #4888
• feat: support filter with windowed sort by @waynexia in #4960
• feat(index): support SQL to specify inverted index columns by @zhongzc in #4929
• feat(puffin): apply range reader by @zhongzc in #4928
• feat: alter fulltext options by @CookiePieWw in #4952
• feat: introduce vector type by @zhongzc in #4964
• feat: refine region state checks and handle stalled requests by @WenyXu in #4971
• feat: support alter twcs compaction options by @lyang24 in #4965
• feat: add distance functions by @zhongzc in #4987
• feat: implement statement/execution timeout session variable by @lyang24 in #4792
• feat: introduce DynamicTimeoutLayer by @WenyXu in #5006
• feat: Loki remote write by @shuiyisong in #4941
• feat: make greatest supports timestamp and datetime types by @killme2008 in #5005
• feat: update dashboard to v0.6.1 by @ZonaHex in #5017
• feat: CREATE OR REPLACE FLOW by @discord9 in #5001
• feat: also shutdown gracefully on sigterm on unix by @discord9 in #5023
• feat(vector): remove simsimd and use nalgebra instead by @zhongzc in #5027
• feat: reimplement limit in PartSort to reduce memory footprint by @waynexia in #5018
• feat(vector): add conversion between vector and string by @zhongzc in #5029
• feat: add unset table options support by @WenyXu in #5034
• feat: alter database ttl by @CookiePieWw in #5035
• feat: use cache kv manager for SchemaMetadataManager by @v0y4g3r in #5053

🐛 Bug Fixes

• fix: fix broken import by @WenyXu in #4880
• fix: pyo3 ut by @v0y4g3r in #4894
• fix(config): update tracing section headers in example TOML files by @waynexia in #4898
• fix: set transaction variables not working in mysql protocol by @killme2008 in #4912
• fix: prune batches from memtable by time range by @evenyag in #4913
• fix: typo by @killme2008 in #4931
• fix: panic when jsonb corrupted by @CookiePieWw in #4919
• fix: data_length, index_length, table_rows in tables by @killme2008 in #4927
• fix: violations of elided_named_lifetimes by @WenyXu in #4936
• fix: database base ttl by @v0y4g3r in #4926
• fix: pprof by @discord9 in #4938
• fix: the region_stats API will return an error in instance test by @linyihai in #4951
• fix: bugs introduced by alter table options by @killme2008 in #4953
• fix: do not pick compacting/expired files by @evenyag in #4955
• fix: round euclidean result in sqlness by @v0y4g3r in #4956
• fix: column already exists by @waynexia in #4961
• fix: json_path_exists null results by @Kev1n8 in #4881
• fix(otlp): replace otlp trace attr type from string to jsonb by @paomian in #4918
• fix: alter table add column id alloc mismatch by @discord9 in #4972
• fix: physical table statistics info by @killme2008 in #4975
• fix: run install.sh error by @zyy17 in #4989
• fix: obsolete wal entries while opening a migrated region by @WenyXu in #4993
• fix: ensure Create Or Replace and If Not Exist cannot coexist in create view by @lyang24 in #5003
• fix: correct unset_maintenance_mode behavior by @WenyXu in #5009
• fix: distinct respect in range by @discord9 in #5015
• fix: inverted index constraint to be case-insensitive by @zhongzc in #5020
• fix: android build failed due to simsimd by @zhongzc in #5019
• fix: prune memtable/files range independently in each partition by @evenyag in #4998
• fix: find latest window by @v0y4g3r in #5037
• fix: prepare param mismatch by @CookiePieWw in #5025
• fix: prevent metadata region from inheriting database ttl by @waynexia in #5044
• fix(metric-engine): set ttl also on opening metadata regions by @v0y4g3r in #5051
• fix: pass series row selector to file range reader by @evenyag in #5054

🚜 Refactor

• refactor: json conversion by @CookiePieWw in #4893
• refactor: make use of the "pre_execute" in sql execution interceptor by @MichaelScofield in #4875
• refactor: simplify WeightedChoose by @WenyXu in #4916
• refactor: refactor alter parser by @CookiePieWw in #4933
• refactor: pass LogicalPlan to promql execution interceptor by @MichaelScofield in #4937
• refactor: consolidate DatanodeClientOptions by @linyihai in #4966
• refactor: support distinct JSON format and improve type conversions by @WenyXu in #4979
• refactor(mito): tidy memtable stats by @v0y4g3r in #4982
• refactor: use UNSET instead of enable by @CookiePieWw in #4983
• refactor(grafana): update cluster dashboard by @zyy17 in #4980
• refactor: Avoid wrapping Option for CacheManagerRef by @linyihai in #4996
• refactor: introduce MaintenanceModeManager by @WenyXu in #4994
• refactor: split up different stmts by @CookiePieWw in #4997
• refactor: unify mysql execute through cli and protocol by @CookiePieWw in #5038
• refactor: expose configs for http clients used in object store by @MichaelScofield in #5041

🧪 Testing

• test: add fuzz test for metric region migration by @WenyXu in #4862
• test: more sqlness tests for flow by @discord9 in #4988
• test: subquery test migrated from duckdb by @CookiePieWw in #4985
• test: reduce round precision to avoid platform diff by @zhongzc in #5013

⚙️ Miscellaneous Tasks

• chore: bump greptime-meter by @v0y4g3r in #4858
• chore: make pusher log easy to understand by @fengjiachun in #4841
• chore: better column schema check for flow by @discord9 in #4855
• chore: udapte Rust toolchain to 2024-10-19 by @v0y4g3r in #4857
• chore: add schema urls to otlp logs by @shuiyisong in #4876
• chore: graceful exit on bind fail by @discord9 in #4882
• chore: remove struct size assertion by @v0y4g3r in #4885
• ci: install numpy in CI by @discord9 in #4895
• chore: update proto depend by @discord9 in #4899
• chore: provide more info in check batch message by @evenyag in #4906
• chore: short desc markdown about change log level by @discord9 in #4921
• chore: update default cache size to 1Gib by @killme2008 in #4923
• chore: minor refactor for weighted choose by @fengjiachun in #4917
• chore: fix typos in change log level doc by @WenyXu in #4948
• chore: add json path for pipeline by @paomian in #4925
• chore: paginated query region stats by @fengjiachun in #4942
• chore: update cluster dashboard by @zyy17 in #4995
• chore: update greptime-proto to e1070a by @discord9 in #4992
• chore: rename change to modify by @CookiePieWw in #5000
• chore(cli): set default timeout for cli commands by @WenyXu in #5021
• chore: bump version to 0.10.0 by @zhongzc in #5040
• chore: bump version to v0.10.1 by @zhongzc in #5048
• chore: bump version to v0.10.2 by @zhongzc in #5056

Build

• build(deps): switch to upstream jsonb by @CookiePieWw in #4986

New Contributors

• @linyihai made their first contribution in #4996
• @ActivePeter made their first contribution in #4782

All Contributors

We would like to thank the following contributors from the GreptimeDB community:

@ActivePeter, @CookiePieWw, @Kev1n8, @MichaelScofield, @WenyXu, @ZonaHex, @discord9, @evenyag, @fengjiachun, @killme2008, @linyihai, @lyang24, @paomian, @shuiyisong, @sunng87, @v0y4g3r, @waynexia, @zhongzc, @zyy17