This article includes a log of recent significant changes to the Instaclustr Managed Service.
Table of Contents
Release OpenSearch 1.3.5
Add Support for OpenSearch Dashboards Reporting plugin
Updates to logged events.
Requests to the Instaclustr API are blocked from countries that are not Global Trade Compliant.
Resolved an issue where users under an organisation could not be deactivated using the SCIM 2.0 API.
Preview Release of version 2.0.0 of the Instaclustr Terraform Provider with complete Cassandra support, now available through the Terraform Registry.
SCIM provisioned users are no longer signed up for marketing emails by default.
Fixed an issue where verification emails would not be sent to users.
Cadence Development Shared Infrastructure available for Public Preview
Improved the console behaviour of viewing account and organisation invitations by hiding expired invitations.
Fixed bug preventing the enablement of custom log shipping to user Kafka clusters.
Fixed the ability to sign up through the AWS SaaS products for Instaclustr.
Disabled Node Sizes m5l-250-v2, i3.2xlarge-v2, c5d.2xlarge-v2 and m5xl-800-v2 for the Opensearch Offering. Alternative Node sizes with the same configuration are available for provisioning.
Added support for provisioning a Kafka PrivateLink cluster with Kafka add-ons
General availability of PostgreSQL version 13.8 and PostgreSQL version 14.5. Disabled creation of PostgreSQL version 13.7 and PostgreSQL version 14.4 from the Instaclustr managed platform due to reported CVEs
Enabled deprecated node sizes resizeable-large(r5-4xl)-v2 and resizeable-small(r5-2xl)-v2 for AWS Canada (Central)
Fixed inconsistency in some of the metric names in Prometheus API response.
Fixed bug preventing reactivated SSO users from accessing the console.
Fixed disk initialisation issue causing slow provisioning for AWS instance store nodes.
Fixed bug causing the latest metrics to be missing for some nodes.
Improved disk space utilisation for new clusters.
Improved accuracy of disk space utilisation metrics.
Release Redis 7.0.4.
Disabled Node Sizes t3.small-v2 and t3.medium-v2 for the Opensearch offering. Alternative node sizes with the same configuration are available for Provisioning.
When provisioning a PostgreSQL cluster, using synchronous replication mode now requires 2-node minimum.
Add PrivateLink support for OpenSearch to our Terraform Provider.
General Availability release of AWS PrivateLink support for managed OpenSearch clusters.
Rename node sizes i3en.xlarge-2500, i3en.2xlarge-5000 for Kafka to KFK-PRD-i3en.xlarge-2328, KFK-PRD-i3en.2xlarge-4656.
Released Managed Apache Zookeeper 3.6.3 and 3.7.1 in GA.
Support i3en.xlarge and i3en.2xlarge instances for Kafka.
Added feature to restrict usage of API keys with CIDR ranges.
Disabling Opensearch Node Sizes m5xl-400-v2,i3en.xlarge, r5xl-1200-v2, r5xl-2000-v2 and m5xl-1600-v2 since Node sizes with the same configuration but different name are available for provisioning.
OpenSearch version 1.3.4 is now available.
Instaclustr for Redis is Now PCI DSS Compliant
Disabled Lucene add-on for Cassandra 4.x.
Enabled customer initiated resizing through API, Console and Terraform for eligible Azure PostgreSQL clusters
Improved security around GCP service account credentials handling.
Fixed a bug when resizing a AZURE_AZ/GCP Kafka cluster the operation fails immediately.
Added support for provisioning public network GCP Cadence clusters with Load Balancer.
Removed unwanted Cadence node sizes from Kafka’s provisioning list.
Added streamlined solution provisioning for Cadence clusters with Advanced Visibility support.
General availability of PostgreSQL version 13.7 and version 14.3 on the Instaclustr Managed Platform. Disabling creation of all other PostgreSQL versions from the Instaclustr Managed Platform due to CVE-2022-1552 (for further details please see: https://www.postgresql.org/support/security/CVE-2022-1552/)
Added support to allow any number of nodes to provision for Cadence.
Added PrivateLink support for Kafka to our Terraform Provider.
Added PostgreSQL self-serve feature to change the default bundle user via console.
Resolved an issue with viewing Postgres and Cadence logs through the Console.
Public Preview release of AWS PrivateLink support for managed Kafka clusters.
Added advertisedHostName, and privateLink configs to terraform description endpoint.
Added support to migrate clusters between accounts in an organisation. This feature is accessible through the Migrate Clusters menu item in the Organisation view.
OpenDistro For Elasticsearch is now legacy support only.
Fixed an issue where Monitoring API requests would block certain request patterns.
Added ability to toggle authentication for the Web UI on Cadence clusters
Corrected “Client-to-Broker Encryption” to “Client-to-Cluster Encryption” for PostgreSQL connection info page in management console.
Connection pool level metrics for PgBouncer are now available in management console.
Security patching for Cadence clusters.
Insight into the Console’s frontend performance using Datadog’s Real User Monitoring.
Implemented new API endpoint to get a list of PgBouncer connection pools.
Fixed a bug where querying Kafka cluster level metrics using the Prometheus endpoint returns 404.
Fixed a bug where Cadence IPs were not being added to the supporting Cassandra cluster’s firewalls for private network clusters.
Cadence uses the private IP of its dependencies in the config file when using TARGET VPC or VPC PEERING mode.
Usernames may now start with a number.
Added support to provision Cadence related Cassandra keyspaces with Network Topology Strategy and RF = 3.
Updated API documentation to show Prometheus metrics name.
OpenSearch 1.3.1 and OpenSearch Dashboards 1.3.1 are now available.
Amended networking configuration with some Cadence clusters
Added complete solution provisioning option for Cadence clusters.
Fixed issue that prevented OpenSearch Dashboards starting on graviton node sizes.
Deprecate non-Graviton2 nodes for OpenSearch, Open Distro for Elasticsearch, and Cassandra.
Added support for provisioning PgBouncer with PostgreSQL bundle using the terraform provider.
Added comprehensive documentation of customer-facing metrics for Cadence monitoring at https://docs.api.instaclustr.com/. Also added supporting documentation for provisioning Cadence clusters directly via our provisioning API.
Workflow success/timeout/failed/cancel/terminate (aggregated at the domain level) metrics added for Cadence.
Maintenance contacts are now available on organizations.
Organization contacts can now be synced to linked accounts.
PgBouncer options will now be returned under “addonBundles” when querying the “/provisioning/v1/:clusterId/terraform-description” endpoint
New node sizes available for Cadence on AWS, GCP and AZURE AZ
Optimized prometheus endpoint api for Apache Kafka node
Released Karapace Schema Registry version 2.1.2 as an enterprise add-on for Kafka as a public preview version
Resizeable sizes for Cassandra are now retired, and no cluster/CDC can be provisioned using these sizes. Cassandra CDCs are now resizable even if they are not using the resizeable sizes.
Enabled new AWS Graviton2 node sizes on OpenSearch for free trial
Change Cadence web service port from 8080 to 443 using HTTPS
Released new Cassandra AWS graviton node sizes as enabled
Resolved GCP provisioning issues due to broken deb package references
Enabled PgBouncer provisioning through API.
Updated PgBouncer console metrics views for better grouping.
Fixed bug that allowed users with unvalidated emails to access support portal
Added new AWS Graviton2 node sizes for OpenSearch
PgBouncer provisioning capability added for Instaclustr Management Console
Updated PgBouncer console metrics views for better grouping.
Added “Hello World” code example using Go for Cadence on Console.
PgBouncer provisioning capability added for customers through Instaclustr management dashboard
Bundled Use Only deployment discounts are now calculated for you in the confirmation page of our Console cluster creation workflow for RIYOA provisioning.
Removed the ‘needsLoadBalancer’ field from the payload schema for our cluster creation API endpoint. Cadence clusters will receive a load balancer by default.
Added Index Management plugin to OpenDistro for Elasticsearch/OpenSearch’s options and OIDC provider to cluster resource in terraform provider.
Added Custom Virtual Network Id on cluster detail page for RIYOA clusters running in AWS_VPC, GCP or AZURE_AZ.
Added OIDC Id to list of OIDC Providers in Cluster Resources page in Console
RIYOA customers will now be able to see the price in management units of new nodes and clusters
Organization owners removed from an organization will now receive an email notification
Provisioning support is discontinued for older Kafka Connect versions: 2.3.1 and 2.4.1
Added various usability improvements to the Console 2 monitoring graphs pages, including – new graph selections are added to the top of the list and the monitoring graphs page has been made the default page for running clusters.
Enabled some GCP local disk node sizes for Cassandra.
Service Users released – Service users allow for access to the instaclustr API without that access being tied to a specific user. See support article for more detail.
Added feature to allow requiring all new clusters to be PCI compliant to the console account security settings
Updated Zookeeper logo
Fixed bug with Kafka Connect selecting the wrong default connection type to Kafka
Improved backup and resize events date ordering on Console
Disabled GCP N2 node types in US-West-2 California as they are not available.
Removed incorrect GCP N2 node sizes for the Cassandra Spark combination.
Improved the usability of selectors with a large set of options on Console 2’s monitoring graphs and lists pages by supporting auto-complete search.
Fixed conflicting replication factor and node count when adding a data centre on Console 2.
Improved clarity of firewall rule types between Cassandra versions on Console 2.
Fixed validation to allow at-rest encryption for node sizes with ephemeral storage.
Added a new status “STALE_TOPIC” for Kafka Connect Mirroring, and modified the existing behaviours of its “IN_SYNC” and “OUT_OF_SYNC” status.
Removed Zeppelin from the API documentation because it is no longer supported.
Updated Console 2 to be the default console.
Added an Account Security option to select the duration of inactivity before the session times out on Console 2. This option is defaulted to 8 hours for accounts that have not previously enabled it on Console 1.
Fixed some exceptions in some scenarios in the monitoring API.
Improved accessibility for password managers on Console 2.
Fixed error on initial load of Kafka mirroring details page in Console 2.
Added automatic refreshing of mirroring statuses on the main mirroring information page for Kafka Connect clusters in Console 2.
Prevented invalid cluster prices from showing on the cluster pricing page of Console 2.
Updated Redis Config “client-output-buffer-limit”
Support for Kafka 2.7.1 and Kafka Connect 2.7.1 public release.
Added the Health monitoring page for Cassandra clusters on Console 2.
Managed ZooKeeper clusters can now be provisioned via Console 2.
Fixed a typo in the API Documentation’s example provisioning request for Apache ZooKeeper.
Fixed a bug in Kafka cluster deletion process.
Added more Redis metrics to Redis cluster monitoring’s summary page.
Enhanced user support to allow Heroku users to view clusters via Console 2.0.
Policy Updates are now shown to users automatically and require acknowledgement to return to using your account.
Fixed connection info pages showing IPs for nodes that have not hit running status.
Fixed connection info pages showing Kafka Dedicated Zookeeper node IPs.
Fixed situations where it was possible to unlink or change the role of a user when it should not be possible.
Fixed OIDC providers being shown on cluster creation for accounts without OIDC enabled, but with a user who has access to an OIDC account.
Updated summary cards to show more information.
Updated menu toggle to be more visible and usable.
Added Kafka Consumer group metrics pages to Console 2.
Instaclustr’s AWS Marketplace public offering available in Console 2
Added the SSO Enablement Request form and View Identity Provider pages to Console 2.
Added provider account name, EBS encryption key name, and tags to cluster details display in Console 2
Switched support portal integration to the new Console 2 application
Spark UI pages available in Console 2.
Fixed a bug in memory allocation for applications on developer-size Kafka Graviton nodes
Fixed an issue with some metrics returning as fractions instead of doubles
Overhauled the design and functionality of the monitoring graphs page on Console 2 to support viewing multiple metric groups at a time.
Resolved an error with Memory metrics on the Metrics Graphs page on Console 2.
Resolved an issue on the Company Name not pre-populating with the saved value in the Account Details page on Console 2.
Cassandra resize is now possible in Console 2 for resizable node sizes.
Corrected the multi-DC Cassandra provisioning API example.
Updated Monitoring Api backend
Added a new endpoint for users to rotate their own API key
Repositioned settings and support menus in Console 2
Managed ZooKeeper clusters can now be managed completely in our new console, with monitoring, firewall management and connection support all now available
Console 2 UI improvements for smaller screens
Add additional nodes sizes with larger disks to the Open Distro for Elasticsearch offering on Azure with Availability Zones.
AWS Graviton 2 and GP3 support for our Kafka offering
Provisioning support is discontinued for older Kafka versions: 2.1.1, 2.3.1 and 2.4.1
Implemented per customer rate limiting on monitoring API.
It is now possible to retrieve Kafka consumer group states through API https://developer.instaclustr.com/#operation/getKafkaClientsInConsumerGroupV2 and console https://www.instaclustr.com/support/documentation/kafka/monitoring-information/consumer-group-states. Please note that metrics might not be reported for your cluster as these metrics will be enabled gradually across clusters
Add support for the following AWS regions: af-south-1 (Cape Town), eu-south-1 (Milan), me-south-1 (Bahrain).
Removed provisioning limits for customer provider accounts (enterprise).
Fixed an issue on the console where 404s returned raw HTML.
Users can now request for SSO and obtain SAML 2.0 configuration details through the Instaclustr Console.
Fixed the issue that ickafka user’s password could not be reset immediately after the cluster reaches running state.
Added new custom heroku plan in Frankfurt with 6 i3en.xlarge nodes
Enable China region for contracted customers.
New Cassandra versions 2.1.22 and 4.0-beta2 have been released. 2.1.22 is only available upon request.
Added ‘Active Connector’ page in console for managing Kafka Connect connectors.
Kafka Rest proxy and Kafka Schema registry user passwords are now configurable through Dashboard kafka users view.
Added OpenID Connect (OIDC) support for Elasticsearch clusters to facilitate Single Sign-On (SSO) for Kibana.
Enabled provisioning PCI compliant clusters with public IP.
Added a new status property “cdcStatus” for cluster data centres to the cluster status API request.
Released Azure VM types L8s and D15v2 with local storage
Added support for optional Redis client encryption
Released support for general shipping of application logs to Kafka clusters
Instance store node types added to Elasticsearch in AWS.
n::nodeStatus marked as deprecated
Custom node type for specific customer.
Included pre-built ElasticSearch connector in our Kafka Connect image.
Added encryption for Azure temporary storage on L8s and D15_v2 nodes (Azure default encryption doesn’t cover temporary storage), these sizes are available only upon request.
Released functionality to configure a truststore within a Kafka Connect cluster to trust other instaclustr managed clusters.
Two Factor Delete is now configurable with the cluster provisioning API endpoint and the introduction of a new cluster update API endpoint that supports Two Factor Delete.
Fixed missing connector dependency in Apache Zeppelin image
Simplified the dashboard view for customers with large number of clusters
Opendistro-for-Elasticsearch 1.8.0 and Kibana 1.8.0 are now available
Turn on failure listener feature for open distro for elasticsearch with security plugin
Elasticsearch automated backup frequency changed from once every 24 hours to once every 1 hour
Added additional http security headers to the instaclustr console
Fixed an issue causing errors when using the Provisioning API to add encryption keys for some AWS regions.
Released Kibana for Open-Distro Elasticsearch.
Fixed an issue that caused misleading errors when attempting to delete non-existent encryption keys via the Provisioning API and Dashboard.
Updated the Dashboard so that metrics for replaced nodes are now displayed.
Changed IDS logs to group by customer clusters.
Updates to node security.
Updating aliases on nodes for ease of use on PCI clusters
Added extra logging to metrics collector on nodes
Fixed an issue where the following non-PCI verified add-ons could be included when provisioning a PCI compliant cluster : Kafka Rest Proxy, Kafka Schema Registry, Cassandra Lucene Index Plugin
Create Firewall Rules API Endpoint is updated with creating firewall rules mapped by AWS security group ID. With the new changes, the user can create, delete, and list firewall rules for AWS security group ID as well.
Elasticsearch restore enabled from Provisioning API for customers.
Elasticsearch restore enabled from the Dashboard for customers.
Updating aliases on nodes for ease of use on PCI clusters
Added extra logging to metrics collector on nodes
Introduce new metrics for kafka topic message distribution including, outliers with respect to number of messages in partitions within a topic using the MADe method and the standard deviation for the partition message counts as a measure of dispersion. These metrics will be enabled gradually across clusters
Implemented latency and memory metric collection for Redis cluster
Fixed memory available unit in monitoring metrics
Added Kafka Connect plugins for the communication between Kafka and Cassandra.
Added more node types for running Kafka Connect in the AWS and Azure.
Improved memory allocation for Kafka Connect nodes during cluster creation.
Fixed an issue with the provisioning API’s describe-options endpoint.
Replaced Heroku t2 plans with t3 plans
Released Apache Cassandra 188.8.131.52
Added RequestDiscarded and PausedConnections metrics to Cassandra 184.108.40.206
Redis Public Preview is available to provision
Added Kafka Connect Task Level metric collection and exposing via instaclustr monitoring API. This feature will be gradually enabled in clusters. Refer https://www.instaclustr.com/support/api-integrations/api-reference/monitoring-api/kafka-connect-metrics/ for metric definitions
Added feature to allow Kafka Connect logs to be shipped to a connected Instaclustr Kafka Cluster.
Updated pricing information for Elasticsearch.
Released updates to node security.
Changed Native Transport Request’s Total Blocked graph to Differential Blocked Tasks.
Added support for running Kafka on r5 nodes in the AWS SA_EAST_1.
Disabled support for running Kafka on r4 nodes in the AWS SA_EAST_1.
Added additional metrics for cluster management & monitoring
Fixed the inconsistent ordering in pricing page.
Added extra validation to username, company name and phone number fields on account creation page.
Disabled the ability to provision Elassandra on t2.small node size.
Disabled Elassandra 220.127.116.11 for console and API.
Enabled various Cassandra ThreadPool metrics for Console and Monitoring API, including Read stage, Native Transport Requests, Mutation Stage, Counter Mutation Stage, View Mutation Stage, RPC Thread and Total Dropped Messages.
Fixed 500 error on Spark worker page.
Fixed Console impersonation bug
Fixed provisioning issue with some older Cassandra versions
Added User management UI for Kafka clusters
Fixed node size display ordering issue
Removed deprecated node sizes from Console
Removed ability to enable client encryption for developer sizes
Clicking on graphs in Cluster Summary page now takes users to Metrics Graph page instead of 404
Updated Console to display correct memory amount for GCP n1-highmem-2 nodes
Add support for Cross Region VPC peering through Dashboard and API
Improved VPC peering reliability
Updated IBM Softlayer to its rebranded name: IBM Cloud
Updated provider logos for AWS and GCP
Improved restore process for deleted clusters
Updates to node security
Update Kafka connection information on the Console
Fixed issue when the user chooses not to create a firewall rule when creating a Cassandra cluster but a firewall rule is still created
Disabled adding nodes to cluster for users with no billing details
Fixed issue where customers are unable to select EBS encryption keys on the console for non EBS-backed node sizes
Fixed issue where a wrong network is suggested when trying to add cluster data centre to an existing cluster
Updated instance system kernel settings for better performace
Fixed issue of accumulated MigrationStages when cluster is under load
Improved bootstrap logic for node-agent
Add timestamp to bash history for internal TechOps
Provisioning API cluster info now exposes network and use private address broadcasting.
released version 0.6.1 of Zeppelin with Spark Cassandra Connector 1.6.2
Rack allocation displayed in cluster creation, and data centre and node addition forms.
Cluster details page now displays if Private Node Discovery is enabled or disabled.
Added third rack to AWS Sydney region
Clusters in AWS Asia Pacific (Sydney) now support 3 racks (availability zones).
OS Load now available through console and API monitoring
OS Load metric is now available in console monitoring view and through our monitoring API.
Add node process improvements
Automated setting of some configurations when adding nodes and data centres to a cluster to reduce manual intervention required.
Changes to provisioning statuses
Node now moves to “provisioned” as soon as AWS instance makes contact with our system. “Joining” state added for nodes where Cassandra is running but in joining state. (Billing commences on joining or running state.)