Manufacturing Data Engine release notes

• Fix (423567127): Fixed a bug that prevented numeric values to be correctly processed when using file ingestion.
• Fix (441294785): Fixed a bug that prevented the deletion of a message class if the related parser had been deleted beforehand.
• Fix (443710570): Fixed a bug that could use an old version of a parser script when uploading a new configuration package that had the same parser name.

Release signature

6e178bd
1.5.2
9ee64699a1bdf1cf690e7930d968f002

• Improvement (448853115): Added the ability to enable and configure VPC flow logging from Terraform. Collecting VPC flow logs is recommended to detect potential intrusions or anomalies.
• Improvement (450530585): Updated container images and dependencies to address known security vulnerabilities.
• Improvement (448854007): Adjusted IAM roles used by some MDE service accounts following the least-privilege approach.
• Improvement (448148987): Switched to DNS-based endpoint with AIM access control for GKE control plane, following security best practices.
• Improvement (448148673): Enabled SSL required mode on the Cloud SQL instance for additional security enforcement.
• Improvement (448148275): Enabled in-transit encryption and token authentication for Redis.

• Improvement (441727217): Introduced an option to enable parsers after types activation when you upload a new configuration package.
• Improvement (333321871): Refactored scripts to collect additional information for the MDE team when you open a support ticket to help faster troubleshooting.
• Improvement (441660844): Added typesMetadataBuckets and typesTransformations JSON arrays as responses to the Tags API implementation.
• Improvement (435358499): Enabled Streaming Engine for Dataflow in all MDE deployment sizes for optimized consumption, saving resources.

Release 1.5.2

This release is a critical update if you have an existing deployment of MDE version 1.5.0. This release introduces a new configuration package upload option that allows users to automatically enable and activate all entities created following the upload. It also includes other minor improvements and bug fixes.

Follow the Upgrade guide in the MDE documentation for detailed instructions. Note that instructions are different depending on the MDE version you are upgrading from.

Release 1.5.1

This release is a critical update if you have an existing deployment of MDE version 1.5.0. This release resolves a bug regarding materialization of metadata instances created prior to MDE 1.5.0. This release also includes other minor improvements and bug fixes.

• Improvement (407009198): Improved the error handling when upload and parsing configuration packages.
• Improvement (423531705): Improved MDE logging for BigQuery sink related operations.
• Improvement (423530033): Improved manifest validation on configuration package uploads.
• Improvement (423554635): Added CreatedAt column with default sorting on the MDE UI configuration packages page.
• Improvement (430962108): Added more sorting options on the MDE UI Configurations and Metadata Instances pages.
• Improvement (423531714): Various improvements on Helm charts, including image tag management, and k8s secrets/configmaps.
• Improvement (407037164): More descriptive message added to MDE UI to confirm Type deletion.

• Improvement (427447932): Brought Docker images to versions without vulnerabilities reported at the time of the release.
• Improvement (361290775): Modified Terraform deployment scripts to enforce TLS v1.2 for an external MDE UI Load Balancer.

• Fix (417666631): Fixed an issue with metadata materialization for metadata instances that had been created prior to migrating to MDE 1.5.0.
• Fix (420921890): Fixed an error when attempting to update an instance tag metadata and saved it for instances that had been created prior to migrating to MDE 1.5.0.
• Fix (423535516): Fixed inconsistent API response codes when trying to delete non-existing entities.
• Fix (383519276): Fixed missing fields in MDE logging and added more details to make troubleshooting easier.
• Fix (424077359): Fixed instance bucket creation through API. It now adds default createdTime as the time when the API call was received.
• Fix (422991109 and 424084607): Fixed ghost deletion of Types and Metadata Buckets after removing a configuration package.
• Fix (423859259): Fixed removal of BigQuery views when a Type is manually deleted.
• Fix (406803212): Fixed wrong version materialization on MDE system tables.
• Fix (407015039): Fixed Delete button in MDE UI when the system is in PROD mode (Production mode).
• Fix (435653743): Fixed missing Grafana Terraform module.

Release signature

b0fc163
1.5.1
ffb87d39d343c20abebd2f52df74a2d3

Release 1.5.0

This release is not a critical update, unless you are directly impacted by the bug fixes, you don't need to update, and you can wait for future releases before updating.

• Configuration Packages: Introduces a file-based configuration package system for atomic deployments and GitOps integration. For more information, see Upload configuration package and File content.
• Development Mode: Adds a "Development Mode" to allow deletion of MDE entities and configuration packages, use with caution. For more information, see Development mode.

• Metadata Versioning: To enhance the query readability of the metadata instances, MDE v1.5.0 introduced a new field called validFrom that designates the time when a particular metadata instance is effective. MDE uses this field to check which metadata instance picks based on the source message event time, not processing time, enabling accurate historical data representation. For more information, see Versioning metadata buckets.

Release 1.4.2

This release is a critical update if you have an existing deployment of MDE version 1.4.0 or 1.4.1. This release resolves a bug regarding the raw messages writing, where the Cloud Storage Avro files were not being written.

• Fix (381418590): Fixing an issue of the generated BigQuery views for the types when one of the bucket or data field schemas contains arrays.
• Fix (382189410): Fixing issue of the Cloud Storage Writer not writing the raw messages to the Cloud Storage bucket.

Release signature

57c3062
1.4.2
ccc9f5b6c30a1c1a42eae04c995f783b

Release 1.4.1

This release is not a critical update, unless you are directly impacted by the bug fixes, you don't need to update, and you can wait for future releases before upgrading.

• Improvement (376024419): Enable automatic reconciliation of the missing infrastructure elements and for MDE types. When the user creates a new type, multiple infrastructure components are created in the background. To accommodate cases like type activation or upgrade issues, MDE now checks for missing infrastructure components in the startup of the bigquery-sink service and makes sure that it is aligned with the types configurations in MDE.

Release signature

0b539af
1.4.1
0e06a47adf3c1f7efedfcb182235b490

Helm charts - MDE services containers image tag: 3f285a5. - MDE UI image tag: 2.22.0-dev.0.123.4.

• Fix (375367674): Fixing issue when the edge messages contained fields named event or headers which was colliding with the MDE event, and headers attribute that are used during the message-class resolution. The MDE message headers are now reserved in a new keyword called mdeHeaders.
• Fix (371176757): Fixing issue of losing the monitoring metrics after upgrading from 1.3.x to 1.4.0. parsers.
• Fix (377694402): Upgrading the Apache Beam runtime to the latest version of 2.60.0.
• Fix (377672812): Upgrading Terraform runtime and the required Google Cloud provider versions to solve multiple deployment bugs.

Version 1.4.0

This release is not a critical update, unless you are directly impacted by the bug fixes, you don't need to update, and you can wait for future releases before upgrading.

Note: If you are upgrading from older releases, the Terraform may not be able to perform an in-place upgrade for the Cloud Storage Writer dataflow job. In that case, you need to stop it manually, before starting the upgrade.

• Improvement (330297418): Enable the three-projects topology for bigquery-sink: the service to be deployed in a project, the Pub/Sub in another project, and the BigQuery datasets in a third project. Find these as dedicated properties in the BigQuery sink helm chart.
• Improvement (320427338): Creating automatic BigQuery views based on the type buckets schema. This provides a stable column-based schema for accessing the types as well as its metadata in a single view. For more information, see Analyticis view.
• Improvement (320427684): Types data field schema validation. The non-numeric types can have a JSON schema that governs the data field of its records, and this schema is now enforced during the processing. For more information, see Data field.
• Improvement (320427624): New real-time stats service for the ingested tags. For more information, see Real-Time service.
• Improvement (324029584): Automatic deletion for excessive metadata instances to prevent the system performance degradation. For more information, see Metadata automatic deletion.
• Improvement (353244779): New Notification API endpoint to display time-sensitive messages to the user such as the metadata instances approach the maximum size limit.
• Improvement (353247809): New API to stop and start the processing of messages.

Release signature

7176648
1.4.0
cf0b7439bf9862b06aedcea13dd783fe

Helm charts

• MDE services containers image tag: a04af0d.
• MDE UI image tag: 2.22.0-dev.0.123.4.

• Fix (324563314): Enabling the Dataflow Streaming Engine for the Cloud Storage flows.
• Fix (327369559): Listing the parsers endpoint does not return the disabled parsers.
• Fix (320427422): Upgrading the Spring stack to 3.x.
• Fix (353240181): Security and performance fixes.

The vulnerability scanner may flag a critical CVE related to the zlib libraries in the base OS of the container images. The binary of this impacted library is not built in the MDE container images, and it is not impacted by this vulnerability. Nonetheless, once a new Debian OS patch is available, MDE will release an updated container images so that the vulnerability scanner does not flag this issue. For more information about the vulnerability, see the Debian tracker.

Version 1.3.8

This release is not a critical update, unless you are directly impacted by the bug fixes, you don't need to update, and you can wait for future releases before upgrading.

• Improvement (324563314): Enabling the Dataflow Streaming Engine for the Cloud Storage Flows to improve the scalability behaviour of the Cloud Storage jobs.

Release signature

a94aa0e
1.3.8
e74afda073cd5d52a591763c5ea7cbe1

• Fix (324540732): Adding JVM initialization flags to increase the memory utilisation per pod.
• Fix (324040010): Configuration Manager running out of memory when running a misconfigured parser that impacts the CPU and memory allocations.
• Fix (321911849): Message Mapper running out of memory when running a misconfigured parser that impacts the CPU and memory allocations.
• Fix (322148462): Fixing web interface error when interacting with items in a paginated response with size of more than 100 items.
• Fix (327323565): Cloud Storage Writer out of memory error when ingesting large number of items.
• Fix (328190564): Cloud Storage Reader out of memory error when reading a large number of files.
• Fix (327369559): Listing parsers in the web interface does not show the disabled parsers.

Helm charts

• MDE services containers image tag: 09497d7.
• MDE UI image tag: 2.15.0.0.109.2.

Version 1.3.7

This release is not a critical update, unless you are directly impacted by the bug fixes, you don't need to update, and you can wait for future releases before upgrading.

Library updates

• Updated Apache Beam SDK to version 2.53.0.

Release signature

7460a29
1.3.7
3ed1b9c3886f123e0d839126a1577c63

• Improvement (320427011): All the services now use full names for Pub/Sub subscriptions and topics (For example: projects/PROJECT_ID/subscriptions/configuration-processing-subscription). This allows having the topics or subscriptions in a different project that the one the services are deployed to.
• Improvement (320443808): All dataflow jobs now use private IP by default. For more information, see Routes Firewall.

Helm charts

• MDE services containers image tag: 4b79010.
• MDE UI image tag: 2.15.0.0.109.2.

The following Dataflow jobs need to be manually stopped before upgrading:

• eventchange-dataflow-flex-job
• gcs-writer-dataflow-flex-job
• window-dataflow-flex-job
• bigtable-writer-dataflow-flex-job
• gcs-reader-dataflow-flex-job

Version 1.3.6

This release is not a critical update, unless you are directly impacted by the bug fixes, you don't need to update, and you can wait for future releases before upgrading.

The following Dataflow jobs need to be manually stopped before upgrading:

• eventchange-dataflow-flex-job
• gcs-writer-dataflow-flex-job
• window-dataflow-flex-job
• bigtable-writer-dataflow-flex-job
• gcs-reader-dataflow-flex-job

• Improvement (311599436): Added a new field to the output of the Group Event Change Transformation to include the message that triggered the change in the transformation
• Improvement (316554360): When parsing a message, add the output of the headers field to the Pub/Sub headers and preserve through the entire pipeline.

• Fix (315935780): Pub/Sub attributes on input-messages are filtered and can't be used in the source message class or parser.

Helm charts

• MDE services containers image tag: 8199764.
• MDE UI image tag: 2.15.0.0.109.2.

Release signature

1b11952
1.3.6
bfaaf15fad73e78565006891342dbd08

Version 1.3.5

This release is not a critical update, unless you are directly impacted by the bug fixes, you don't need to update, and you can wait for future releases before upgrading.

Release signature

8584357
1.3.5
db04eff8fb5f0d2267d1653498480ec7

Helm charts

• MDE services containers image tag: 7a994ee.
• MDE UI image tag: 2.15.0.0.109.2.

The following Dataflow jobs need to be stopped before upgrading:

• eventchange-dataflow-flex-job
• gcs-writer-dataflow-flex-job
• window-dataflow-flex-job

• Fix (309074623): Updates to different libraries, particularly Redis ones.
• Fix (308779353): Cloud Storage Writer configuration manager connection should use gRPC.
• Fix (307925709): When a message class has multiple parsers and one fails the others fail as well.
• Fix (300928888): Whistle parser shouldn't fail when it doesn't produce any records.
• Fix (307925851): Event change transformation is not working for discrete boolean output.
• Fix (308551795): Upgrade Apache Beam to 2.51.0.
• Fix (309078367): Pub/Sub subscriptions have a 30 days expiration date.