Major Update of Loginom Analytical Platform Quick view of the calculation results functionality was implemented for selection of the activated node. It became possible to cancel the request execution while importing from the DB. Convenient functionality of components reuse was implemented. Publication of web services became available in the Team edition. New visualizer (Data Quality) appeared, and some improvements were introduced into the OLAP cube.
Many improvements and new options aimed at the platform use convenience were introduced. The most important changes are listed below:
As the components reuse mechanism has been significantly modified in the new version, additional conversion will be required for the packages that include the nodes with the "public" or "published" visibility.
It is recommended to save copies of the updated packages separately before new version opening. Conversion will be automatically performed when opening such packages. It is required to save them after update once again. More detailed information on the changes and special features of packages conversion is available in the following section: Conversion of the earlier version packages to 6.5 format.
The order of the automatic fields names generation was changed. More detailed information is available in the following section: Change in the automatic fields names generation.
The standard "All" user group was added to the administration settings. More detailed information is available in the following section: Changes in the Administration Section.
Log files encoding was replaced with UTF8. This change will be very important if the automatic processing of log files is set when using scripts. More detailed information is available in the following section: Changes in Logging.
Data analysis task can be somehow completed using different tools. That is why it is particularly important to provide the specific product operation convenience for a user. Data analysis logics debugging and design take much time of the analysts who are the main Loginom users. It is necessary to make this process comfortable for a specialist to the greatest possible extent. It must meet all his requirements.
To provide usability improvements of the analyst work, quick view when selecting the activated node without additional windows opening was implemented in Loginom 6.5. This option provides visual control of the calculation correctness in each workflow node and detection of errors in a timely manner. The option is started when activating the "Quick view" mode in the right toolbar corner by double clicking on the output node port or using "Advanced quick view" item of the context port menu.
Option of the Calculator calculation results preview was implemented. Now, when writing a formula, it became possible to view the results by the example of the first 100 records at once. And both the fields used in the formulae and calculation results are displayed that facilitates visual comparison of the calculation correctness. The option is activated when pressing the "Preview" toolbar icon in the Calculator wizard. This functionality is particular relevant when working with complex formulae.
Calculator Calculations Preview
It became more convenient to design and debug the packages connected on the Reference page. For this purpose, it became possible to move such package to the user session if it is used only in the current session. It enables to edit the connected package if nobody uses it with exception of the current user session.
User confirmation request to deactivate the node before configuration of the node or its ports was added.
Publication of web services is one of the most popular platform functions. For example, when creation of decision-making system is meant, almost the whole functionality is based on the web services access. Let's consider some changes in detail.
It became possible to publish web services in the Team edition using Loginom Integrator in the new platform version. It somewhat differs from the integrator for the Standard and Enterprise editions:
The new node ("Web services") was added on the navigation panel. Its activation enables to open the packages publication configuration page.
New "Web Services" Page
List of all published packages, name and path to the package file, publication date, versions and auto update status are displayed on this page. List of all published nodes is displayed for each package. Status of the published web services is also highlighted: the green color means that the web service complies with the saved package; the orange color means that the web service does not comply with the saved package; the grey color means thet there is no access to the published package.
It is possible to add or update the web service publication, edit the published package or cancel the publication on this page. If "Update publication on package saving" parameter is set in the package publication settings, it will be automatically republished taking into account the recent changes after each web service editing and package saving.
Besides, a separate role appeared in Loginom 6.5 administration. And it became possible to control the web services publication rights.
Opportuniity to reuse components and ease of this operation is one of the most important features of the Loginom operation. Due to the reuse of the earlier designed and checked components, the quality of the final workflows is improved, and possibility to share such components increases many times the analyst's performance.
It is sufficient to select "Create derived component" submenu in the context menu of the node to reuse the components. It is possible to select two available options in the opened window: "Node Execution and Loop" and "Derived nodes". If "Node Execution and Loop" option is selected, the component execution or loop execution will be available. If "Derived Nodes" option is selected, it will be possible to modify the component before its reuse.
All components available for the reuse both for the common execution, and for editing are displayed in the "derived components" menu in the new platform version when connecting any package. When dragging the derived component to the workflow area, Execution node is created by default; if Shift is pressed, Loop is created, and if Ctrl is pressed, the derived node is created. If it is not allowed to create Execution/Loop nodes in the component settings, the derived node is always created.
The components of the Derived Components panel are currently grouped by the packages into which they are included. The components are sorted in alphabetic order in each package. The context menu commands were also added:
Starting from 6.5.0 version Node Execution and Loop nodes refer not to the node but to the component. To provide the backward compatibility with the packages designed in the previous platform version, the workflow conversion mechanism was implemented.
Before opening the packages designed in the previous platform versions in the new platform version, it is recommended to make backup copies of the source packages.
When opening such packages in Loginom 6.5.0 and higher, the derived components are automatically created for all nodes to which Node Execution and Loop nodes refer in the current package. These components receive the same caption and visibility as the source node.
The derived components for all nodes with the "Public" or "Published" visibility are automatically created in the same way as references to these nodes can be available in other packages. These components receive "Node_caption (public node)" or "Node_caption (published node)" captions according to the visibility of the nodes by which they have been created. Some of these components can become excessive if there are no Node Execution or Loop nodes that refer to them in other packages, and it is not planned to create them. Such situation is most probable for the derived components created by the published nodes as these nodes are most frequently used for the work based on Loginom Integrator or for the Batch processing. All excessive derived components can be manually deleted. Then a package can be saved.
When publishing the packages containing the nodes with the "Published" visibility, a user will be warned that the package will be converted and additional derived components will be created. This message will be also received if there are no such nodes in the package but they are available in one of the connected packages.
As the conversion process can take much time, it is recommended to perform the successive conversion of all used packages and save them after the executed conversion. Beforehand it is required to delete the derived components that are not planned to be used where relevant. If the component libraries with no edit access are connected to the package, it is necessary to request their developer to provide the new libraries versions. The same goes for the designed application solutions.
It became possible to show the required locale for the saved package in the new platform version. This option is relevant when it is required to process the multilingual text files or date-time data. When designing the workflow, one locale can be configured in the server settings, and the other one will be configured on the server where the designed workflow will be executed. It can cause locale conflict when starting the workflow. To solve this problem, the package locale fixation functionality was implemented in the new Loginom 6.5 version. The workflows configured in such a way won't be dependent on the language Windows and Loginom settings.
Different Loginom versions and editions provide different options. Sometimes there are situations when it is required to take into account the functionality differences when designing workflows. For example, it is possible to execute the SOAP requests in the server Standard edition, whereas there is no such option in the desktop Personal edition.
To provide unification of the workflows related to different platform versions and editions, global variables ProductVersion and ProductEdition that could be used for workflows design were added in Loginom 6.5.
Everyone who deals with data analysis is aware of the fact that data quality improvement is one the most important tasks. For this purpose, the special visualizer (Data Quality) was introduced into Loginom. It was implemented in the summary form by all types of problems.
Summary of the Data Quality Visualizer
The visualizer enables to analyse data set and access quality of this data specifying possible problems for each field. The general data set summary and detailed summaries for the continuousand discrete data are available.
Detailing of the Data Quality Visualizer
Only such standard indicators as Minimum, Maximum, Average, Null count are calculated when the visualizer is initially activated. Other indicators are displayed as follows: Not calculated. To calculate them, it is required to perform the following action: "Calculate quality indicators". List of the indicators calculated in this case: Extremes, Lower extremes level, Upper extremes level, Outliers, Lower outliers level, Upper outliers level, Median, Mode, Histogram, Unique, Quality indicator and Quality summary. The quality summary is based on the quality indicator and null count according to the indicator set in the settings - "Maximum allowable percentage of nulls". Then one of the values is specified for each field: "Unsuitable", "Suitable", "Preprocessing".
OLAP cube is one of the most popular visualizers. Its functionality was significantly improved in the new platform version:
Displayed Series
Headers Levels
When pressing on Ctrl+C, the data selected in the quick view window is currently copied in the Excel compatible format. Namely, when inserting to the table editor, the inserted values will have the correct data format. This option is available in the quick view window on the workflow panel for the quick view of the output ports content, quick view of results in the calculator and property inspector.
It became possible to set the periodic auto backing up of all opened packages in the server editions on the package configuration page. When pressing on "Auto save", it is possible to enable this parameter and also set "Auto save period" in minutes. The packages are saved with the set intervals in the files with .autosave extension near .lgp files. .lgp files are not rewritten in this case.
It became possible to cancel the request execution while importing from the database. This option is convenient in the case of incorrect writing of the complex requests to the databases for preparation of which DBMS could spend the long time that is sometimes comparable with the time that is required for receipt of the resulting data. Previously, it was required to wait when the operation execution was finished. Now it is possible to cancel it quickly and correct afterwards.
One more novelty is represented by the flexible configuration of the data import from the storage using variables.
Connection via ODBC drivers for Apache Hive DBMS using connection strings was added. The following ODBC drivers are allowed for use: Simba Hive ODBC Driver, Microsoft Hive ODBC Driver, Cloudera ODBC Driver for Apache Hive, MapR Hive ODBC Connector.
When importing from the text file for the automatic assumption concerning the used data types, the parameters set on "Configure Import Formats" page are taken into account as well as "Row count for analysis" parameter on "Import parameters with separators" page. Generally, it enables to specify the data types used in the imported set more precisely.
The automatic definition of the imported file structure became possible. When pressing on "Assume structure" button on "Configure Import Formats" page, the following parameters will be automatically defined and set: column delimiter, text qualifier, decimal separator, date separator, time separator, date format. .
When exporting to the Loginom Data file, the string data compression was optimised. The large data sets read speed was increased by 2 times. It is sufficient to export any data to the lgd format, and it will be possible to work with it several times faster, and it will take much less memory size in the warehouse. The lgd files read speed was increased by 10 times as compared with the text files and by two times as compared with lgd files 6.4.
To create the shared folder with the access rights provided for all users, including the newly added ones, on server, the standard user group "All" displayed when configuring the shared folders rights was created.
When it was referred to a user with "everyone" name, when starting the server, the following error occurred: "Configuration file loading error: Users.cfg" (the message will be kept in the Windows log and log file) and the Loginom Server did not start. To avoid this error, it is required to delete the user with "everyone" name in the previous Loginom version, and then update is required to provide the new Loginom version. If the Loginom Server has already been updated, it is possible to edit the xml file Users.cfg manually and delete the user with "everyone" name from it.
It is allowed to use such characters as point, minus and space in the user name and shared folders, such names can also start from a digit. And the names ending with a point or space are forbidden. This restriction complies with the restriction concerning the folder names in Windows. When entering the new user name, the leading and trailing spaces are automatically deleted in the Administration section.
When authenticating a user via the LDAP server, it is allowed to use the Russain layout characters. Previously, if the user whose password in Active Directory contained the Russian letters, could not log in Loginom via LDAP.
It became possible to configure packages publishing rights of each user on the Administration panel. Only the users with "Workflows design" role are allowed to get permission to publish the packages. When the new user with authentication via LDAP is automatically added, the user becomes entitled to publish the packages if he has "Workflows design" role. When starting the server after update up to Loginom 6.5, all users with workflows design rights will be also entitled to publish packages.
Now it is possible to set the role for a new user that is automatically added via LDAP using "New user role" parameter that can be configured in the Administration section of the LDAP section.
When fields set is generated, all invalid characters are replaced in the source name and the correct field name or the default value is generated if it is not possible to generate the correct identifier. Logics of fields names generation:
This change applies to the following handlers:Import from text file, Import from Excel file, Import from Database, Import from 1С Request, Import from XML file, JavaScript, Python, XML Extraction, SOAP request, Cross chart, Table to tree, and also "Data tree" type port.
The logged information was complemented in Loginom Integrator. When the "Info" logging level is enabled, request URL and node name are saved at the time the request execution begins. Previously, the node name logging was provided only for logging to the database.
Previously, if the REST request transferred via Loginom Integrator did not contain the request body or valid json object, the following uninformative error message appeared: "No property specified WebBodyFormatMessageProperty" . Now, one of the following messages is shown to the user in this case: "The request body must not be empty" or "Request must contain json object".
UTF8 (ANSI encoding was used earlier) is used for the Loginom Server logs files by default. The log files of the newly installed server will be in UTF8 encoding. When updating up to the new version, if the log file already exists, ANSI encoding will be used. It complies with the user locale used for Loginom Server start.
To use UTF8 encoding of the earlier installed server, it is required to make a backup copy of app.log log file and then delete it. If the automatic processing of log files was previously configured using the scripts in which it was assumed on the log files encoding, in this case it is required to pay attention to this change.
The following processing components were significantly improved in Loginom 6.5:
One more calculation parameter was introduced into the Calculator in Stat function — mode.
Fetch API support was introduced into JavaScript. It is especially important for the REST and JSON requests operation and access to the external web services. Support of the global property globalThis was added. It can be used by the third-party modules. setTimeout and clearTimeout functions were also added. Data encoding/decoding support in Base64 format was implemented.
"Allow invalid names" setting is not available in the wizard, whereas "Unique values as field names" setting is preserved. The previous version packages will be used in the same way until "Unique values as field names" setting is enabled. The following logics will be used for all newly created nodes: if "Unique values as field names" setting is active, the valid unique values will be used without changes as the column names, and the invalid unique values will be generated using transliteration.
After deactivation and repeated activation of "Unique values as field names" setting, the column names will be generated in a new way with transliteration.
Calculation of the model parameters was corrected in the linear regression as follows: Log-likelihood, Standard deviation, F-statistic, Model P-value, Coefficient of determination.
Calculation of the model parameters was corrected in the logistic regression: Chi-square, -2 Log-likelihood, McFadden pseudo-R^2 and McFadden pseudo-R^2 (adj.), -2 Log-likelihood of null model. Such parameters as "Wald Coefficient" and "Wald criterion" were renamed into "Wald statistic" parameter, and "R^2" was renamed into "McFadden pseudo-R²".
Output of degrees of freedom for model was added in the "Regression Report" visualizer for the linear and logistic regressions, and output of the Deviance null model was added in the the logistic regression.
The following parameters are added for the "Summary" output port of variables: for the linear regression — Log-likelihood, Standard deviation, Degrees of freedom for error, F-statistic; for the logistic regression — Degrees of freedom for error, -2 Log-likelihood of null model.
"Likelihood maximum" parameter name in the linear regression report was replaced with "Log-likelihood", and "P-value (model)" parameter name was replaced with "Model P-value".
If errors occur when working with 1C.Enterprise, it was complicated to define whether the problems connected with 1C or directly with the Loginom workflows occur. Now description is provided for all messages sent to 1C. Such typical errors are as follows: "1C connection error ...", "Error preparing 1C request ...", "Error executing 1C request ...", "Error preparing 1C request fields ...", "Error of 1C request results loading ...".
Now it is possible to exclude the columns from the further processing in Field features handler. For this purpose, such parameter as "Exclude" was added to the fields wizard. If "Exclude" property is set for the field, this field will not be added to the output data set.
It became possible to create fields or variables based on the existing ones in the fields and variables mapping wizards. All field or variable attributes are copied while cloning with the exception of the value of the connected field or variable from the source data set.
It became possible to copy and paste data via clipboard in Row filter component when selecting "listed / not listed" filtering criterion. It can be done using the context menu or the standard hotkeys (Ctrl+x, Ctrl+c, Ctrl+v for Windows OS).
It became possible to specify some filtration parameters using control variables in the Import from Data Warehouse Wizard. The following values can be specified using variables:
Filtration options by the "first", "last" date ranges using the dates from the data set were added to the Row Filter handler.
It became possible to open the package from the "Recent opened packages" list on the homepage. For this purpose, it is required to hold down "Ctrl" key while clicking on the package name ("Cmd" for MacOS). It is also possible to open the package in the new tab when pressing on "Create package", "Create draft", "Open package" buttons with "Ctrl" ("Cmd" for MacOS).
TLS 1.3 support was added. OpenSSL 1.1 is used for the secure connections. When working via HTTPS and WebSocket, TLS version below 1.2 cannot be used.
Sorting of the discrete fine classes of the Coarse Classes node was replaced with the stable one. Previously, the fine classes model could be changed while execution, and the execution result could differ from the expected one.
Collapse to supernode button is active now even if one node is selected and this node is not supernode. Previously, it was possible to collapse only several selected nodes to supernode.
The cells with uncalculated parameters and calculation start buttons are explicitly specified in Statistics visualizer. The unique values are highlighted in orange, the order statistics are highlighted in red.
When exporting to the databases, if "Update existing table records" export type is selected, and the key field contains NULL values, the correct comparison of values is performed with the use of IS NULL construction.
When starting the desktop Loginom editions, the working directory specified in the shortcut parameters is installed as the working directory. When installing the platform, "My documents" folder of the current user is used as the working directory.
