Technology of data migration in large projects. Migration database data migration methodology

how to implement free

Migration to free software like migration to a newer operating system. As an example, you can mention the appearance of the first Windows options in our country. No less bright example - migration on Windows NT, the ideology of working in which differed sharply from Windows 9X. You can give another example - each new version of the MS Office package differs from the previous one not only differences in the interface, but also file formats. So, the migration task is relevant even in such a case when used by from a single manufacturer.

This article proposes a general description of the migration methodology with the coverage of significant moments. The general principle of migration is thoughtful, careful implementation of the process by gradual changes. Migration consists of several logically integral sites, stages.

creating a working group (who does)

When making migration, it is necessary to provide solving issues of both technical and non-technical nature.

It is important to consider the legal problems that recently are highly relevant for some CIS countries, in particular, Ukraine. In some cases, intelligently discuss the administrative tasks of the "Employer-User Administrator" relationship. Historically, these relationships are not sufficiently regulated by the internal corporate rules and instructions.

In the process of preparing the material, conversations were carried out with professionals in the field of security, computer law and system administrators. The overwhelming majority declared the need for documentary design rules for users with the organization's information system.

Proper planning also includes solving financial issues. It is necessary to assess the costs of legalization of the existing information system, the cost of introducing a new one, assess the cost of ownership in the foreseeable future.

Any project, including a migration project, may face the underestimation of the human factor. Naturally, the use of human resource management methods will be required. Most of the most famous authorized system administrators and IT managers are not experts in personnel management or finance areas. A similar comprehensive task cannot be solved by the same IT Department.

The first task on the way to migration to a new one, the target system is to create a working group of migration planning. This group is responsible for conducting migration and, therefore, must have quite broad powers.

The purpose of the project is to build an economically justified IT infrastructure. A good candidate for the position of head of the working group will be the top manager of the enterprise or organization, for example - financial director. Naturally, this group includes the head of the IT department, which owns the vision of the entire IT infrastructure, both at the moment and in perspective. As part of the Group, an experienced system administrator is required, preferably with the experience of operating free software. The size of the group cannot be estimated - in some cases, other employees of the company are involved. It is possible to attract a third-party consultant with experience, or - specializing in such decisions of the company. The result of this group is a detailed migration plan with an assessment of the value of migration. Either - explanations of the ineffectiveness of migration for free solutions for the organization.

research (what is)

The first step should be an audit - a description of the existing (inherited) system.

It is no secret that, in the years approved by the "avral" informatization, without taking into account the payback spent on the funds, the result was, as a rule, not only economically, but also technologically unbalanced systems. Audit of the enterprise software is a revision of the installed programs, determining compliance with their business requirements.

The result of the audit process is:

Description of the technical characteristics of the installed software;

A list of identified risks associated with the use of unlicensed software;

Counting the cost of acquiring licenses for the installed software;

Counting the cost of removing unlicensed and installation of licensed software;

Determination of the feasibility of further use of software;

A list of identified risks associated with the use of software;

inventory by

Some studies show that most organizations of organizations pay more attention to the functionality of the software used and much less - the rights to the products used are fulfilled.

Unfortunately, in most organizations there is no IT culture. Sometimes even representatives of IT services really do not know what and where installed on computers from workers, and ordinary employees independently decide and establish software products obtained from dubious sources.

The inventory of the software allows you to identify unlicensed software in the organization. It should be emphasized that this event is always beneficial. The result of the inventory can be used to estimate the costs of legalization according to both the use of free and using non-free software.

Organizations are often interested in a clear financial planning of the cost of using and developing software. The interest of top managers in such such detail is quite understandable - the top management of companies is interested in turning the software in the asset of companies, which is taken into account, is monitored and developed similarly to other types of assets.

The audit of the procedure, which, as a rule, is quite a long time, requiring at the stage of analyzing highly qualified personnel information and knowledge of various specific information. The recommendation may be an appeal to the company specializing in these services. Nevertheless, it is quite possible to audit by the IT Department.

In some organizations, it is inconvenient (often it is impossible) to carry out a large-scale inventory of software. The reasons may be the size of the organization or security policy. It is necessary to find a compromise between inventory efficiency and factors complicating such a process.

You can consider two options for audit. The first option is a complete audit about which all computers, a local network and a perfection are studying. The advantage of this method is high accuracy, disadvantage - big cost, high time and inconvenience for users. Additional advantages of this method is the ability to identify independently installed by users of software and explore user requirements to software on their workplaces using specially prepared questionnaires. The second option is an audit of some typical computing tools, a local network and a perfection. At the same time, the choice of audit objects is dictated, as a rule, functional duties of users. Such an approximation method significantly reduces the cost of inventory, but has a great error.

Small organizations can conduct an audit manually, and make information about computers and servers, as well as the software installed on them, into an electronic simple table. At the same time, you should specify the presence or absence of necessary licenses, authentication and copyright certificates for each of the found software product.

For medium and large, it is possible to recommend the use of a specialized software or invite a third-party organization specializing in similar services. In the process of creating a document, work was carried out on the review of the automatic inventory of software and hardware (GASP, PC Inventory, MSIA) were carried out. EXPONENT NAVIGATOR (http://www.e-x.ru/pages/expnav.html) can be a recommendation (http://www.html), Exponent.

Exponent Navigator

The product is designed to revise equipment and software over the network. Computer information includes accessories information (processor, motherboard, hard drives, memory modules, video card, network cards, printer, and other devices), operating system, drivers and software.

According to the Creators of the Program, after organizing the automatic information collection of computers, it is possible to view and organize this information, prepare printed reports and web publishing, unload data to Microsoft Excel, XML and other formats. Capabilities:

Automatic diagnostics of computer configuration;

Automatic collection of information about network computers;

Determination of installed equipment;

Definition of installed programs;

Definition of file characteristics;

Advanced sorting, search and data selection capabilities;

Preparation of printed reports;

Export data in MS Excel;

Automatic generation of web publications.

In the free version of the program there is a limit - accounting to 25 computers; The license cost is $ 1 per 1 computer.

design (what we want)

Processed results of the study of the existing system are the basis for modeling a new, target system. This question is extremely important and complicated. It complicates the consideration of this issue and the historically established lack of acquaintance with free software, in particular Linux, most IT managers and system administrators.

There is a lot of literature, including Russian-speaking, about Linux, which describes the advantage of this platform from a technological point of view. However, all these advantages are important with the main issue - the existence of a wide range of applied different directions. A long time ago, the myth is widespread that under the Linux platform there is a limited number of applied software for corporate use, including office automation. In the overwhelming majority, these myths are created and fueled by the creators and sellers of proprietary software and have little in common with reality. The debunk of this myth is not the main goal of this book. Nevertheless, it is worth noting that, for example, a latitude of applied software is an absolute phantom, taking into account the established historically standards for the exchange of documents. So, for example, the actual office of the office editor is Microsoft Office, the raster graphics editor - Adobe Photosop, and the Corel Draw is highly distributed as vector graphics.

Another issue is, often, the redundant functionality of proprietary products dictated not to the needs of the market, but the opinion of marketers. And for this redundant functionality, the user pays quite large money: the cost of a license for the right to use programs that increases the complexity of operation and increased hardware requirements.

Recently, the situation changes - a mass of information devoted to the application of Linux appears. Probably the best material will be this document :-), in which it is planned to highlight many administrative and technical issues.

However, at the moment the document is only being created and information can be found in different sources. It is impossible to pass by the materials of Valery V. Kachurov, Neskov Artem "Analogues of Windows programs in Linux - Table of Compliance." (http://linuxshop.ru/linuxbegin/win-lin-soft/). This resource contains a lot of valuable information. Unfortunately, the authors seem to abandon this work. This section of the site is regularly unavailable, but a copy of the table can be found at http://www.blif.net/modules.php?name\u003dlinwin. You can advise Open Source Applications Foundation resources
(http://www.osafoundation.org/), especially http://www.osafoundation.org/desktop-linux-overview.pdf.

The result is:

Creating a prototype of workstations;

Counting the cost of licenses for the proposed software;

User learning;

Creating an exemplary calendar implementation plan;

List of risks in the introduction of free software;

Support free solutions;

Counting the economic efficiency of the new system for up to five years.

pilot Project (Checking Boot)

Due to the large number of factors in the inherited system, a large number of users who potentially affect the migration, it is recommended to experimental fulfillment of migration in a small scale - a pilot project. This stage is required to verify and adjust the migration plans and the prototype of the new system. It is a pilot project that is the basis for making a decision on the introduction of a new information system based on the SPO. The second value of the pilot project is the ability to inform users about the new system, to get feedback from users. The third argument in favor of the pilot project will be the possibility experimentally determine more accurate project cost.

When choosing a test object, you need to find a gold middle. First, the selected object must provide reliable data to evaluate. Secondly, the pilot project should not have a critical impact on doing business.

At this stage, system administrators and end users are also trained: a prototype of methodical materials, documentation, resources on the Internet are provided. It is recommended users who participate in the pilot project to "unload" and enable part of the time to use the new system.

The experimental stage is especially in demand:

If the possibility of migrating users from the inherited system to the new system was not proved;

If there is a custom skepticism, which will be able to slow down the migration process;

The organization lacks corporate culture (which, unfortunately, is distributed in the CIS);

If there are limited resources for large-scale migration;

The organization is large, and many users are not involved in the experimental project;

If inherited proprietary systems were rapidly evolved both in - technical terms and in reducing the cost;

The fully economic effect of migration is not found out.

To succeed, pilot project:

Should not refer to the critical area of \u200b\u200bthe business;

Must be sufficiently important for business;

Should not require an intermediate resource of people who are already limited in time;

Must have a significant support group;

Feedback from users (Help Desk systems) must be provided;

It should not be in the sphere of other limited (for example, an important period for business).

Depending on the size of the organization, one or more experimental stages are possible, which make it possible to more accurately determine the lines and the cost of migration to free. It is important to estimate its results and determine whether the vision and purpose of practical or they should be changed or maybe even left. Data from these pilot projects should be used to adjust the plans and counting the final cost. In the process of the experiment, you need to refine questions:

Description of the prototype of workstations;

Description of specific user settings:

Average costs of deploying stations type;

Transfer data from the inherited system to a new one;

User learning;

Counting the cost of implementing software;

Support free solutions.

planning (what and how)

1. Creating a migration plan. The migration plan must answer questions:

Description of the system construction phases;

Definition of support needs;

Description of the completion of migration.

In fact, the final choice is made, as migration will occur. Approves the estimate of migration costs.

2. Description of the phases of the system construction. Evaluation of the phases of building a system that best support custom needs priorities should be made.

The plan must be responsible for the following questions:

To what extent and what stages the system should be installed and deployed to maximize the needs of users?

What is needed for each phase of migration to a new system from customers organization and user users?

What will be the impact and risk of using the system at each stage of increment?

The stages of the deployment of the system must be clearly designated in terms of migration; Customers, developers, and users must be familiarized. Risk assessments must be completed before the conclusion of the system constructing plan. It is necessary to make sure that planning evaluations are reasonable, the approach is well conceived in accordance with the priorities of the organization, and the potential impact on customers and users is acceptable.

3. Determination of support needs. It is necessary to ensure the optimal level of support to help users use the new system. In addition, users often require support to help them understand the general abilities (possibilities) of the target system.

Questions that need to be solved include:

What learning and helping users will require?

What is the overall level of migration support that users will be required to guarantee successful migration?

How to achieve the adoption of the target system by users and avoid the resistance of users to implement a new system?

How will customers and users who inevitable changes in the features of systems and services will be reported?

Is it possible to support the free solutions of the IT unit of the organization or the best option will be outsourcing?

The migration plan should focus on solving these issues, planning user support in areas:

Elemental communication system;

Technical assistance services for new systems;

Technical assistance to users migrating to the new system;

User manuals for transition and subsequent periods;

Learning for users in learning and adapting to the new system, to perform the same types of tasks;

The ability to test the use of new systems;

Demonstrations of using new systems to show existing users inherited systems, as a new system works and how they can perform comparable tasks;

Overcoming current operational problems.

At the stage of user training, you will pay special attention to those who are a supporter of the old system and / or an opponent of new systems.

4. Description of the completion of migration. After completing each new phase of deployment and training, developers and implementations should ensure that users migrate as comfortably as possible to the new target system. The migration plan should provide to speed up the force of migration and to remove the inherited system as soon as possible.

An additional effort must be provided that it may be necessary for the "latest adepts" and other users who are experiencing unforeseen problems. Another aspect of this activity estimates the time and cost to complete the transition of all users to the new system and to remove old systems constructed on non-license zone proprietary software.

The management of the organization, developers and implements should consider the inclusion of several approaches to help migrate to users inherited systems:

Report to each group of users, as and when they must make the transfer of their tasks to new systems, how the workload changes for the period of migration with inherited systems;

Establish stimuli to action to completely switch to a new free system and eliminate dependencies from inherited systems;

Provide assistance (software and additional personnel) to convert inherited data to a new system; - the order of the output from the operation of inherited systems;

Archiving data inherited systems and storage.

migration (do)

Everything that remains at the last stage is to work according to plan.

Actively manage and monitor the migration processes:

Set measurement criteria and monitor the migration steps and resource costs for migration;

Make periodic reviews of the situation and familiarize yourself with them, according to the authority and organizational policies, interested people (management, project managers and sponsor);

Set the tracking system (Tracking) to manage the promotion of processes (progress), problems, solutions, and other business issues related to the planning of migration and execution of plans.

Vadim Mashkov, UA-FOSS, [Email Protected]

In this article, we would like to systematize our experience in conducting data migration in large corporate projects related to the transition of customers to work in the configurations "1C: Enterprise 8".

At the same time, the main emphasis in the article will be made, first of all, on the technological component of the migration process. The organizational component is also affected, but to a lesser extent.

Terms and Definitions

Under the data migration, it is customary to understand the final sequence of work, a project aimed at one-time mass movement of data from source systems (historical systems) into the receiver system. In this case, the operation of this data in the source systems is terminated.

It should be distinguished migrating data from data integration. Integration, in contrast to migration, is the permanent part of the IT architecture, and is responsible for data flows between different systems and data warehouses - and is a process, and not the implementation of the project.

The migration scheme generally looks like this:

Fig. one

Historical systems - Customer's company databases, which are planned to be fully or partially replaced when implementing a new system.

Receiver system - Target system, arbitrary configuration "1C: Enterprise 8".

Initial data - Data unloaded from historical systems into an arbitrary XLS format. In this case, the XLS format is represented as one of the most convenient, since the possibility of unloading to the XLS file is present in many accounts of "previous generations".

As a modern alternative, it is possible to consider the XML file format.

There are also options for using an intermediate database.

Transformation, conversion - The process of converting the source data to the data for download. The data transformation occurs in accordance with the load templates. The result of the transformation is the data for download.

Data for download - Data designed to boot to the receiver system. In this article, as well as the initial data, XLS-Format is considered.

Data Templates for Download - Description of data tables to download to the target system.

Stages of migration

Consider the phased process of preparing and conducting migration.

The following items include the organizational stages of migration:

· Determination of migration strategy. At this stage, the Contractor and the Customer agree on the technology of migration work;

· Determining the composition of the Migration Working Group. The working group should include specialists and artist and the customer, familiar to a sufficiently with the work of historical systems (from the customer) and the target system (by the Contractor);

· Preliminary migration plan. The migration plan along the project will be repeatedly adjusted;

· Periods of data unloading data from historical systems, data volumes. Periods of data cuts for migrations, dates of test and final migrations. This information can be attributed to the migration plan;

· The composition of the data to be migrated. Reference data, classifiers, transactional data, residues, turnover, etc.;

· Questions of quality verification, correctness and integrity of data in the process of migration and, according to the results;

· Questions to roll back to the previous state in case of failures.

Let us dwell on the technological stages of migration.

Fig. 2.

1. Preparation of data load templates

The data load template contains technical descriptions of data tables for download, algorithms and load rules for the current template.

Each template is generally designed for one or more related tables in the target receiving system.

The template indicates:

· Description of all XLS fields for download, including:

o Field name

o sign mandatory field fill

o Field Fill Example

o Note

· Description of the rules for downloading the target system table based on data for loading (session in the case of several related tables, search algorithms for key fields, etc.)

· Description of the fill in directly fields of the target system tables in case anything different from transferring data "one to one" from the data file to download. Actually for reference fields, for example.

In the process of work on this stage, the Contractor must also prepare the data loader for downloading data. In the case of working with XLS files, this task is not particularly difficult.

2. The evidence of data sources

This stage can begin with the previous step "1. Preparing data download templates. "

As part of this stage, the customer's specialists determine which systems and which data can be unloaded. You should also determine what data possibly May be needed.

As a rule, in large migration projects, the detection of a complete exhaustive list of data sources can occupy sufficiently long time and occurs as work at subsequent stages.

It is often of the situation when, to ensure further integrity of information, some data have to be transferred from printed sources (digitize) or even enter the tables from the words of key customer employees.

However, at this stage you need to try to identify as much as possible data.

3. The load of the source data

The process of unloading data from historical systems can take a sufficient amount of time, especially if there are many systems, they are different and the customer's different divisions are responsible for them. It is necessary to take into account the moment in test and final migrations.

The most convenient option seems to unload in XLS files. Many old IT-Systems support such an option.

There may also be unloading options in CSV format, DBF, XML formats and others.

It is worth noting that for one reason or another (security issues, for example), the customer can not always provide uploading data in full at this stage! Only data structure and several test items. Thus, such a situation may arise that during test and final loads, poor-quality data will be found in the source tables, which will lead to unplanned errors.

To minimize this problem, in advance the volume of test unloads from historical systems should be stated.

4.Mapping data

Mapping (Data Mapping) - in general, the process of mapping data of historical systems and a receiver system. That is, the source data and data for download.

The Mapping Stage is the most time-consuming stage and can take more than 50% of all work on the task of migration.

At this stage, the entire working group of the Migration Project is fully involved.

In the process of mapping data, it is necessary to highlight the formations of mapping tables and mapping fields.

· Mapping tables, or mapping templates - mapping of source data tables and data templates for download. Compliance can be as 1: 1 and n: n. As a result of this work, the registry of the mapping tables is compiled and maintained. This sub-step is needed for the next sub-stage of mapping fields and to track the overall state of mapping.

Team group 1C.

Template name 1c

File name

source

Rules for forming a source file

Responsible

Status

Note

NSI

Template_

Nomenclature

Nomenk.

latural .xls.

In the system n set the selection
. Save in txt
. Open in XLS, columns - text
. First string - hat
. Number of columns - 15
. Check the number of rows in TXT and XLS
. The name of the sheet is always "sheet1"

Ivanov I.I.

in work

· Mapping fields - mapping of table fields within the framework of the already defined mapping tables. The result of this work is the registry of mapping fields.

ПП

CL. field

Mandatory

The name of the 1C template field "template_namenclature"

Description

Name of the field "Nomenclature.xls"

Filling algorithm

The code

Code of reference item

The code

Name

Yes

This group

Contains one of the values:
. 1 - for groups
. 0 - for elements

If code length \u003d 11 characters and the last 4 characters<> "0000", then this element is "0", otherwise the group is "1".

Full name

Name of reference element

Name

If the etco group \u003d 1, then "", and the etco group \u003d 0, then the name.

Within the framework of this stage, it should also carry out possible data on data normalization.

5. Preparation of transformation rules

Unlike previous stages, this stage is technical and involves the work of the developer of the Contractor.

Based on the agreed registries of mapping fields, the artist's specialists develop data transformation rules.

For operational work in the process of preparatory migration stages and further, in the course of test and final migration, it is important that there existed a convenient development environment for the data (scripts) of data transformation and the source data conversion environment to the data for download.

At the same time, the requirements for this environment include:

· Convenience and speed of development of transformation rules;

· Data conversion rate. The files at the entrance and the output can be hundreds of thousands of rows!

· Ability to work with multiple input files at the same time;

· Ability to save transformation rules to individual files.

For its migration projects, we have developed a specialized developer AWP, taking as a basis the standard processing of "Console requests" 1C.

Processing "Request Console" was finalized to be able to make direct requests to XLS files.

Let us give an example of combining two source XLS -Files Staff.xLS.

Employee Code	Surname	Name	middle name	Date of Birth
2423	Ivanov	Ivan.	Ivanovich	17.11.1992
1523	Petrov	Basil	Aleksandrovich	04.02.1991
4363	Sidorov	Kirill	Nikolaevich	01.05.1995
	Denisov	Denis	Denisovich	01.01.1990

and Operations.xLS.with pages:

Writing off

Employee Code	date	Sum
2423	01.02.2014
1523	02.02.2014
4363	03.02.2014
	04.02.2014	100000
2423	05.02.2014
1523	06.02.2014
4363	07.02.2014	2356
	08.02.2014	140000
2423	09.02.2014
1523	10.02.2014
4363	11.02.2014	23523
	12.02.2014	80000

and Arrivals:

Employee Code	date	Sum
	01.05.2004
	02.05.2004
	03.05.2004
	04.05.2004







2423Date of Birth	Amount of arrival	The amount of write-off
Ivanov Ivan Ivanovich	2423	17.11.1992	1341234	1010
Petrov Vasily Aleksandrovich	1523	04.02.1991	245245
Denisov Denis Denisovich		01.01.1990	380000	320000
Sidorov Kirill Nikolaevich	4363	01.05.1995	613382	26336
TOTAL:			2579861	347842

Note that an example is artificial, specially selected for the demonstration of all possible stages of data sources transformation.

The technological sequence of transformation operations here is as follows:

Using the Access SQL query language (gives significant additional features, compared with the 1C request language), an initial request is created, extracting data from the XLS file to 1C. At the same time, at this stage there are various checks and data normalization.

Access technology ADO provides high speed.

Fig. 3.

2. Subscriber in 1C language - the main query that implements the field of mapping fields. As well as: enrichment of data downloaded data from the base 1c, rearrangement, combining with the results of queries to other source XLS -phals, etc.

3. Concessing the result of the query 1C if necessary. It is implemented using a script in 1C.

For example, the adding line "Total" in terms of sums is implemented here.

4. Recover the outcome dataset in the XLS file.

In general, at the output, we receive final files to download to the 1C target database.

Also, this tool allows you to save data conversion rules to a separate XML file:

In addition, it is implemented to work in batch modethat is especially relevant with a large number of heterogeneous migratory data.

During the previous steps, the preparatory part of the work is generally ends - all data sources are revealed, the source data is unloaded from sources, the download templates are prepared to the target base, mapping data and finally developed data transformation scripts.

It should be noted that before the final migration, several tests should be carried out. During test migrations, the Contractor, together with customers, detect:

· Conversion errors, data load errors

· Conduct a preliminary assessment of the quality of the data being downloaded to the target system.

· According to the results of test migrations, the basis of the final migration is drawn up / update

7. Master of data

Checking the quality of downloaded data should be made both after test migrations and at the end of the final migration. During the reconciliations, the following indicators may be checked:

· Coincidences of final amounts on the remnants, according to documents;

· Quantitative coincidences, such as the number of OS;

· Correctness of filling out individual sample entities;

Please note that those or other verification of migratory data, data normalization issues need to be solved throughout all migration processes. It is always necessary to wonder what needs to be done at the current stage to avoid errors at subsequent stages.

For example:

· Check on duplicate on key fields. You can and need to be carried out on the source data;

· Bringing field types;

· Reference integrity;

· Mathematical inconsistencies. For example, checking for empty numerical fields on which division division is scheduled;

· In general, verification of mandatory full fields;

· Replacing incorrect characters. For example, English characters in Cyrillic fields ("O", "A", "E", etc.) especially relevant for key fields!

· Checking the values \u200b\u200bof string fields for compliance of the type of receiver system (limits in length)

After completing the final migration, according to a certain strategy of migration and the migration plan, a decision is made to further exploit historical systems.

Often, exploitation is completed immediately after the final reconciliation of data and fix the success of the migration - the users of the new system are no longer recorded in parallel in two systems, but are completely moving to a new system. In this case, access to the old system is saved in reading mode.

In some cases, the parallel operation of two systems during the trial operation (OE) can occur and even more than this period. The question of parallel operation of users in two systems is closely related to the possibility of a rollback to the old system, if the migration (or, in general, the work of the new system!) Will be considered unsatisfactory.

Conclusion

In conclusion, I would like to note that when it comes to the migration of large transactional systems, which include many configurations "1C: enterprises", the transition to a new system can be very time consuming.

Therefore, it should be remembered that any similar project requires careful preparation and must be accompanied by an individual plan. However, regardless of the type of migrated systems, database volumes, etc. The overall migration scheme looks almost identical.

Leif Polesen for Intech

The automation systems for the production and collection of information on production processes live relatively long. They are often upgraded or replaced before process equipment reaches the end of its life. For many companies, the management of the replacement or modernization of such automation systems without stopping production turns into a real call. Therefore, the objective need for upgrading or replacement is ignored until anything happens. This article is devoted to how it can be successfully solved by this task, thanks to careful planning and organization.

Two main factors are as needed to upgrade and replace industrial automation systems and industrial IT systems: technical degradation of these systems, as well as changes in the requirements of business processes that are supported by these systems.

Reliability of technical systems will decrease over time if companies ignore the need to upgrade operating systems, databases, by applications. The operational risk of equipment failure increases accordingly.

Thanks to careful planning, the operational risk can be held at an acceptable level, in addition, investment protection is provided and expenses throughout the system life cycle are minimized. For a typical automation or IT system, only 20-40% of investments are accomplished by the system. The remaining 60-80% go to maintain its high availability and adaptation to periodically changing requirements.

In addition to assessing the activities necessary to prevent technical degradation, new challenges must be taken into account, as well as potential business opportunities. The business environment is constantly changing, and all the possibility of improving existing or introducing new technologies should always be considered. Typical business opportunities that can become incentives for migrating notes automation systems - the rate of output of new products to the market, competitiveness, growth, quality and compliance with regulatory requirements.

Long-term migration plan

Development of a long-term system migration plan allows companies to support systemic risks at an acceptable level. In addition, it provides risks management and timely support of business goals. The migration plan must necessarily take into account such restrictions as "best production practices", technology functionality, inevitable ease of production.

In general, the approach to long-term planning is depicted in Fig. 1. The migration plan is being developed in order to determine where the company wants to be in five years, what is necessary to make actions for this and whether resources required for this are required. This approach is based on the principles of architectural design set out in Togaf Standard, which is widely used in the development of systemic architecture of industrial enterprises.

Figure 1. General approach to creating a long-term migration plan.

It is necessary to distinguish between the existing architecture and the target, desirable. The difference between them reflects the difference between the position of the company at the present time and the situation that she wants to take in the future. The migration plan paves the path from the existing architecture to the target - perhaps through several transitional stages.

Each architecture can be described as a series of "layers" serving the link between business and technology - as shown in Fig. 1. Attention must be given to the following "layers":

Business goals This is part of the overall work planning strategy. They allow you to choose the correct direction of the process.
Business model Provides a context in which production and business processes are understood. As a rule, it includes a high-level description of the streams of materials and processes.
Description production and business processes It is important for the successful application of technology and the correct assessment of their value from the point of view of business.
Information, Data and Documents Important to communicate processes and applications. Especially important interaction and management of information flows between applications.
Descriptions appsallow high-level requirements and identify interfaces.
Definition infrastructure, computing and network Requirements (hardware, fault tolerance, performance).
Provided services Determine the requirements for ensuring efficient operational management and solutions support.

Development of a migration plan

Development of a migration plan for a whole organization, or even for a separate production platform, may be a truly complex task involving many people. It is recommended to separate the development process into several steps described below.

Part II.

Stage 1: Mobilization

Basic goals:

achieve a common understanding of tasks and goals
mobilize the organization in which the project is planned
detail the plan, describing the milestones and the results of the project phases
collect all the necessary / available information
provide proper understanding of concepts, practices and theory

planned meetings
workshop dedicated to the project

Results:

detailed consulting plan
common goals
process Overview

Stage 2: Analysis

The purpose of the analysis stage is:

analysis of business and production processes in order to:

Assess the readiness of the staff serving the IT and automation system

Find out the need for data and functionality for future architecture

Identify the key advantages of future architecture for the setting of goals and the implementation of the Economic Justification

analysis of existing architecture

Determination of existing production processes in their connection with automation systems, data collection, production management systems

Determining existing business processes and their links with production automation systems

Determining existing applications, data, logical and physical infrastructure, as well as technical support services

During this stage, the following activity is carried out:

seminars and discussions on various processes
enterprises visits to obtain contextual information
seminars and discussions dedicated to existing systems
evaluation of services to determine the degree of their maturity and compliance with the regulation requirements

Results:

determining the existing infrastructure
analysis documentation
list of ideas on the challenges and capabilities of the new architecture list of ideas on the challenges and capabilities of the new architecture

Stage 3: Purpose

The purpose of this stage is the definition and description of the needs formulated during the analysis stage.

Solution, or target architecture will describe:

future business processes and functionality
target types of applications, with their functionality, users, information and interfaces
infrastructure requirements and revised support standards

During this stage, the following activity is carried out:

seminars and discussions on improving processes
seminars and discussions on improving architecture

Results:

future architecture (presentation)
brief description of applications

Stage 4: Justification

The goal of the substantiation stage is the primary economic rationale based on approximate estimates of the costs and the benefits obtained during the project.

The gap between the existing and desirable position is usually leading to the emergence of a number of ideas. The rationale of ideas will make it possible to distinguish the "necessary" from "desirable", and, after that, to submit and develop ideas to top management.

During this stage, the following activity is carried out:

approximate estimation of expenses and benefits
the first version of the presentation

Results:

common goals
changing business ideas
evaluation of the necessary resources

Stage 5: Plan

The purpose of this stage is to plan a project based on priorities, resources and dependencies:

planning the sequence of the stages of the consolidated project
providing resources and competencies required for the following steps
project Management Activities
completion of consulting and the transfer of the results of all stages to the customer

During this stage, the following activity is carried out:

development plan for implementation
development of an investment plan
risk assessment

Results:

implementation plan
evaluation of the workload for personnel involved in the project
estimation of the risks of the project
investment plan (in the first approximation)
final version of the presentation of the project

Practical example

The following example illustrates the application of the described approach in real conditions. To comply with confidential conditions, anonymity is observed in the description. It is about a fairly large enterprise producing active ingredients for pharmaceutical products. Production facilities were put into operation more than 20 years ago, and, although since then, a certain modernization was carried out, a number of outdated systems requires replacement. The automation systems of buildings and RSU are first of all, as they are based on obsolete technologies that are difficult to maintain. In addition, production should adapt to new business requirements, including the cessation of the production of alone products and the launch of others. In general, there is a need to work on a migration plan covering both technical and business requirements.

First, it is necessary to create a general list of equipment that is currently used throughout the enterprise. This information is often "hidden" in various documents (and staff memory). It must be removed and visualized, so that it becomes the basis for migration planning. To this end, we, as a rule, create a process chart of process modules, showing the main equipment and the movement of raw materials and materials in each industrial division. As separate layers of "top" equipment, we show which systems which equipment is supported.

An example is shown in Fig. 2. Data on the established systems, also contains in the system storage (or simply in excel files), and they can be used for further analysis and planning.

Figure 2. The "layer" of automation allows you to evaluate existing systems

Before discussing the migration plan, it is necessary to determine the main business motives of changes in production. In the case under consideration, the management allocated the following motifs:

1. steady and error-free compliance with regulatory requirements

2. Minimum time costs to enter the market, flexibility

3. Success, competitiveness, operational perfection

4. Uncompromising quality

5. Growth production

These goals need to be transformed into more specific tasks, the execution of which can be quantified.

Then, we must find out how well-existing systems support current and future business processes. To do this, we use a standard reference model (based on the ANSI / ISA-95 series). It includes 19 high-level business processes detailed to the extent that allows you to see weaknesses in their practical implementation and the need for change for effective business.

In addition, we also need to evaluate the technical capabilities of existing systems to support business processes in the future. This is systematically implemented using the information storage described above. For each system, information about which is in the repository (in our case it is about 70 systems), it is necessary to assess the following aspects:

Condition of equipment (bounce history, average time of operation for failure, age of equipment, accessibility of spare parts)
Condition of software (support for vendors, availability of documentation, personnel with the necessary competences)
Opportunations of systems (reservation, average service life before repair)
Impact impact assessment (provision of information, data errors, inaccessibility)
Indicative indicators (reliability of the system, critical importance of the system, etc.)

The technical assessment revealed the need to modernize and replace a number of systems:

The TP ACS is based on the usual, outdated RSU and many different PLCs, a number of which are already "matured" to replace.
The building automation system is based on a newer platform, but also requires upgrading to meet new requirements.
A number of secondary systems also require upgrades, or even replacements.
The infrastructure serving all systems requires better segmentation and protection for compliance with modern safety requirements.

Part III

After analyzing the business targets scheduled for the future, it became apparent that none of the existing systems correspond to the future needs fully. This understanding served as a reason for the emergence of a number of ideas about the introduction of new technologies, as well as a production executive system. According to the results of the analysis, 16 different projects were proposed, which, with consistent implementation, will help the companies to meet future technical and commercial requirements.

The maintenance of technical works and the cost of each project is estimated; For each project is preparing a brief single-page summary, which can discuss management. (See Fig. 3).

Fig. 3. Single-page description of the potential migration project

In order to choose priority projects, the potential results of each of them are evaluated. The results are evaluated from the point of view of business purposes, as well as the reliability of the ACS TP.

As a rule, you will have to evaluate several implementation scenarios to assess the overall need for resources and financial resources for each plan (Fig. 7). One of the main limitations that should be considered are "windows" in the production process, during which you can replace or modify systems. As a rule, these "windows" fall on the weekend - and this is a serious "bottle of throat."

Fig. 7. Consolidated migration schedule review

Since the time to replace the systems and their adjustment is always "in the edge", the preparation must be very careful. Everything should be melted in detail. An important aspect of planning is to verify the implemented systems.

In the case described by us, the implementation of the long-term migration plan was performed in six different streams, see rice. eight.

Fig. 8. Organization of migration projects in six different streams

Part of the preparation is a thorough assessment and prevention of the risks of the project. In fig. 9 shows typical risks characteristic of migration projects.

Fig. 9. Evaluation of typical risks of migration projects

Processes for business support

The approach to the management of the life cycle and the planning of long-term migration described in this article is determined by the needs of the business. It includes an assessment of current and future business goals, as well as a thorough analysis of how technical systems will be supported or replaced for the best support for these purposes. The approach is based on the principles of Togaf, which provide for the consistent implementation of the project, depending on the availability of budgets and qualified personnel. The assessment of current and future system architectures is a key element in identifying future migration projects. Finally, it is necessary to comply with the principles of management of changes in the organization, which ensure the timely involvement of key project participants, so important for the success of migration projects. The effectiveness of this approach was repeatedly demonstrated in practice.

Leif Poulsen) ( ), leading specialist in the field of automation and IT in NNE Pharmaplan. It has a workshop degree in the field of technological management. NNE Pharmaplan Polesen is responsible for the development of technologies, methods and competencies in the field of industrial automation and IT, and works as a senior business consultant.

The PLM industry has been developing for more than ten years and the moment that enterprises, testing the decisions selected initially, are quite reasonably decide to change the existing PLM to another, or significantly change the configuration of the current one. And then there is a question of migrating PLM data from the old system to the new one. As we know, even the broadcast of the geometric data themselves is not easy and ambiguously solved, the migration of PLM data is even more complicated and still a little-known process.

As it happens

The choice of a new PLM system was made, a team of executives with relief celebrates the result of the complex efforts of this choice. In fact, only now the present work, planning and making data migration from existing systems to a new PLM target platform begins. Existing companies contain a significant part of intellectual property (IP) and, therefore, the competitive advantages and capital of the company. The success of the introduction of a new system largely depends on the performance of effective migration of existing data, intelligent, efficiently and timely.

This scenario is regularly repeated in one or another world and in any industry. With a rapid rate of improvement of technologies, strengthening competitive pressure, leads to the fact that many companies constantly evaluate their PLM solutions and improve improvements. The decision may be caused by the recognition that the current solution, or, more often, its maintenance, or the use of several systems, is not satisfied with the management of the company. In addition, there is another common motive - the consequences of the company's acquisitions and the realization that the consolidation of their various PLM solutions is justified. Another scenario is the need for massive reconfiguration of the current platform. Regardless of the causes of the changes, the effective implementation of changes will depend on the successful migration of existing data on a new platform.

Data migration problems

Data migration makes numerous questions to the fore. For example, it may find out that this is the first audit of how the company chose the product management tools. In order to overcome the obvious disadvantages in the current data model, which has developed over time, the company may have to change parts of the model or expand it. In any case, these changes will have additional pressure on the transition process from the old to the new, simple data transfer may not be possible. One of the most important moments of migration is that you first need to determine all the data that is subject to consideration. Many believe that we are talking only about the transfer of digital data, however, experienced specialists certainly recognize that some part of the company's critical IC can still be in print, or at all its carriers are heads of leading employees.

After the data that is subject to migration has been identified, it is necessary to develop and execute the processes of checking their correctness. Often, data may be outdated, since the latest changes have not been made to earlier versions of the data. In addition, duplicate data is often used (or supported in several systems), requiring a constant or periodic test of consistency and cleaning data. Determining the full amount of data for migration requires the use of knowledge of the most experienced employees of the company.

Perhaps one of the biggest problems of data migration are the deadlines for migration. Old data will be constantly replenished and modified, since the company cannot stop its work and wait for the completion of the new implementation of PLM. In addition, in reality, the migration team has a very limited technical time for real switching, it is typically weekends or holidays. The need to meet in an affordable calendar time requires the implementation of migration algorithms using special tools, as the data can easily contain hundreds of thousands (or even millions) entries.

An example of a solution

Let us turn to the experience of those who implemented and migrate PLM data on a regular basis. One of the recognized specialists in the field of data migration PLM is the German company PRION GROUP, which has eleven-year experience in providing such services and effective tools for their implementation. Since the PRIR portfolio includes interfaces for the most common PDM and inherited systems, from which the data must be transferred, in each case, the company has no need to re-develop software for migration. This allows you to quickly develop a transition plan taking into account the characteristics of a particular company and quickly perform migration to minimize its impact on product development and production. The figure below shows a diagram of a typical data migration process for Prion methodology.

The most significant is that the performance of this scheme is repeatedly confirmed by the PLM data migration migration projects from numerous PRIR clients. Moreover, repeated attempts to produce direct broadcast of data from one PLM system to another, 100% of the inoperability of such a primitive approach were proved. Among the factors determining this state of affairs: data collection from several sources, the need to transform and clean the data, their certification and loading to a new system (s), which can also be physically distributed. Thus, there are completely unacceptable when switching to a new PLM platform.

To reduce these risks, Prion has developed migration tools that use an intermediate database. Data is exported to an intermediate database and, before downloading to a new PLM target platform, are converted to this database. Such an approach does not lead to immediate change in working data and business can continue as usual, while the rules and details of the data migration process are developed. The critical factor for the success of migration is to create a change management system so that not only to track changes made during the migration of the data themselves, but also changes to the data made after downloading to the new PLM platform. This change management system must support the specific requirements of the Customer from the migration process itself before launching a new operation system in real conditions.

The fact that Prion can use many of the migration scenarios from their extensive library, which was developed for former customers reduce migration risks and reduces costs for future customers significantly. This approach helps in many difficult migration scenarios, especially when the target system is a distributed solution.

For more information about Prion tools and services I recommend contacting the site.

Data transfer is the movement of stored digital information between computers, systems or formats. Data transfer occurs for a number of reasons, including replacement or server maintenance, changing data centers, data consolidation projects and system updates. Since most of the corporate knowledge and business analysts of the Company are contained in its data, any data migration project must be carefully completed to minimize risks.

Unloading "Data Migration"

Data transfer is a significant risk of business continuity if it is incorrect. Data loss, of course, is the worst scenario, but companies should also deal with idle time, compatibility issues and the overall system performance problem. Data transfer is hampered due to a large amount of data, various data formats and differences in data in corporations.

To minimize the data migration risks, the company creates detailed data transfer policies, which, as necessary, suspend the backup, movement of movement and parallel data environment. If the company cannot launch a preliminary migration environment when preparing a new environment, then there will be a significant idle time, as business operations on current applications are suspended to allow data migration. This type of stopping, transfer and starting may be required when switching to new platforms or in the presence of hard restrictions on physical storage, and swap or corrections are required for the existing storage technology.

Data migration with zero downtime

The migration model with zero downtime depends on the presence of a sufficient number of repository to create and start two full environments. A full copy of the company's data is taken into a new environment and is checked while employees remain in the old environment. Errors are developed from the new system, ensuring that all applications are still working, and everything is where it should be. Upon completion of testing, a new copy appears, and all employees go to a new environment. The old data environment sometimes remains open for several months so that employees can receive files from the old data system, but did not write new data to these servers. All data migrations are made to check the data after migration to verify data loss.

One thing that can improve data migration is cleaning and standardizing data practice to migration. The company's data organization is often a reflection of the various habits in filing their people. Two people with the same role can use completely different methods. For example, the preservation of contracts on the supplier in one case, as well as during the fiscal year and month in another. The unification of data methods can be a much more important task than the actual data migration, but clean, coherent organized data, supported by clear policies, help future evidence of the company's data for many migrations is still ahead.