Prevent import synchronizer from creating new elements in Target.
This request is based on experiences with MooD 15 v116, but i see it as still valid for MooD 16. My apologies if this has already been resolved in MooD 16 v45
In brief:
Similar to how there's a setting that prevents MooD from creating new elements when a relationship lookup fails, it should be possible to prevent MooD from creating new elements when no matching element is found for the target.
Purpose:
We often have situations where the customer will extract a list of elements from MooD to work with and gather further information or data for new fields to be added. When they do so the customer will often base their extract on the element name rather than other fields that might be more ideal for identification purposes, or they might base their extract on another system they have running in parallel with similar information. Either way the outcome is often that there are names of elements in the file to be imported that doesn't match the name of an existing element in MooD. Could be that the name was changed in the meantime, could be that it never existed in MooD, could be that the spelling is sligthly off, so if we import this file we end up with a bunch of new elements being created that pollute our data, and if we want to avoid it we need to do a bunch of data quality checks in excel before we import, where we need to extract data from MooD to compare with. Compounded with the case that we often don't have at-will access to current production data it becomes difficult to mitigate the risk of these kinds of errors occurring.
Ideally we could avoid these kinds of errors by configuring our imports to only allow the update of existing elements and not the creation of new elements.
Possible implementation:
In the Excel and Database import synchronizers under properties -> Map a new option is added "when element is not found in target" with the same selection options available as for "when a relationship lookup fails"
Guessing here but, in the code it might be implemented as a duplicate function of the existing "Insert or Update" SQL function, where "Insert" is removed from the function. To reduce technical debt it might be built so that this portion of the SQL statement is dynamically inserted based on the selection, rather than having to maintain two SQL statements.
Test:
1. Extract from MooD a dataset containing the element name and a field you want to update
2. Change the element name in MooD of one of the extracted elements
3. Add data to the extract for the fields you want to update
4. configure simple import, mapping element name as the key to the type's element name and the field to the field you want to update.
5. Set the new Map property to "ignore" and execute the import, confirm that all elements besides the element where the name was changed have been updated, and there doesn't exist a new element with the old element name.
6. Set the new Map property to "create it in the target" and execute the import, confirm that all elements besides the element where the name was changed have been updated, and there exists a new element with the updated value for the field and any other default values that an element of this type has.
Hope you find this relevant.
Br
Rune
Staun & Stender.
-
I agree, that's a solution, but
1. It's also a performance hit. We have situations with 30-60 SATs running in succession to achieve copies/versions of objects, updates of data and relationships and anything that can cut down on SAT execution time is money well spent.
2. It's also an implementation of an old anti pattern called "The Travelling Gypsies". Basically, you have things appear to make them disappear, then appear, to make them disappear. This creates audit logs inside MooD, CDC records if you're using CDC (build 117 of MooD 15 and forward). It's simply bad design.
We optimize our SATs in such a way that if we have 5 excel exports, we convert them into 1 excel export with tabs just to not having to open, stream and close files 5 times but only once. Every little performance hit matters in this area.
-
Yes, completely agreed Soren. It would definitely be better! Faster, fewer needless records, and also in addition less effort to set up in the first place and less likely to make mistakes and delete the wrong thing or miss anything.
I'm just offering a potential solution for anyone trying to work around this problem right now with existing functionality in case someone is reading this who has this exact issue and can't wait for a future build :)
Please sign in to leave a comment.
Comments
4 comments